MERL Tech DC 2018

Adaptive Management: Is MERL helping or hindering its potential for success?

Adaptive Management is this year’s must-have accessory for any development programme – but is it living up to its potential, why aren’t more programs adopting it, what happens when it goes wrong?

DIAL (the Digital Impact Alliance) will be hosting a fun, interactive, panel show to look at the above questions through a MERL Tech lens.

The panelists, with help from the audience will challenge some common myths and debate the key questions:

• Where is Adaptive Management being adopted and what successes and failures are we seeing?
• To what degree does our integration of MERL technologies help or hinder taking a more adaptive approach?
• How can the MERL Tech community do more to help adoption and success of adaptive approaches – and should we be doing this?

Adaptive Management and MERL Resources
Adapting Aid case studies from Mercy Corps and IRC (2016) - https://www.mercycorps.org/research/adaptive-management-case-studies

Making M&E work for adaptive management requires strategic skills (INTRAC, May 2018) - https://www.intrac.org/making-work-adaptive-management-requires-strategic-skills/

Monitoring and evaluation: five reality checks for adaptive management (ODI, Dec 2017) - https://www.odi.org/comment/10588-monitoring-and-evaluation-five-reality-checks-adaptive-management

USAID Learning Lab: https://usaidlearninglab.org/node/14636 -- has both monitoring toolkit and evaluation toolkit

Harvard PDIA in practice series - notes and a podcast series https://bsc.cid.harvard.edu/pdia-practice-series

 Oxfam AM rules of thumb (June 2018) https://oxfamblogs.org/fp2p/seven-rules-of-thumb-for-adaptive-management-what-do-you-think/

 The fashion for agile management is spreading (The Economist, July 2018) https://www.economist.com/business/2018/07/05/the-fashion-for-agile-management-is-spreading?cid1=cust/ddnew/email/n/n/2018079n/owned/n/n/ddnew/n/n/n/nNA/Daily_Dispatch/email&etear=dailydispatch&utm_source=newsletter&utm_medium=email&utm_campaign=Daily_Dispatch&utm_term=2018079

 How can Monitoring, Evaluation and Learning support adaptive management? (July 2018) https://oxfamblogs.org/fp2p/simplicity-accountability-and-relationships-three-ways-to-ensure-mel-supports-adaptive-management/

 How USAID can bridge the disconnect between its procurement process and evidence (July 2018) https://www.cgdev.org/blog/usaid-thinks-about-procurement-and-program-design-it-should-keep-evidence-mind

 How can we support the Adaptive Development Community?https://drive.google.com/drive/folders/1NQm-21WlTGmojVAXUJak_Bd21ogneuJ6

Innovations in evaluation: How to choose, develop and support them (May 2018) https://www.betterevaluation.org/sites/default/files/Innovations%20in%20evaluation_BetterEvaluation-UNICEF-EvalSDGs.pdf

Email mhaikin@digitalimpactalliance.org to add your resource

Dashboards: everyone says they want them, but many sit unused on web portals and shared folders. During this hard talk session, four data enthusiasts who work on dashboards every day will share their experiences on pitfalls that go beyond color palettes and tech stacks, and explore how we can be more mindful of the human side of dashboard design and use.

The panelists will debunk common assumptions and practices, share stories of their own dashboard design failures, and surface disagreements in our development community about why dashboards fail, succeed, or sit barely used on shared drives.

==========

Favorite Resources from the Panelists

Big Book of Dashboards - great primer on dashboard development; examples are in Tableau workbooks but the concepts are widely applicable
Flowing Data - cross cutting blog that hits on design, tools, development processes and more
New series from Ken Flerlage - SQL for Tableau - highlighting how data viz developers can benefit from learning some basic coding skills

Blogs on dashboard design tricks (primarily Tableau-focused, but ideas can be more widely applied):
Ryan Sleeper
Viz Wiz (Andy Kriebel)
Data + Science

Chart selection heirachies, for ideas on what chart to use when:
Visual Vocabulary
Data Viz Catalogue
Data Viz Project

Blogs on data viz best practices:
Policy Viz
Storytelling with Data
Ann K Emery
Evergreen Data

Inspiration for what to do (and not to do) in data viz design:
WTF Visualizations
Junk Charts
Makeover Monday (#makeovermonday on Twitter)
SWD Challenge (and #SWDchallenge on Twitter)
Energy in America (Tableau dashboard example Amanda called out)

The most challenging aspects of purposeful Monitoring, Evaluation, Research and Learning (MERL) are most often: 1.) selecting informative metrics and 2.) performing MERL efficiently. Many individuals in MERL assume that technology and big data will dramatically improve both the selection of informative metrics and MERL efficiency. However, as a community, we also understand that existing structural constraints in the MERL ecosystem may either hinder or render ineffective technology-enabled tools. In addition, the biases present in large data sets may mean that metrics selected using machine learning are no more informative than those already in use and that MERL will ultimately be ineffective at improving institutional performance. In this interactive panel, we will discuss the existing structural barriers to the successful implementation of novel technology in the MERL space and what realistic expectations should be regarding the ability of big data to improve MERL outcomes.

In this interactive discussion, our participants will both try and answer the question:

Will technology and big data improve MERL or will existing structural issues in our communities, paired with known biases in existing data sets, make MERL all-the-more challenging?

Session slides

In ICT4D, we literally do not know what we are doing.

Innovative solutions are piloted and trialled - but ​few scale successfully, and we aren't sure why​. We duplicate past experiments with little intentionality about testing different methodologies or iterating on an idea. We don't understand our impacts, intended and unintended, so we may be causing harm to the most vulnerable. Risk identification and mitigation ahead of deployment is generally poor. And innovations can be solutions looking for problems: disconnected from the context, real needs and capacities, and the wider field’s existing efforts.

Understanding the impact of our work through evidence, holding ourselves to account against sound ethical and best practice frameworks, and acting on this knowledge to improve practice over time, would improve outcomes; allow organizations to take better risks; and most importantly reduce harm to populations. These goals are weakened by underinvestment, a lack of openness around learning and evaluation, technical challenges and uncritical discourse. Monitoring and evaluation of innovative projects carried out by staff unaccustomed to new technologies may miss relevant assessment criteria. Pilots, innovation challenges, and other common funding mechanisms may have very light reporting requirements. And we don't have a culture of transparency around learning and evaluation.

But it’s time we professionalized. It's time to stop ‘innovating’ and start taking incremental steps towards excellence, fuelled by evidence.

In this provocative panel, we will discuss the evidence gap is - how many evaluations SHOULD we have? What SHOULD we know? How can we challenge policymakers to commit to starting to change the culture, and establishing a norm of evidence-based action in ICT4D?