Last week I attended Festival de Datos in Uruguay. While my work is more… ‘data adjacent’, it’s a short flight from Peru (my current temporary home) so why not spend 3 days immersed in a world of data for development. I am sharing a few reflections / questions from the sessions I attended here. If you’re interested in seeing what went down, session recordings are available on YouTube.
There was much discussion of generative-AI at the Festival, so as an experiment, I thought for my final reflection on the event, why not enlist the aid of ChatGPT. So, I collated my very rough notes and crafted a prompt . . . The blog below is 99% the verbatim output, aside from a few very minor tweaks, some formatting, and the addition of related links.
Worryingly… despite being a bit wordier than my style – its not bad!
Festival de Datos recently brought together minds from across the aid and development sector, igniting enthusiasm for the potential of digital information in shaping a brighter future. Amidst the celebration of data’s critical role, a plenary comment echoed a familiar sentiment from the business world:
“What we can’t measure, we can’t value; and what we can’t value, we can’t improve.”
While this statement resonates with the festival’s theme, it also raises a crucial question about the potential pitfalls of overemphasizing measurable outcomes.
In the three-day festival designed to illuminate the power of data, the idea that our time, attention, and budgets should be directed toward measurable entities seems appealing. Yet, as we delve into the nuances of this approach, a reservation emerges – one that has plagued not only the aid and development sector but also broader business for decades.
Some things resist accurate measurement, and this isn’t indicative of their lesser importance. Consider concepts like human happiness, love, mental health, or (moving closer to the pursuit of the SDGs) – the challenging terrain of structural inequalities, advocacy for political change, protection from digital harms, or the potential shift to a world of “tech feudalism”. These are vital aspects of our work, yet they often elude reliable quantification.
Certainly, many of these critical components have related indicators, but they remain difficult or impossible to measure with precision. However, their importance cannot be understated, potentially holding far more weight than the easily quantifiable data that tends to focus on immediate results and outcomes, sidelining complex, hard-to-attribute social impacts.
In pondering this, we must ask ourselves: Are we risking swinging the pendulum too far? The historical lack of measurement, value, and understanding of data posed significant challenges. We have made great strides in overcoming these hurdles, but is there a danger of now excluding equally valuable work that doesn’t neatly fit into our measurement frameworks?
Reflecting on this, we find a potentially instructive example in the work of J-PAL, renowned for their expertise in Randomized Control Trials (RCTs), often considered the gold standard of measurement in the evaluation realm. While RCTs are undeniably valuable, they have their limitations, particularly in measuring certain types of interventions. The sector’s initial fervor for RCTs led to a concentration of funding on interventions amenable to this gold standard, potentially neglecting equally significant work that did not align with RCT methodologies.
J-PAL itself has since evolved, acknowledging in its own presentations that some subjects, such as political change, operate in ways that RCTs are not always relevant to – a recognition that doesn’t diminish the importance of these endeavors. This highlights the need for a nuanced approach to measurement, acknowledging the strengths and limitations of various methodologies.
So, how do we continue to place data at the center of our work without inadvertently excluding vital initiatives that may not yield significant, reliable, and accurate data? Striking the right balance requires a collective effort and a willingness to engage in an ongoing dialogue.
In the comments below, I invite you to share your suggestions and reflections?
How can we navigate the data dilemma, ensuring that our focus on measurement doesn’t inadvertently sideline crucial, yet harder-to-quantify, aspects of our work?Let’s pool our insights and shape a future where data remains a powerful tool without overshadowing the importance of all-encompassing, impactful work that defies easy measurement.
Matt Haikin 