Search
Systemic evaluation is the developmental evaluation (Guijt et al., 2012) of systemic change.
Techniques for systemic evaluation combine conventional principles and tools of developmental evaluation with concepts from systemic design. These techniques provide changemakers with the ability to assess the accuracy and completeness of their theories of systemic change and action (Murphy & Jones, 2020). They also allow evaluators to examine the progress of systemic strategies (Murphy et al., 2021).
The map represents your current mental model of how this system works.
Leverage analysis examines the patterns of connection between phenomena (using algorithms adapted from social network analysis and graph theory) in order to present relative rankings of the phenomena of the system.
These rankings are entirely dependent on the structure of the map. All phenomena are equal, and all connections are equal. It is theoretically possible to encode the degrees to which one phenomena influences another in strict mathematical terms and formulae. In turn, we could represent the map as a systems dynamics model and use it to simulate the behaviour of the system. However, this is usually impractical, especially with imprecisely-understood or hard-to-quantify concepts (e.g., what exactly is the rate of change in wildlife due to climate change, or how exactly does culture influence conspicuous consumption?)
For this reason, using leverage analysis is a fuzzy procedure. It depends on your intuition. Fortunately, the goal of leverage analysis is not to inductively estimate how the system will change, nor deductively falsify hypotheses about the system. Instead, using leverage analysis for strategic planning involves abductive logic: the generation of creative, useful conclusions from a set of observations.
The goal here is to look at the model as it is rendered and to think creatively about strategic opportunities. Broadly, this means asking several questions:
These questions can be answered via the following process.
The systems map represents a kind of high-complexity theory of change: it describes how all of these phenomena interlock and respond to one another. We can therefore use leverage analysis to weave systemic theories of action:
The chains of phenomena (and any loops they connect with) that result from the three steps above are the seeds of systemic strategy. Use them to identify key intervention points for programming (e.g., how might you take advantage of high-leverage phenomena? how might you address bottlenecks?), signals for monitoring and evaluation, and to communicate your theory of change/theory of action to others.
A prominent—infamous, even—feature of many popular productivity systems is the review.
The basic concept of a review is self-explanatory. You ask yourself questions like “what have I done?” and “what do I need to do?”, aided by lists of checked items or apps that serve up active and dormant projects.[^There can be more to it. See this episode of the Getting Things Done podcast for a more detailed discussion.]
Reviews are infamous, however, because they are notoriously challenging to do continuously. There are even whole podcasts dedicated to the challenge.
The review process is the keystone of most systems. It’s how we monitor, celebrate, and forgive the progress we make on the things we care about. It’s literally the most important feature in these systems for “staying organized.” So then why is it so difficult?
Perhaps it’s because this seemingly-basic process is actually quite complex.
Complexity is one of those topics that has an intuitive definition for most people. When something’s complex, it’s difficult! There’s a lot of steps or parts. It might be difficult to separate the components of a complex thing into separate pieces.
That intuitive definition, however, doesn’t appear to explain why reviews are hard. At face value, there’s not a lot of separate pieces in a review—only “what’s completed?”, “what’s not?”, and “what’s next?”, across the various projects you might have.
In practice, that intuitive definition of complexity is imprecise. We can learn more about complexity by comparing it to its siblings: complicated and simple.
A simple problem doesn’t have many steps or components, and the solution to a simple problem is the same regardless of the environment. Tying your shoelaces is a simple problem. Once you’ve learned how, you can follow the steps and arrive at the same conclusion every time.
A complicated problem might have many parts, but its solution is usually algorithmic. It might be more complicated to figure out a complicated problem, but once a solution is found, that solution can be applied again and again to get the same result. People like to say “this isn’t rocket science” to suggest that something’s not simple—and they’re right. Rocket science is complicated. Yet, once we have figured out how to launch a rocket, we can apply the same resources and processes to the same problem over and over again and get the same result.[^ Note that this doesn’t mean rocket science is easy. In fact, there are so many moving parts in rocket science that consistently solving its problems requires immensely powerful systems to make sure everything is done correctly and completely. “Murphy’s Law” is actually a parable of rocket science. Despite having the entire process of launching a test rocket completely mapped out and followed, a small mistake or malfunction still caused a test launch to fail, leading Edward A. Murphy, Jr. to suggest that if anything can be done wrong, somebody, somewhere will do it wrong. Murphy actually wanted his law to be the inverse: “if it can happen, it will.”]
A complex problem may have many parts and steps, but in addition, the application of those steps depends entirely on the system within which they are implemented. Raising a child is a particularly illustrative example of complex problems. Clearly, it’s impossible to raise any two children the same way. The same rules and incentives will apply completely differently to two siblings, let alone to children in different households or cultures.
So why are reviews complex? Well, no person ever reviews the same project twice, for it’s not the same project and they’re not the same person. We change, the world changes, and our responsibilities change. Arguably reviewing even has a quasi-quantum property: by observing our responsibilities, we change them. Ergo, even if you were to conduct a second review immediately after finishing a first one, the second review would yield different results.
From my perspective, this complexity is hidden. Reviews seem like a simple —or complicated, at worst—thing. That’s because we (are supposed to) do them regularly, and the content of our reviews are the things we deal with on a daily basis. Surely we shouldn’t be challenged simply by the idea of looking at these things to make sure we’re not missing anything.
Hidden complexity in a problem is itself a problem. Hidden complexity is a problem because we fail to use the right mindsets, tactics, and techniques to deal with the dynamics and uncertainty created by that complexity. Without the right approach, we exhaust our resources (in this case, our motivation and working memory) while failing to produce solutions. This means that we fail to either fully address our reviews or, worse, that reviewing becomes an impossible habit to stick to.
So what? How does this help?
One takeaway is to take advantage of the components of a review that are simple or complicated. For example, create a checklist what, exactly, you should do in a review. You could make this a template or you could create it at the outset, but either way, you shouldn’t engage in the process without without first explicitly defining its scope or path. Personally, I have a Shortcut that creates a new checklist in Trello for my review process. I just need to tap that, and then a boundary for the review is defined for me. Apps like OmniFocus can also help boil out complexity. OmniFocus encourages you to define review cycles for each area or project in your life, so that (for example) “Maintain the garden” doesn’t show up each week in the middle of Winter.
Second, acknowledge the limitations of your working memory. A comprehensive review makes you face down every single challenge you’ve decided to take on. It’s overwhelming by definition. The whole reason you wrote all of those things down and put them away in a list or an app is because you can’t think about them all at the same time… yet here you are, trying to juggle them all in your head at once. You would think that’s enough. Sadly, no: you’re also trying to grapple with latent personal changes and shifts in the world around you that have taken root since you last looked at the items in front of you. As a result, you probably experience cognitive overload. This overload ruins your ability to deal with the information in front of you while draining your capacity to continue with the review.
This means that you can’t actually do a review with only your lists of responsibilities and projects. Instead, to review effectively, you should also have your calendar(s) open, quick access to any potentially-relevant reference materials, and a freeform “review cache” (e.g., a blank page) where you can offload any of the questions or thoughts that come to mind as you look at the ideas in front of you. Ideally all of these things are visible to you at once. Switching back and forth between windows or pages is a sure way to overtax your working memory, as you’re trying to keep both concepts and the locations of information in your short-term memory.
The purpose of the “review cache” is to offload your thoughts into a semi-permanent visible space. When you think of a question or idea that doesn’t have an immediate answer, destination, or action, mark it down. Feel free to list, mindmap, doodle, whatever—as long as there’s somewhere to turn whatever’s on your mind into temporary reference material. If you do this effectively (which can be difficult—we are often tempted to hold onto a thought for “just a second”), it should make the review process easier and more joyful.
A third (but perhaps most important) lesson from this reflection is that the complexity of reviews are rarely acknowledged. It may be beneficial simply to realize that the review process is a potentially taxing one, and that you should be careful to go into it with lots of space and energy. For instance, I have always defaulted to trying to do a weekly review at the end of a day later in the week—by which point other responsibilities have had plenty of opportunity to get in the way and drain my stamina. By the time I get to my self-scheduled timeslot, the act of reviewing seems unimaginable.1 Based on these reflections, I schedule reviews at the outset of a day. By reviewing with a clear head and lots of energy, I’m actually able to get through it mindfully. In turn, the process itself is invigorating, I am encouraged by the feeling of control it gives me, and I look forward to it instead of dreading it.
So, to sum up, there’s a reason why it’s so hard to stick to a regular review schedule. To better equip yourself to do so, (1) try to simplify the process as much as possible through tools like checklists. (2) While you’re doing the review, limit cognitive load by keeping everything you need visible and by caching your thoughts as you work through the review. Finally, (3) acknowledge the actual complexity inherent in the process of conducting a review. Give yourself appropriate time and space so that you can actually engage with the content successfully.
Good luck!
And of course, I beat myself up over this because I should be able to muster enough energy to do a single stupid review! Hurrah for vicious, self-defeating feedback loops. ↩︎
In this research article, the authors point out that the cycles of translation from English to the language of the context and back again can be costly and inconvenient. But, they point out three benefits to investing in translation and multi-lingual research spaces.
First, the authors argue that disseminating the results of research in local languages not only makes your research accessible to stakeholders, but it also helps stakeholders value all research more. They write:
Translations are expensive and time-consuming, so a large part of our work stays in English unavailable to the local stakeholders, who may have participated in the research process. This is an issue not only because it reduces their possibilities to learn from the systematized outcomes of the processes in which they participate, but because it reduces their perception of the value of research. When stakeholders feel that researchers write exclusively for other foreign researchers, their readiness to support and fund research may decrease.
The second benefit:
Second, academics who don’t read English may find it difficult to continue building on knowledge published only in that language.
This takeaway is obvious. So many publications are translated to English, but the reverse is rare.
Third, and by no means least, naming complex issues or ideas only in English impoverishes other languages. When we forsake finding a word for a particular concept or idea in a given language, we impoverish that language.
This is quite insightful. Language is intrinsic to organizational learning. If the concepts advanced in our research are never introduced to the local language, then it may be impossible for that learning to take root.
The authors recognize a fascinating tension in this work. They demonstrated the possibility of multi-language research spaces via a virtual research commons for their project.
What we have learned from working with different languages and acknowledging them during the full research cycle, including the dissemination stage, is that they are time- consuming, costly and even a bit messy and uncomfortable. For example, in the case of the virtual space above, some participants complained that having to find their own language among texts written in other languages begs an extra effort from them and slows them down. However, the alternative is renouncing inclusion and plurality, which is at odds with the challenge faced by academia to address complex societal problems.
There is a cost to complexity, but solution spaces need to be more complex than the problems they’re resolving.1
See Ashby’s Law of Requisite Variety; http://pespmc1.vub.ac.be/ASHBBOOK.html, p. 207 ↩︎
