Search
Systemic evaluation is the developmental evaluation (Guijt et al., 2012) of systemic change.
Techniques for systemic evaluation combine conventional principles and tools of developmental evaluation with concepts from systemic design. These techniques provide changemakers with the ability to assess the accuracy and completeness of their theories of systemic change and action (Murphy & Jones, 2020). They also allow evaluators to examine the progress of systemic strategies (Murphy et al., 2021).
We seek leverage to find the best ways of making change.
Leverage points are places in systems where a little effort yields a big effect (Meadows, 1997). They are also ideas that help us grab on to strategic ways forward when we’re working in complexity (Klein & Wolf, 1998).
Acting on leverage points may accelerate systemic change towards progress and reform, but acting on the wrong ones may instead accelerate systemic change towards regression and deformity. Well-designed leverage strategies may be catalyzing or even transformative, but poorly designed ones may merely be futile (figure 1).
One way of finding leverage points is to think through your system with reference to Meadows’s (1997) 12 types:
Table 1. Twelve types of leverage points, in order of increasing power (adapted from Meadows, 1997).
| Twelve types of leverage points, in order of increasing power | Example |
|---|---|
| 12. Constants, parameters, numbers (such as subsidies, taxes, standards) | Wages, interest rates |
| 11. The sizes of buffers and other stabilizing stocks, relative to their flows. | Current levels of debt/assets |
| 10. The structure of material stocks and flows (such as transport networks, population age structures) | An individual’s financial structure (e.g., fixed costs and incomes) |
| 9. The lengths of delays, relative to the rate of system change | How long it takes to find a higher-paying job |
| 8. The strength of negative feedback loops, relative to the impacts they are trying to correct against | Rising costs of living vs. fixed income |
| 7. The gain around driving positive feedback loops | Recession causing reducing spending |
| 6. The structure of information flows (who does and does not have access to what kinds of information) | How aware you are of impending recession/future rising costs |
| 5. The rules of the system (such as incentives, punishments, constraints) | Who suffers as a result of poorly-managed recessions |
| 4. The power to add, change, evolve, or self-organize system structure | Central banks, Ministries of Finance |
| 3. The goals of the system | GDP Growth |
| 2. The mindset or paradigm out of which the system—its goals, structure, rules, delays, parameters—arises | Growth above all |
| 1. The power to transcend paradigms | Sustainable development, flourishing |
Another approach, which may be complementary to the above, is to model the system as a causal loop diagram (e.g., Kim, 1992) and then to conduct leverage analysis (Murphy & Jones, 2020) on the model.
An understanding of leverage in a system allows us to generate systemic strategies (Murphy & Jones, 2020). These strategies can also be adapted into Theories of Systemic Change (Murphy & Jones, 2020).
Donella Meadows (1997) popularized the idea of leverage in systemic change with her essay “Leverage Points: Places to Intervene in Complex Systems.” She proposed a typology of phenomena in a system, suggesting that acting on certain types of phenomena are higher-leverage than others.
In an article published in the Contexts journal of systemic design, I challenged Meadows’s (1997) paradigm, proposing a few other possible ways of viewing leverage. My aim was to link the search for leverage directly to the design of powerful strategies for systemic change, and to propose a few ways forward in advancing our understanding of leverage in complex systems.
My article “Leverage for Systemic Change” was recently published in the inaugural edition of Contexts, from the Systemic Design Association.
The article ultimately proposes a few key directions for a research agenda on leverage in systemic design (see the table below).
Table 1. A research agenda for leverage theory in systemic design
Research area Research questions Existing research Possible studies Possible contributions Dimensions of leverage - Is Meadows’s (1997) typology complete? - What other features of the “physics” of systemic change might matter?- A systematic literature review (Okoli & Schabram, 2010) of leverage points, especially using forward citations (Haddaway et al., 2022) from (Meadows, 1997) - Understanding the nature of leverage and other mechanisms of change potential in systemic change Methods for leverage - What methodologies are best to identify and select leverage points? - What kinds of evidence will help validate leverage?- How might systemic designers design theories of change (Gregor & Jones, 2007) for leverage theories?- How might systemic designers limit indeterminism (Lukyanenko & Parsons, 2020) in leverage theories?- Meadows’s (1997) typology’s order of effectiveness - Leverage analysis [Murphy & Jones, 2020]- Assessing potential for change (Birney, 2021)- Surveying practitioners in systemic design on how they identify, assess, and address leverage points to identify common habits and best practices - How to identify phenomena useful for leverage - How to evaluate and compare possible leverage points in the analysis phase- How to evaluate the effectiveness of chosen leverage points with evidence gathered from implementationsStrategy with leverage - How is leverage best used in developing strategic plans for systemic change? - How are leverage-based strategies best presented and communicated?- How are leverage-based strategies best evaluated and measured?- “Systemic change labs” tracing and comparing the impact of interventions using different kinds of leverage Execution on leverage - What are the best ways to target different kinds of leverage for systemic change? (E.g., how might we help actors in a system track all of the relevant paradigms?) - Fruitful friction as a tactic for transcending paradigms (Buckenmayer et al., 2021) - Systemic change happens via multiple dimensions of change (Mulder et al., 2022)- Design Journeys offers several chapters on taking action after identifying leverage points (Jones & Ael, 2022)- “Systemic change labs” tracing and comparing the impact of interventions using different kinds of leverage - How to design innovations for each type of leverage
Some other key takeaways:
Our toughest problems resist conventional strategies for change. In this talk from Peter Jones and I, we show how designerly approaches—namely methods from systemic design—can help create and implement systemic theories of change. Those theories may then be used to develop effective strategies for wicked problems.
We presented this talk at the Academic Design Management Conference in 2020, and it led to a follow-up paper.
The map represents your current mental model of how this system works.
Leverage analysis examines the patterns of connection between phenomena (using algorithms adapted from social network analysis and graph theory) in order to present relative rankings of the phenomena of the system.
These rankings are entirely dependent on the structure of the map. All phenomena are equal, and all connections are equal. It is theoretically possible to encode the degrees to which one phenomena influences another in strict mathematical terms and formulae. In turn, we could represent the map as a systems dynamics model and use it to simulate the behaviour of the system. However, this is usually impractical, especially with imprecisely-understood or hard-to-quantify concepts (e.g., what exactly is the rate of change in wildlife due to climate change, or how exactly does culture influence conspicuous consumption?)
For this reason, using leverage analysis is a fuzzy procedure. It depends on your intuition. Fortunately, the goal of leverage analysis is not to inductively estimate how the system will change, nor deductively falsify hypotheses about the system. Instead, using leverage analysis for strategic planning involves abductive logic: the generation of creative, useful conclusions from a set of observations.
The goal here is to look at the model as it is rendered and to think creatively about strategic opportunities. Broadly, this means asking several questions:
These questions can be answered via the following process.
The systems map represents a kind of high-complexity theory of change: it describes how all of these phenomena interlock and respond to one another. We can therefore use leverage analysis to weave systemic theories of action:
The chains of phenomena (and any loops they connect with) that result from the three steps above are the seeds of systemic strategy. Use them to identify key intervention points for programming (e.g., how might you take advantage of high-leverage phenomena? how might you address bottlenecks?), signals for monitoring and evaluation, and to communicate your theory of change/theory of action to others.
A presentation from Peter Jones and I at RSD9, virtually in Ahmedabad, India.
This investigation answers the questions: What are the mechanisms of education systems change? How might we use systemic design to power a reform movement? What does systems thinking teach us about opportunities and barriers for education reform in Newfoundland and Labrador?
Education reform is not a new idea. Many organizations and initiatives have reimagined the schools, universities, and colleges that are the backbone of our economy and society—yet, to many, it seems like little has changed in recent decades.
As Jal Mehta, Robert Schwartz, and Frederick Hess began their book on the subject, “if we keep doing what we’re doing, we’re never going to get there.” Traditional education reform approaches have depended on a best practices approach in what is glibly called a “silver bullet culture”.
A single idea, found successful in a specific institution or district, becomes hailed as the be-all-end-all solution. This solution is then celebrated and championed across contexts until actors realize that expected results have not materialized, and reformers move on to the next silver bullet solution.
This approach to education reform has not worked. According to Mehta, Schwartz, and Hess: “If we are to deliver transformative improvement, it is not enough to wedge new practices into familiar schools and districts; we must reimagine the system itself”. In other words, education systems change has been found to be more than difficult. It is a wicked problem: ill-defined, constantly fluxing, with many conflicting stakeholders and no true solutions.
How can we provoke change in wicked problems?
I argue that reforming education is a sociotechnical problem, involving human psychology; social, political, and economic factors; and complex interactivity–what Don Norman and Pieter Stappers have called a “DesignX” problem. These authors suggested that DesignX problems can only be solved through a process of muddling through, developing incremental sub-solutions through deep analysis. This deep analysis means partitioning the problem into modules and recognizing the intersecting dimensions of the problem.
Peter Jones provides further advice for this kind of problem solving, defining an approach called systemic design. Systemic design integrates systems thinking and systemic methods–ways of understanding complex problems through the relationships of the phenomena and actors involved–with design thinking and design methods, applying human-centred design to these seemingly intractable, large-scale problems. With systemic design, Peter says, we use “known design [tools]—form and process reasoning, social and generative research methods, and sketching and visualization practices—to describe, map, propose, and reconfigure complex services and systems”.
With this in mind, I turned to several types of modelling in order to use systemic design on the complex problem of education reform: process modelling, actor mapping, and causal loop mapping. In this modelling, I focused on Newfoundland and Labrador—my home province—and the education of innovation, examining how our system currently provides innovation learning and how we might do better.
Finally, I applied centrality analysis – a quantitative approach to assessing leverage points and bottlenecks in networks and systems maps—in order to surface potential opportunities and challenges in the system. I examined four types of centrality analytics:
Reach efficiency takes an element’s reach (the proportion of the network within two steps of that element) and divides it by the number of neighbouring elements it has. Elements that score the most on this metric tend to be less connected but have high exposure to the rest of the system, making them low-hanging fruit for change efforts.
Betweenness assesses the number of times an element lies on the shortest path between two other elements. Elements high on the betweenness metric are bridges throughout the map, controlling the flow of phenomena throughout the system. This means that these elements may be bottlenecks or single points of failure.
Eigenvector centrality measures an element’s connectedness to other well-connected elements, computing an overall value that is an indicator of the element’s influence over the whole system.
Torque calculates each element’s reach efficiency weighted by its eigenvector centrality (e.g., how influential that element is). Elements with high torque should be relatively easy to impact (as they are not densely influenced by other phenomena in the map), but will impact the rest of the map substantially. These are key leverage points of change.
The detailed methods and findings from each of these approaches are discussed in turn in the links below.
button: Process Modelling: Where might innovation learning come from?
button: Actor Mapping: who is involved in this system?
button: Causal Loop Mapping: how does the system behave?
I aimed to see the education system for what it is in order to describe strategies for the transformative reform that Mehta, Schwartz, and Hess called for. The education system is therefore composed of a number of interrelated components, organized in a hierarchy, whose emergent phenomena lead to its own dynamics. Yet, many might say that this systemic chaos implies a system of constant change, while education is hallmarked for its derelict stagnancy in the 21st century. How is it that such a system has not evolved?
Well, perhaps the system is not actually that broken. As eloquently argued by Ryan Burwell, an instructional designer at the MaRS Discovery District:
The school system is not broken. It is perfectly aligned to provide equitable access to a canon of high-quality, standardized content with greater rigour and organization than any other knowledge delivery system we currently have. However, it is not designed to foster the problem-solvers, innovators and entrepreneurs that are becoming an increasingly significant part of the global economy. Incorrectly identifying this misalignment as a broken system has created a culture of fear and failure around education, leading to top-down reforms and increased numbers of mandatory programs.
I return to Mehta, Schwartz, and Hess’ depiction of school reform’s silver bullet culture. Many stakeholders with competing interests and different priorities are invested in every debate on education systems change.
Thus, there are many potential silver bullets–and many advocates for them. The misunderstanding of the problem described by Burwell and the complexities of education reform described by Mehta, Schwartz, and Hess perfectly capture the need for a systemic design-based approach to change.
From process modelling, it is clear that while NL’s education system currently offers some opportunities to learn certain constructs of innovation, the availability of these opportunities is not densely packed throughout their study. It is easy to recognize a dearth of access to the domains of Foresight and Scanning, Vision and Purpose, and Adaptability and Resilience. Further, the degree to students learn the domains and constructs of innovation skills from the public system remains unclear.
Ultimately, now that these models exist, further analysis will be able to examine these constructs more closely as students progress through the system.
This is especially true for many of the “optional” components of the broader education system. After school programs, hobbies, sports and recreation, volunteer and extra-curricular roles, self-directed learning, and employer training could each be vital sources of innovation education, but it was impossible to study these aspects of the system in any meaningful way in the present study. A dedicated effort should examine the availability of these sources and assess their utility for innovation learning.
One research approach would be to survey learners along the learning journey, testing their abilities in the different constructs I’ve outlined. This ethnographic approach could reveal hidden truths: perhaps, for instance, certain regional cultures in the province actually provide powerful learning in design through a community culture alone.
Systemic modelling reveals the power and wealth subsystems active amongst the actors of the education system.
Centrality analysis of the power subsystem illustrates that parents and the provincial government have efficient influence on the system, and change that can mobilize those bodies of actors will quickly take shape.
Meanwhile schools, the School Board, and educators have substantial global influence over the system–change efforts that engage these actors may be slow but momentous.
Finally, power bottlenecks are educators; schools and school councils; and the Department of Education and Early Childhood Development. This suggests that these actors will ultimately need to be involved if any reform effort were to achieve success.
Reach efficiency analysis of the wealth subsystem shows that the federal government, parents and students, and the provincial Department of Advanced Education and Skills each strongly influence the distribution of wealth. The Federal and Provincial Governments have powerful incentives with which to motivate and control reform efforts.
Betweenness centrality revealed that the whole system is tightly linked, making it potentially volatile: economic issues in one component of the system may ripple out and impact the others.
Finally, these maps intimated a causal loop diagram illustrating how innovation education reform might happen in the public education system.
Several loops and one archetype demonstrate significant effect over the system. The Low Definition loop describes an acceleration of the impact of ill-defined innovation on our ability to educate on it. The We Teach What We Know loop shows how a lack of innovation education leads to a lack of people capable of teaching it and vice-versa. The New Economy loop shows how economic shocks driven by drops in commodities pricing has raised our awareness of the importance of the innovation economy. The Innovation-driven Growth loop shows how innovation capacity will accelerate jobs in the knowledge economy, which will in turn drive our ability to create more innovators through education. The Limited Resources loop balances our ability to reform education for innovation due to a lack of funding for the reform effort due to austerity budgets, driven by drops in the price of oil. Finally, the R&D, Not Innovation archetype is an instance of the Fixes that Fail systems archetype, showing how a conflation of innovation with R&D efforts fails to improve our innovation capacity while also distracting from true innovation education.
The result of centrality analysis on these causally-linked phenomena is rich with pragmatic insight.
Three phenomena with efficient reach over the whole system are innovation learning from outside of the public education system, lack of emphasis on innovation education, and low price of oil. The former points to an accessible lever of change: introduce innovation education through extra- and co-curricular programs, volunteer and leadership roles, sports and recreation, or self-directed learning, and the system may catch up by offering its own programming to match. The leverage of a lack of emphasis on innovation education offers another route: increase awareness on innovation education in order to encourage the system to improve on it. Finally, low price of oil retains leverage as a dampener on the system: if the economy continues in recession, the system is less able to offer resources for reform efforts.Â
Other calls for reform is one force with substantial torque over the system, indicating that reformers must be co-opetitive with other education change efforts, else all reform efforts might fail due to competition with one another. The availability of accessible and practical models for innovation education is another high-torque element, however, elevating the potential of the present research to create change in the system. A third element with high torque is the generational shift in work, evidence that a substantial source of impetus for innovation education reform could come from changes in work and careers.Â
Finally, betweenness centrality offers a picture of the bottlenecks and points of failure within the system. Innovation capacity and innovation education are two forces semiotically central to the system, and thus it is intuitive that they will be slow to change, no matter what else is happening within the system. On the other hand, recognition of innovation deficiency, the perceived innovation gap, and the search for solutions to the innovation gap are three phenomena that are clear points of fragility in any systemic change effort. If the system does not recognize its deficiencies, perceive the gap in innovation capacity, or opt to search for solutions, reform efforts are liable to be frustrated.
Despite these clarion recommendations for systemic design, several limitations prevent wholesale adoption.
One key limitation of the presented results of systems modelling is that the connections defined in these models are unquantified. In the refined actor map, for instance, it may be that educators have little power over their school councils, or perhaps the NL Federation of School Councils has far less lobbying capacity than the NL Federation of Teachers. Evaluating the strength of these connections and including these evaluations in our analytics would improve the acuity of those metrics substantially.
As previously mentioned, if innovation learning is not coming from the public education system, it must be coming from somewhere else. Yet, these potential sources arguably include the whole of the human experience—as we have, after all, been learning to innovate since pre-history. Future research might take on an ethnographic approach to understanding the system, investigating different student-innovators and where they learned their innovation skills, or a longitudinal approach, following students as they become innovators through their years in the education system. These exercises fell outside the limits of the present research, unfortunately.
Another limitation is that, while the scope and approach to mapping were designed to increase the variety of the system as much as possible, the mapping was still completed with the perspective of only this author. The representativeness of the systems models would therefore be strengthened considerably with Delphi-inspired methods as seen in previous research, bringing the mapping process to others in order to iteratively refine and the map from alternative stakeholders’ points-of-view.
Another potential future study is to “bring the whole system into the room”. This would mean convening a group of stakeholders who were holistically representative of the actors of the system, engaging them in a systems modelling process to develop a map with their collective perspectives.
A few immediate actions stem from this research.
First, we must adopt a model of innovation skills and competencies.
Regardless of whether the adopted model is the one developed through this project or another alternative, it is imperative that we begin to recognize the skills and competencies used by successful innovators. By identifying these skills, we will be capable of examining our weaknesses and, in turn, developing ways of resolving those weaknesses. To spur this discussion, I plan on sharing the models developed here widely.Â
Second and in tandem, we must include the role of the education system in nurturing innovators in our provincial and national innovation strategies. Many approaches to innovation policy discuss the post-secondary education system with respect to its role in public-private partnerships and the commercialization of research. We must expand this role to include the development of innovation skills and competencies as well. In the near future I hope to meet with policymakers involved in the development of Newfoundland and Labrador’s innovation strategies to advocate for this approach there.
Education reform movements must be united in their calls for change. A host of movements relate to the notion of innovation education, from code.org (a non-profit urging computer science and programming education in K-12) to the 21st century learning movement (a pedagogical framework for the skills and knowledge necessary for the 21st century; cf. http://www.p21.org).
The present research shows that these reform efforts may conflict, however, if they are brought forward asynchronously by their champions. It is therefore crucial that these efforts learn to “co-opete” (as in “co-opetition”) and engage educators and policymakers with aligned advocacy. I hope to work with the education systems change movements I already have relationships with in the immediate future in order to begin this dialogue.
As explored by David Stroh in Systems Thinking for Social Change, systems change is only possible when the actors of the system collectively recognize the tension between where the system is and where they want it to be.
That realization isn’t possible, however, before the actors have even talked to one another—let alone come to consensus about a shared vision for the future.Â
We realize that Canada’s future prosperity is predicated on our ability to leverage the boons of our resource economy and evolve it into an “innovation rich” leader in the knowledge economy. Yet, as discussed at the beginning of this paper, the danger is that education’s role in this transformation has yet to be recognized in full. We are not talking about how to create innovators, let alone what strategies we should employ in doing so, or how the system is stuck in becoming better at innovation education. Worse, there are many simultaneously conversations happening in both education reform and innovation—conversations that compete with one another, threatening the potential of the whole.
This research offers a model of the education system in Newfoundland and Labrador. Yet these models are untested, and as I have noted, the research is sorely lacking a futures perspective that observes both threats and strategic opportunities in our changing environment.Â
**How might we spark a collective, integrative discourse on innovation and innovation education? **Then, how might we elevate its importance such that collective action is taken—before we’ve missed the opportunities of the knowledge economy? How might we refine the systemic models, and how might we augment this work with a futures perspective, using environmental scanning to develop and integrate changing trends for strategic leverage?Â
These questions point toward a need for a powerful, strategic theory of change, and the willingness to muddle through. In other words, **this change will not come about through the efforts of ad hoc standalone initiatives like this one. **
We need a sustained effort. We need a lab that brings together design science and systemic design, creating and testing designs of the system itself, making sure they are valid constructs of the concepts they are intended to represent, all while obeying the principles of systemic design.Â
This is not a new idea. Many have articulated the notion of social labs, design or change labs, or social innovation labs. In fact, the OECD’s Centre for Educational Research and Innovation seems to operate such an approach for systemic innovation in global education.
I argue that Canada–or at least, Newfoundland and Labrador–needs to take a lab-based approach to navigating complex education reform in education. This lab must unite the perspectives, strategies, and actors currently engaged in similar pursuits; build, maintain, and refine models of the systemic change taking place; be engaged in environmental scanning and strategic foresight to monitor for both threats and opportunities; and prototype change initiatives, taking lessons back to these models and strategies.
Only a dedicated, intelligent effort will help us build the education systems that will develop the skills and knowledge we need to answer the 21st century.Â
Innovation Education is a major research project presented to OCAD University in partial fulfillment of the requirements for the degree of Master of Design in Strategic Foresight & Innovation. This work was supported by the Social Sciences and Humanities Research Council of Canada.
What is innovation? How do we define innovation, its outputs and processes, and what are the skills and competencies necessary to practice and excel in innovation?
Many strategies and policies, both federal and provincial, have attempted to improve Canada’s innovation capacity in the last few decades. Universities and colleges are often discussed in these strategies for their role in facilitating new partnerships and in developing (potentially) actionable research.
It is rare, however, for these strategies to recognize education plays in the creation of innovators.
This is counterintuitive: education is an obvious mechanism with which to develop the knowledge and abilities of a population. Yet we lack a holistic understanding of what it takes to practice innovation, let alone the kinds of curricula that might provide those skills and competencies. Moreover, we are inconsistent in the definitions and language we use to define innovation—often obsessing over technology and commercialization. We tend to assume innovation comes from research and development processes, and that innovators are simply highly skilled people.
Presented here is the result of an intensive review of reports, strategies, policy, and theory on innovation in the Canadian context. This literature was scanned and coded—inspired by the ethnographic methods of grounded field theory—in order to synthesize a holistic theory of innovation.
This theory includes a universal definition of the innovation process, a recasting of different focuses of innovation as innovation orientations, a comprehensive model of the innovation process, and a synthesis of innovation skills and competencies into 13 learning domains, 47 learning constructs, and 227 learning outcomes. Use the interactive map below to explore the model yourself! These tools provide utility for policymakers and educators in pursuit of understanding and improving innovation capacity. In particular, the model of innovation education is the most comprehensive of its kind, providing an extensive set of concepts with which to understand education gaps and build curricula.
Perhaps the most important contribution of this research, however, is the recognition that our conversations about innovation strategies and education reform must be aligned. **How exactly do people learn to be innovative, and how are our education systems currently facilitating that process? **With this study we have begun to seek answers to these questions, but there is much more work to do.
If you use these ideas and learn something from your experience, or if you have thoughts on how to improve them, don’t hesitate to make suggestions and to share your work.
Find resources on this page that detail the definitions and models of innovation developed through this research and how these concepts may be applied in innovation education programming.
Across dozens of perspectives on innovation, a set of 13 domains of skills and competencies emerged. These domains have been interactively visualized using Kumu.io. You can play with the work yourself by clicking the button. Use the controls in the bottom corners to focus on specific components of the model, to surface the innovation process so that you can explore both simultaneously, and more. Note that the visualization is best explored on a desktop!
Play with the visualization here.
This document provides an executive summary of the insights and models on innovation education uncovered through this research and guides educators to put these concepts and tools to use.
Education reform presents an opportunity to improve innovation education and, in turn, advance innovation capacity. I synthesize the framing and strategy of resources from provincial, national, international, and theoretical perspectives on innovation in order to develop a holistic model of innovation and a curricula for innovation education. Then, I use systemic design to model Newfoundland and Labrador’s current education system and to suggest strategies for reform to enable improvement in Newfoundland and Labrador’s innovation education. Finally, I explore how systemic reform in Newfoundland and Labrador may serve as a systems laboratory for reform efforts in other jurisdictions.
Innovation Education is a major research project presented to OCAD University in partial fulfillment of the requirements for the degree of Master of Design in Strategic Foresight & Innovation. This work was supported by the Social Sciences and Humanities Research Council of Canada.
