- > In the context of CLDs, the rationale behind using these metrics, which rely on a factor’s position on short causal chains, is that one could assume that causal power gradually diminishes while going down a causal chain8, making interven- tions on the factors that are involved in short causal chains the most likely to shift system-level behaviour. That is, the hypothesis is that the shorter the causal chain between a factor X and a factor Z, the more of the causal power from a change in X is left when we get to Z—and, thus, the more causal power is exercised on Z by X.
Page 3
- > first, we assess whether betweenness- and closeness centrality identify the same leverage points based on CLDs that represent the same system but differ in inferred causal structure. In other words, we assess whether the metrics provide reliable results. Second, we consider conflicts between the assumptions underlying betweenness- and closeness centrality and CLDs to understand whether the current practice of applying these metrics to CLDs is theoretically sound.
Page 3
- > we compute the metrics for five CLDs (Fig. 1)—a baseline CLD and four alternative versions of that baseline CLD—that differ in causal struc- ture but represent the same system.
Page 3
- > the five CLDs differ in causal structure due to modelling choices that can be made one way or another depend- ing on the research question as well as the modeller(s)20,21, with each choice being justifiable.
Page 5
- > The ranking of the factors from least to most important as determined according to betweenness centrality, betweenness centrality*, and closeness centrality is thus inconsistent over the CLDs, while closeness centrality* is mostly consistent. Betweenness- and closeness centrality, with the exception of closeness centrality*, do not provide reliable results: they identify different leverage points based on CLDs that represent the same system. they should not be expected to be to produce reliable results, as the bottlenecks suggested by analysis of each system model are the bottlenecks in the given model.
Page 7
- > As CLDs are graphical representations of mental models of the important feedbacks believed to be responsi- ble for a problem17, what flows through a CLD is best described as ‘causality’ or ‘causal impact’, where a change in a factor propagates through all allowed paths with different intensities and, at a given timescale, may even reinforce or suppress itself. Hence, it may be at best premature and at worst incorrect to use metrics that imply the assumption that what flows through a CLD relies on knowing and taking the shortest path. However, this is the implicit assumption being made when applying betweenness- and closeness centrality to CLDs. As a result, the factors that are identified as mediators by betweenness centrality and as spreaders of causal power by close- ness centrality may not actually have these roles in the type of flow process that is represented in CLDs and may therefore not qualify as leverage points.
Page 7
- > To use the metrics there should be node distinctiveness, where nodes are causally related and independent rather than constitutively related and overlapping68. Why?
- Find the explanation for this idea
Page 8
- > That is, the metrics do not provide us with any information about how interventions on multiple factors in the CLD may interact—for example about whether two interventions might function in synergy, where the effect of the interventions combined is greater than the sum of the effects of each intervention separately. This issue is not resolved by simultaneously intervening upon several factors identified as leverage points by the metrics, such as the three highest ranking factors, as applying the metrics means assuming that interventions occur in only one leverage point at a time. It disappointing that the authors do not include any discussion of systemic strategy here
Page 8
- > While the directed variant of closeness centrality in this case unexpectedly showed consistency over the different versions of the CLD, the theoretical conflicts uncovered suggest that this might be coincidental. Of course it is coincidental, as these models are tiny
Page 8
- > The key finding is that network analysis metrics are highly sensitive to changes made to a CLD. This means that, when network analysis is used, different leverage points may be identified due to modelling choices that can be made one way or another, depending on the research question as well as the modeller(s), and that do not alter the system that is represented by the causal structure.
Page 8
- > Even if betweenness- and closeness centrality were to provide reliable results, the six conflicts recognised give us reason to believe that the metrics may still leave us with ‘the wrong answer’ in terms of leverage points.
Page 8
- > For social networks and psychological networks, scrutiny of these metrics has even gone a step further by testing whether real-world interventions on the nodes that the metrics identified as important indeed had a large effect on system-level behaviour, which also did not lead to the anticipated results61.
- Look into this study
Page 9
- > To facilitate such reflection, if network analysis metrics are used, as a minimum requirement, the equations used to compute the metrics should be given—which is currently not always the case.
Page 9
- > Eigenvector centrality appears to have less restrictive assumptions65 and was shown to correlate with causal influence in directed acyclic graphs59. A step in the right direction could also be the addition of edge weights, which allow a modeller to indicate a larger or faster causal effect with higher weights19,58,61,77. Incorporating edge weights in CLDs could result in network analysis metrics being more consistent across CLDs that represent the same system and has the potential to mitigate problems with node exchangeability. Polarity could be accounted for as well with negative edge weights58,77. Methods to identify an optimal set of important nodes rather than a single important node have also been developed78.
Page 9
- > the definition of the term ‘leverage point’ as it currently tends to be used in the literature relating to the development and analysis of CLDs may indeed be too narrow82.