MERLIN restoration case studies assessment
Scoring of the monitoring results
Scoring of the monitoring results was carried out in two steps: firstly, the results of the individual indicators were scored, and next these scores were transformed to an aggregated score for the 13 EGD policy criteria.
Regarding the indicators a score for the response observed in the reported data (i.e., positive, negative, or no change) and a score for the strength of evidence supporting the assessment (Box 1). These scores provided consistency across assessments. The indicator scores were then compiled and summarized. This initial scoring underwent expert peer review by the Case Study Leads, who provided revisions along with supporting rationale.
Box 1: “Traffic light system” for assessing the impact of restoration measures
Negative impact:
– Clear evidence of deterioration, supported by consistent numerical data or well-defined negative narratives.
– Indications of deterioration, but data are somewhat ambiguous (e.g. negative trend with fluctuations) or narratives are vaguely negative.
No impact/irrelevant impact:
– No detectable change, with stable numerical data and explicit narratives confirming the absence of change.
– No significant change, but this is conveyed through less precise or more cautious narrative wording (e.g. "no significant impact").
Positive impact:
– Clear evidence of improvement, supported by consistent numerical data or well-defined positive narratives.
– Indications of improvement, but data are somewhat ambiguous (e.g. positive trend with fluctuations) or narratives are vaguely positive.
– Effects cannot be assessed yet, either due to lack of data or because confirmation is currently not possible.
– Effects remain uncertain, with narratives that are vague or inconclusive.
No data: NA – Indicator is not relevant to that CS or data was not reported for other reasons.
A standardized methodology was developed to enable Criterion Assessment Leads to aggregate individual indicator scores into an overall criterion score, as outlined in Box 2. Case Study Reporting Leads were again consulted on the aggregated scoring results. In cases where a decision was made to weight the indicator scores (i.e., following Option 2 in Box 2), this was justified with a clear rationale.
Box 2: Indicators-to-criterion aggregation
N = 1 indicator
→ The assessment of the indicator is directly taken as the assessment of the criterion.
N > 1 indicators
- If all indicators yield the same result
→ The criterion assessment is the same as the shared indicator assessment.
- If indicators yield different results
Option 1: All indicators are weighted equally.
→ Calculate a weighted average, assigning numerical scores to each assessment, using these scores:
- : +1.0;
: +0.5;
: -1.0;
: -0.5;
: 0;
: 0;
: not counting (=excluded from calculation);
: not counting (=excluded from calculation)
- The resulting average is then classified as follows:
>0.5 to 1: ;
>0 to 0.5: ;
0: Special case (see below) <0 to -0.5:;
<-0.5 to -1:
- Special case. Individual indicators all yellow = or
(depending on majority of
or ,
where 50:50 → ;
where 50:50 red:green transparent
OR 50:50 red:green full:
or 
)
Option 2: Indicators are weighted differently, depending on:
- Their relevance to the type of restoration measure.
- Their alignment with the intended impact (as defined in the ToC).
- Any other justified expert judgment.
→ Exclude indicators that are irrelevant or inappropriate for the specific case. Then proceed as in Option 1 above.