Notes on measuring what matters

The most useful sentence in Fieldstone's 2022 external review is on page 31: "The organisation's retention reporting is not wrong so much as it is answering a question the organisation did not intend to ask."

We had been publishing teacher retention as a headline figure. We believed it described how many of our fellows were still teaching five years after joining. It described how many were still teaching three years after joining, in two of our three learning regions, because the underlying cohorts had been assembled that way in 2017 and nobody had revisited the definition when the fellowship was extended.

We withdrew the 2021 figures, restated the series back to 2016, and published both the original and the restated numbers side by side, with the difference highlighted. The restated retention figure for the Vega Alta Ridge was 34% where we had published 61%.

The restatement cost us two institutional funders. I mention that not as a grievance but because organisations considering the same decision should know the price is real and should make it anyway.

Every figure Fieldstone publishes now carries three things: the question it answers, the population it was calculated over, and the date the definition was last changed. It is less quotable. It is considerably harder to be wrong in an interesting way.

We can count children enrolled, teachers retained, systems functioning, and euros held in member reserves. We cannot measure the thing we actually claim to be doing, which is whether a community's own institutions are stronger than they would have been had we never arrived.

The closest proxy we have found is the proportion of a partner institution's operating costs met from its own members rather than from us. It is a crude instrument. It is honest about being one, which is more than the previous headline figure managed.