Measurement
Part of Women founder funding measurement: stages, metrics and impact limits
Women founder funding attribution methods: seven Test and Learn features for testing assumptions first
Women founder funding attribution methods start with small pilots: the seven Test and Learn features, four risky assumptions to test and a three-month plan.
Most attribution plans for a funding service for women founders begin with the method: matching, comparison groups, contribution claims. An earlier step matters more. Find out which assumptions could sink the programme, and test those first.
What to take away
- Attribution is only worth the cost once the delivery assumptions behind the programme have held up in practice.
- Test the assumptions with the least evidence and the greatest risk before spending on a full evaluation.
- A pilot is not a substitute for a counterfactual design. It prepares the ground for one.
- Match the claim to the design. A pilot supports learning, a matched or cut-off design supports an effect estimate, and contribution analysis supports a reasoned contribution claim.
Why test before you attribute
Assumptions made in the design stage are often only tested years later, by which point it can be extremely costly to discover they were wrong. That sentence comes from HM Treasury's Test and Learn annex to the Magenta Book.
For a funding service the risky assumptions are easy to list. Will eligible founders apply? Will advisers deliver the same service in every region? Will founders who receive finance use it as planned? Each can fail long before a business outcome is measurable.
Test and Learn is most valuable when a problem is complex or uncertain, assumptions are untested and teams need rapid feedback. It is less valuable when the problem and solution are well understood and delivery follows a fully planned approach.
Seven features applied to a funding service
Test and Learn has seven features. The table reads each one against a women founder funding service. The right-hand column is illustrative, not prescribed.
| Feature | In a funding service |
|---|---|
| Multidisciplinary delivery | Policy, delivery, data and adviser staff plan the test together |
| Shared, measurable outcomes | Partners agree one outcome, such as completed applications from eligible founders |
| Evidence-based understanding | Application records, adviser notes and founder interviews inform the problem |
| Tests risky assumptions | Start with the step where least is known, such as eligibility checks |
| Iterative | Feedback loops combine numbers and founder comments |
| Scaling and local adaptation | Check what works everywhere and what needs a regional change |
| Transparent and open | Decisions and progress are visible to partners |
Partners should agree and unify around a common outcome, which the testing is then designed to support. Without that agreement, different partners count different successes.
What to test first
Issues that are critical to achieving key outcomes but have weaker evidence or greater uncertainty should be prioritised. These assumptions often relate to behaviour change, delivery feasibility, unintended consequences or value for money.
A short list for a women founder funding service:
- Behaviour changedo founders act on the information they receive?
- Delivery feasibilitycan partners process applications within the promised time?
- Unintended consequencesdoes a women-focused route deter or confuse eligible mixed teams?
- Value for moneydo advice costs fall on founders or on the service?
Learning is generated through feedback loops that combine qualitative and quantitative evidence, often using multiple methods in parallel. Plan the loop before the pilot starts, so each question has an owner and a date.
From pilot to a causal claim
Test and Learn is not a type of evaluation. It integrates research and evaluation thinking throughout the policy cycle.
Used well, it builds the evidence and confidence needed to progress to programme-level evaluation, including counterfactual impact methods where feasible. It strengthens evaluation and does not replace it.
Evaluation methods are set out in the Treasury's Magenta Book Annex A for that stage. Propensity score matching can be considered when randomised controlled trials are either not feasible or not desirable. Contribution analysis does not provide definitive proof that the intervention has had a causal effect; it gives an evidenced logical line of reasoning.
Choose from these according to how the programme selected people, and write the choice down before results arrive. For the outcomes to measure, see key metrics for women founder funding, and for outside comparisons benchmark research.
Example: a three-month pilot
The following plan is illustrative. It shows the order of work rather than recommended numbers.
- Month 1agree one measurable outcome with partners and name the riskiest assumption.
- Month 2run a small, time-limited pilot of the step that carries the assumption, with one feedback loop.
- Month 3review findings, change the design where the assumption failed, and record why.
- End of month 3decide whether to scale, adapt locally or stop, and document the decision.
- Afterwardschoose the evaluation design for the full programme and set the comparison group before launch.
Report failures with the findings. A failed assumption found in the pilot is cheaper than one found years later. The guide to measurement mistakes lists errors to avoid when reporting, and a reporting dashboard can keep dates and denominators in one place.
Report a bounded conclusion
Match the verb to the evidence.
A pilot allows "we learned that"; a matched or cut-off design allows "is estimated to have"; a contribution analysis allows "the evidence supports a contribution". Make processes, decisions and progress visible to partners as they happen. For the wider framework see measurement and reporting across the funding journey, and for cost questions the method in return on investment.
Common questions
Does Test and Learn replace an evaluation?
No. It strengthens evaluation but does not replace it. A causal claim needs a counterfactual design.
When is Test and Learn a poor fit?
When the problem and solution are well understood and delivery follows a fixed plan, it is less valuable.
Who should own the pilot?
A multidisciplinary team that includes delivery staff, analysts and the partners who run the service.
