TayLP was a £2.8m Heritage Lottery Fund (now National Lottery Heritage Fund) scheme led by Perth and Kinross Heritage Trust with a range of partners and funders. Following a development phase, it was operational from 2014-2018. The project set out to enhance landscapes where the rivers Tay and Earn meet, by working on aspects of natural, built and cultural heritage in the region. Ketso came into use in evaluating its effectiveness.
By May Johnstone, Project Perspectives
TayLP was delivered through 28 projects, each fitting into one of four themes:
1. Conserving and restoring natural and built features
2. Participating in the landscapes heritage
3. Access and learning about the Inner Tay Estuary
4. Training in traditional skills
Project Perspectives had worked with TayLP staff in 2014 to help formulate and pilot their monitoring and evaluation framework, and in 2017 was again appointed to make various contributions to the Manager’s Final Evaluation, including a qualitative review with the Steering Group, TayLP staff, and community groups.
Ketso chosen for qualitative review
Ketso was chosen as the mechanism for the qualitative review because it would give all participants a say, and the opportunity to hear from each other and hence trigger further feedback. It was also expected that use of the Ketso spreadsheet would make it a cost-effective approach.
With some helpful advice from Joanne on using Ketso for feedback rather than generating ideas, four core questions were identified for the 6 Ketso discussions. Those questions were:
- What has TayLP achieved?
- What should we take forward?
- What were the problems?
- What have we learned?
The branches were labelled with the four themes of the project – plus two additional ones, Running the TayLP Scheme and Other Responses.
All six discussions were facilitated in November/December 2017 by Project Perspectives, and also attended by a TayLP staff member, who was able to answer detailed operational questions that arose. Discussions lasted between 80 and 100 minutes. Not surprisingly, the Steering Group and staff gave feedback across the range of projects, while the community groups were more focused on the project(s) they had been involved with.
The Ketso felt from each group was photographed, and the responses from all leaves were entered into a Ketso spreadsheet, which has functions for easy filtering by group / question / theme, and production of summary charts. Altogether, a total of 496 comments were written on leaves.
Responses to each of the 4 questions were later reviewed, grouped into categories, and summarised using Excel Pivot Tables.
From the analysis it was clear that the project was well run and promoted awareness, achieved a good network of contacts, restored orchards, Pictish stones and clay buildings and improved learning around the aspects of the projects. The 113 suggestions for what should be taken forward were also wide-ranging.
“Ketso was chosen as the mechanism for the qualitative review because it would give all participants: a say, and the opportunity to hear from each other and hence trigger further feedback. It was also expected that use of the Ketso spreadsheet would make it be a cost-effective approach.”
A few key insights emerging from the analytical exercise with Ketso are explored below:
- Key achievements identified by participants were skills and increased awareness of heritage.
- Key things noted to take forward were clearly dominated by access and learning.
- Key problems were various aspects of running the Scheme – particularly time and workload constraints for staff with too many projects to run, and difficulties with landowner permissions. Engaging communities and their capacity also proved challenging.
- Key learning mirrored the problems, focusing on running the Scheme, including various aspects of good practice, and set-up issues including the need to get landowner permissions before the start. There was also learning about unexpected intangible outcomes.
Overall results suggest that most participants identified a good range of achievements and things to take forward, while staff and the Steering Group were aware of both problems and learning – including good practice – associated with running the Scheme.
Project Perspectives feedback on Ketso
Ketso was used to good effect. Helpful and interesting insights from its use were shared with the Ketso team, which you can see here. These bullet points may come in useful for facilitators.
- Ideally branches would be identified with less overlap, as this led to more work in analysing results. In this case it demonstrated how much the 4 HLF Themes overlapped (they have since been replaced with 3 Themes and 9 Sub themes).
- In order to have more of a facilitating role, it might have been better for the facilitator to stand up rather than sit with the group, though this would have felt awkward in very small groups in someone’s home.
- Facilitating a Ketso discussion in someone’s house required some flexibility of approach e.g. when one group preferred not to use Ketso, and another wanted to discuss before writing their feedback on leaves.
- It was a challenge to get through the questions in the time available, and there wasn’t enough time to prioritise results such as key achievements or learning.
- The Scheme Manager reported that some participants would have liked to see the questions in advance so they could give them some thought.
- There’s a wealth of advice on the Ketso website, well worth reviewing in detail!
If you want to share your experiences of using Ketso for evaluation then please contact us. Please also take note of the last point here, that our website is well worth looking at for information and advice.