Evaluation of co-production 

Blog 1 of 4 in the Insights From Researchers on Co-production blog series

EPIC blogs usually focus on parent, teacher and clinician readers. This series of blogs are the findings from a workshop for researchers entitled Overcoming barriers to co-production with children and adults with neurodevelopmental disabilities which took place during the Neurodevelopmental Disorders Annual Seminar 2022 held in Edinburgh. Following on from this series we will have the Insights from a Lived experience on Co-production blog series which will focus on getting involved in decision making about research from a parent perspective.

During the workshop, we split into groups to work on challenges associated with co-production. Attendees self-selected to one of six groups: 1) Recruitment and Partnership building, 2) Payment, Expectations, and Trust, 3) Facilitation of co-production with adults, 4) Facilitation of co-production with children and adolescents, 5) Evaluation or 6) Dissemination and public engagement. 

No one went to the Evaluation group. And so, I decided to write this blog first as there is seemingly a lack of confidence in this area with researchers. The other reason Evaluation is the first entry in this blog series, is because evaluation is something you need to think about at the beginning. 

Evaluation of co-production can be thought of in two parts: evaluating the process of co-production and evaluating the impact of the co-production on the project. Evaluating the process can be formative and summative (den Houting, 2021). Formative evaluation is evaluating the design for a given purpose and context. Collecting this formative feedback could take the form of for example, 1) checking with your project patient and public involvement co-ordinator if the structure of the workshop you’ve designed makes sense or if the questions are clear, 2) checking the activity with 1 or 2 children before doing it with ten, 3) asking a colleague who has conducted similar work if they think the design will work well. If co-production for a project is long-term e.g. over a year, formative evaluation could be collecting feedback from your  group about how the co-production process could be improved e.g. shorter meetings, sending the agenda in advance, switching to texting instead of email for communication. Recording who came to a meeting, their ages, diagnoses, ethnicities, or other relevant information like membership in charities/organisations or research experience is important for reflecting on whether other people may need to be recruited. It can be helpful if an indicator of understanding amongst the group is collected e.g. asking members to summarise the main points of the day at the end of a workshop – this point is more relevant to working with children. 

Summative evaluation of the co-production process is important as it holds the researchers accountable and provides impetus for the research team to reflect and change their processes for the next project. This can involve asking the co-production and the research team about how the co-production went. den Houting (2021) provides some examples of questionnaires and interview questions to evaluate the process of co-production. 

Finally, recording the impact of the co-production on the project. Evaluating impact of the co-production on the project can be supported by a framework like the Public Involvement Impact Assessment Framework (PiiAF) (Popay & Collins, 2017). The framework can also be used as a means of setting expectations with the co-production group at the start. When co-production is carried out well it is embedded in the research design process and gradual; it can be tricky to look back and remember exactly what the co-production group contributed to the project. The researcher might think a year later “was adding the a measure of well-being to the project my idea or their idea? Or did it arise in a discussion we had…I think it was my idea…but I think I had that idea as a result of something the co-production group said..oh I can’t remember!”. Developing a plan at the beginning of co-production to record the feedback avoids this problem. This can take the form of recording notes during each meeting and then under the notes writing a reflective piece about which parts will inform the project. When it is time to write your paper, you will be able to succinctly write how involvement impacted the project. Do not forget to record praise for the project as well. 

Patient and public involvement is a part of the research process like funding, design, data collection, analysis. I agree with Edelman and Barron (2016) when they say patient and public involvement (PPI) should not be evaluated like a sort of complex intervention on the project. They make the interesting point that PPI is a ‘contribution of expertise and advocacy, equitable to the contribution of clinicians, statisticians or others.’ Would you test the contribution of a statistician on a project? With the help of a framework like the PiiAF, thinking about the aims, desired outcomes, and indicators of those outcomes is useful for 1) designing the structure of your co-production work (how many sessions will be needed, what we need to cover in them), 2) setting expectations for the co-production group and the rest of the research team, and 3) looking ahead to decide how you will record feedback and how it informs the project. But testing co-production, like you would other variables in your research project, is not necessary in most cases. 

References and more resources: 

Popay, J., Collins, M., & with the PiiAF Study Group. (2014). The public involvement impact assessment framework guidance. Universities of Lancaster, Liverpool and Exeter https://piiaf.org.uk (full and short version available)

Collins, M., Long, R., Page, A., Popay, J., & Lobban, F. (2018). Using the Public Involvement Impact Assessment Framework to assess the impact of public involvement in a mental health research context: A reflective case study. Health expectations : an international journal of public participation in health care and health policy, 21(6), 950–963. https://doi.org/10.1111/hex.12688

den Houting, J. (2021). Participatory and Inclusive Autism Research Practice Guides. Brisbane. AutismCRC.https://www.autismcrc.com.au/access/sites/default/files/resources/Participatory_and_Inclusive_Autism_Research_Practice_Guides.pdf

Edelman, N., & Barron, D. (2016). Evaluation of public involvement in research: time for a major re-think?. Journal of health services research & policy, 21(3), 209–211. https://doi.org/10.1177/1355819615612510 

This workshop was funded by the University of Edinburgh College of Medicine and Veterinary Medicine Public Engagement Seed Fund through the Institutional Strategic Support Fund.