Researchers' experiences of patient & public involvement

Measuring impact of involvement

We asked researchers what they thought about trying to measure the impact of involvement. This covered three main areas: what people knew about the current evidence base for involving people; whether they felt more measurement of impact was necessary; and if so how this might be done.

The current evidence base
Generally researchers agreed the evidence base for involvement was not very strong and needed improving. Some said they did not really know enough about the evidence base to comment.
Our sample included some researchers with a particular research interest in patient and public involvement who had more detailed knowledge of the available evidence or lack of it. Some mentioned recent studies which are attempting to improve the reporting and measurement of impact, though it is still very early days to see how this will work out in practice.
Do we need to measure impact?
Views on the need to find ways to measure impact were mixed. One view was that this is vital to convince funders and colleagues it is worth doing, and to make sure we understand how to do it better. Bernadette felt she herself needed better evidence. Sarah, Andy, Jo and Pam all suggested that PPI partners themselves might also want to know they are making a difference. Hayley also pointed out that people will understand not everything they say can be used.
But it could also be argued measuring impact is inappropriate or unnecessary. Clinical and quantitative researchers such as Carl, Sergio and John, who recognised people might assume they wanted randomised controlled trial (RCT) evidence, actually felt involvement was just a matter of common sense and something you would do anyway, regardless of evidence. Sergio commented: “Sometimes we do not really need data to figure out whether something is worth its while or not… I think that there are enough logical arguments there to allow us to sustain that it's a positive thing.” Jim and Ann wondered why we single out patients or members of the public when we don’t evaluate the impact of any other member of the research team.
While people were conscious that there was a lot of pressure to produce evidence of impact, they also felt a contrasting pressure not to question the value of PPI, and a sense that any critique or evidence that it did not ‘work’ or even caused harm would be unwelcome. Felix insisted it was important to be honest about negative impacts as well as positive. Ann commented that criticism is rare and that “There are people desperately trying to prove the impact because they want to prove that it has a positive impact.” Alison added: “I think there is a touch of Emperor's new clothes going on, that this is so valuable, important and useful….People don’t really question it now.”
There was also a fear that some people would remain sceptical no matter what evidence was produced, unless they personally experienced the impact of involvement. As one person commented, “I work a lot with surgeons and surgeons are very firm in their opinion as to what is right... I'm not convinced that any evidence presented to them would make a huge difference. It would take a research project that they'd tried to do and failed to do to be then inputted by PPI to succeed to make a difference to them, I think, and that is a big challenge… I can be as enthusiastic as anything, but I'm not going to convince a true sceptic.” Others echoed the idea that a positive and enriching experience of involvement at first-hand was an effective route to help people see its value.

How to measure impact
Regardless of whether researchers felt there was a need for better evidence of impact, there was recognition that actually getting such evidence is not easy. The lack of agreement about what we mean by either ‘involvement’ or ‘impact’ remains a problem for trying to come up with suitable measures. Felix and Andy suggested it was important to clarify what impacts you were expecting at the start of the project, and Jo suggested involving PPI advisers themselves in defining what impacts might be reasonable. Chris recommended keeping track of possible impacts during the study rather than trying to do it retrospectively. As Sarah A commented, ‘no one really measures or reports PPI as it goes along, it just kind of happens… So if you’re not reporting it how can you ever demonstrate the impact it has?’
There was disagreement whether it was better to stick to measuring processes of involvement (such as number and diversity of people involved) rather than trying to show a difference to the outcome or progress of the research itself. Pam suggested formative evaluation to help improve current PPI practice might be more important than trying to prove whether it ‘works’ or not. Further difficulties were deciding the time point at which to measure, and tracing impact through what might be many years of an evolving research idea. Sometimes involvement may contribute to a change in culture that assists and enables other changes which are never directly attributed – or attributable – to that initial involvement.
There was little support for the idea that conducting RCTs to establish the effectiveness of PPI was realistic or desirable, although a few people felt it was possible, and there are a growing number of examples. Pam argued that ‘PPI is not a thing… it’s a set of complex relationships’ and that developing a defined and measurable intervention was always going to be challenging. Both Jim and Suzanne suggested it has to be thought of as a ‘complex intervention’ with lots of interconnecting components.
Like Pam, most people thought qualitative approaches to trying to describe impact were likely to be more practical, but this could take a lot of time, reflection and resources. Retrospective analysis of funding and ethics applications or published studies was suggested, to compare levels of PPI and various outcomes (such as funding success, speed of approval or recruitment rates). But again this relies on a clear agreement on what ‘PPI’ means and good reporting of PPI activity. 

Pam also drew attention to the fact that the impact debate tends to assume that to have value involvement must change things, whereas sometimes patients may agree with researchers or validate the research design, and this is also useful. John  made a similar point: ‘I think that endorsement is a good thing to have… You say “I got this idea for some research” and the patient says, “That’s a great idea”.’


Please use the form below to tell us what you think of the site. We’d love to hear about how we’ve helped you, how we could improve or if you have found something that’s broken on the site. We are a small team but will try to reply as quickly as possible.

Please note that we are unable to accept article submissions or offer medical advice. If you are affected by any of the issues covered on this website and need to talk to someone in confidence, please contact The Samaritans or your Doctor.

Make a Donation to

Find out more about how you can help us.

Send to a friend

Simply fill out this form and we'll send them an email