www.publicinvolvement.org.uk

One of the reasons we work with Dan is that he shares a past with Colin Adamson - both worked for a customer satisfaction measurement research and consultancy agency called TARP Europe. We therefore share a common and common sensical outlook on surveys - Dan for example helped us with our survey on complaint handling. So naturally we wanted to know what he thought of the Picker pronouncements and he tells us below.

Dan Wardle writes...

Picker Institute published their report on Mixed Mode Surveys and their potential use in the NHS acute patient survey programme in June (opens in Adobe PDF).

To my mind, it was very disappointing - heavy on academic socio/cognitive type issues or methodology with the disappointing and wholly unadventurous conclusion

"principally mailed methodology with paper questionnaires as the sole means of data collection to be the only viable approach to conducting the main national surveys."

The report finds that internet surveys are not currently viable, partly because of problems collecting valid email addresses for patients (a fair point), and yet there is a desire to pilot the use of SMS reminders in the near future when finding a solution to the main problem ie collecting email addresses and piloting internet surveys would be a better move?

Using more than One Mode of Survey

The report's objective was to look at using more than one mode of survey to improve response rates and the cost benefits. If you have an established survey programme using paper and mail then yes - literally transferring it online will present a hurdle as soon as you ask "and where are the email addresses?". Considering the amount of money pumped into NHS patients surveys and Picker's established reputation and track record with NHS surveys, you'd have thought the authors could come up with a creative strategy than the old research company gambit (yes we plead guilty too) of "more research is needed".

The major obstacle the NHS has for conducting patient surveys is ensuring a robust, representative sample. A mixed mode strategy has to be the way to go, those groups that do not have internet access should still be reached by the mailed questionnaire, but a text message isn't going to encourage them to fill it in either. Nevertheless where you have a large proportion of your customer base using a technology that can deliver substantial benefits in terms of speed of response, accuracy, efficiency and ultimately costs - I mean of course the Internet - this has to be an option and one that can be used now without waiting for more research.

How do you collect email addresses?

To be fair, it's not for Picker to get hospital departments to change their practice in recording patients' email address and contact details, but this should be HIGHLIGHTED WITH A BIG FAT MARKER. They (the authors) are unaware of any acute trusts that routinely record emails, yet it is a simple enough task (although the importance of recording the email address accurately cannot be understated) and offers such potential benefits.

Unfortunately, the report doesn't explain how the mailing samples are currently put together, but this is where Picker and the Healthcare Commission should be focussing on whether they can make the survey programme internet ready as of now. And where the report acknowledges that other industries face the same problem (p14), it should also acknowledge that they have found solutions.
 
Comparative Costs - Internet Survey Vs Mail Questionnaire

There are also no costs* presented on any pages of the report (response rates or volumes mailed for the NHS surveys are not revealed either). So let us give you an idea.

* Costs are discussed in under 'Cost Implications' without actually presenting any hard numbers.

What might a survey cost?

Printing and mailing 4,000 x 4 page questionnaires + reply paid envelope; 2,000 responses returned and entered (50% response rate) might start from £6,000. (Because we are concerned with a comparison between two distribution channels/choices, we are just talking survey mechanics here and have not included assistance on questionnaire design etc, nor costs for management, mail-merge and printing/stuffing envelopes, Q.A. etc.)

Approximate cost for the online version of a 4 page questionnaire might start from £2,000.

The more people, the more money on stamps – Internet Costs do not change

If you only had to mail 2,000 questionnaires to the non-internet users and the rest were fielded online, likely savings here should still be £2,000 maybe £3,000. Obviously, these are small numbers (perhaps the size of a survey managed by a single trust?) but postage is not discounted the more you send - the cost for a national survey to 16,000 participants will increase proportionately. Compare this to the internet survey and the only increased overheads will be for email management and marginal costs for bigger capacity to collect increased number of records. Overcome the email address collection problem and this becomes very hard to overlook.

Mixed communication

Mixed mode surveys could also take the form of mailing a postcard or letter asking for the participant to complete the survey online. This incurs outbound printing and postage costs but zero costs for collecting responses. Or the same could be trialled via SMS messages...

Misplaced emphasis

Costs are important but not the whole story. Much was made of what resolution the participants' screens are - a non-issue if the survey's formatting is well designed - while important arguments such as the ability to collect verbatim comments were buried. Picker seemed content to raise these questions without an answer either from them or indeed from others. Their arguments may be music to the ears of those who want to do these large and important surveys using tried and tested methodologies costing big bucks, yet the costs are very small (in terms of million pound contracts) for piloting a couple of studies that could save the NHS not insignificant sums of money for years to come.

We would also like to see smaller NHS units being encouraged to do cost-effective and professional surveys and for them, the emailed survey to selected groups would be a productive route. The big survey that asks "how satisfied are you with the NHS this year?" produces tabloid headlines and questions in the House but no actionable data for people on the ground because they have no idea what experiences might create the levels of satisfaction reported.

It's normal for the big dinosaurs to be wary of evolution.

Dan Wardle is the Managing Director of Surveylab Limited.

M-A-C like small and dirty surveys that cover the ground quickly. This does not mean we believe that self-completion surveys can cover the whole universe of NHS users. Face to face or telephone interviews remain a necessity for those who would not or could not fill in a questionnaire whether on screen or on paper. But when that mode is the way to go, look at the email survey as cost-effective and quick. Try it first doing a survey to outpatients who have just had a consultation - get the email at the desk and send the survey the week after they attended. See if it works.

You can read more about M-A-C's services for conducting research here.