With an expectation that the TSA will shake-up the way resident satisfaction data is collected we have been closely following recent thinking and debate on what changes should be made. Have we been impressed? Well, not really.

While there must now be doubt that resident satisfaction is high on its priority list given the bigger issues the TSA faces since the change of government, we have been rather disappointed that so much of the debate has focused on methodological and statistical reliability. Herriot-Watt University’s report for London & Quadrant (A Measure You Can’t Trust) published earlier this year calls for ‘a tighter and less permissive framework’. Denise Raine, Director of Acuity (Inside Housing 19/2/10) stresses ‘just how important it is that the National Housing Federation’s STATUS methodology is adhered to’.

We are not for one moment suggesting that social housing providers should use anything less than robust methods in collecting and analysing satisfaction data or that they should shy away from achieving levels of statistical reliability at or at least close to the prescribed levels. But what does a +/- 5% margin of error at the 95% confidence level really mean? In broad terms it means that we can be sufficiently confident that the views expressed by survey respondents reflect the views of all residents, that the data is reliable. But confident about what?

Let’s look at a couple of examples from satisfaction surveys we have recently carried out on behalf of social landlords. When you start to get beneath the figures and into challenging discussion sometimes things are very different from what they first appear.

In the first example, 70% of survey respondents expressed a positive view about the overall quality of their home and 69% of respondents expressed a positive view about the general condition of the property in which they live. Our client was aghast that they could be pretty confident that these figures reflected the views of all residents, that they could reliably report a substantial fall in performance on these two measures when compared to survey results over the previous decade. Clearly they had fallen short on maintaining the quality and condition of their homes.

Why were they so upset? Because over the same ten year period and in response to residents’ expressed priorities they had invested millions into improving the quality and condition of their homes. Board members agreed more needed to be done to understand residents’ perceptions and expectations to make sense of the satisfaction levels expressed.

In the second example, 66% of respondents expressed a positive view about the repairs and maintenance service. Not a great result but similar to previous survey results and not dissimilar to peer-organisation comparisons. They could reliably report that around two-thirds of all residents were satisfied with the repairs service. When we drilled down into the specifics of residents’ experiences of this service a different picture emerged:

Clearly, residents were more satisfied with some aspects of the service than the headline result indicated.

Next we broke the findings down in different ways to look for any patterns in satisfaction.

When we looked at the difference between the
in-house repairs service and the service provided by external contractors something else stood out.

Is it at all useful to ‘reliably’ report that two-thirds of residents are satisfied with the repairs service?

Actually, residents are much more satisfied with most aspects of the repairs service but are less satisfied with specific aspects of their experience (e.g. the standard of work and completion times).

Residents are very satisfied with the in-house service and much less satisfied with the service provided by external contractors.

Oh, and by the way, this latter comparison isn’t statistically reliable. Despite this would you want to have a word with your external contractors?

We would love to hear your views on resident satisfaction and the way ahead for STATUS. You can share them on our blog.