In Part Two of our Rough Guide to the Tenant Satisfaction Measures (TSMs), Denise Raine & Steve Smedley  take a closer look at the mandatory survey requirements that underpin the 12 TSMs that are derived from a perception survey (as set out in Annex 5 of the RSH documentation).

In Part One, we explored the finalised set of 22 TSMs and associated requirements as set out in Annex 4. Whilst Parts One and Two of this rough guide provide helpful summaries of the regulator’s requirements and should prove useful to staff and board members, those responsible for compliance should familiarise themselves with the regulatory documents noted in the next section.

You might also be interested in what we learnt from piloting the TSM perception metrics with a number of clients in this briefing.

TSM surveys must be compliant but there’s flexibility in how you achieve compliance

As noted in Part One, the key documents that set out the Regulator of Social Housing’s (RSH) TSM requirements are as follows:

• a new TSM standard – Annex 3: Tenant Satisfaction Measures Standard – from 1 April 2023
• 22 TSMs & detailed definitions – Annex 4: Tenant Satisfaction Measures – Technical requirements
• detailed survey requirements – Annex 5: Tenant Satisfaction Measures – Tenant survey requirements

In order to comply with the new TSM standard, Registered Providers (RPs) must adhere to the TSM definitions and calculations as set out in Annex 4 and the 12 perception measures must be generated in accordance with the survey methodology contained in Annex 5.

The RSH’s survey methodology aims for a consistent approach across providers, facilitating comparisons, whilst providing some flexibility given the sector’s diversity.

So, whilst the RSH has made specific methodological requirements mandatory to ensure robust results, it does not provide detailed guidance as to how to conduct the survey. In keeping with the RSH’s co-regulatory approach, it is for RPs themselves to decide how best to meet the RSH’s survey requirements in a proportionate and appropriate way, taking advice where necessary from organisations such as ourselves.

If you need help devising a compliant approach, contact: denise.raine@arap.co.uk

The 12 survey TSMs

The 12 perception measures generated by a survey are shown below:

Survey timings

Data used to generate tenant perception TSMs must be collected as part of a single integrated survey exercise. The fieldwork may happen at a single point in time, through a phased approach, or on a periodic rolling basis (eg quarterly or monthly). Transactional data, ie triggered by a recent interaction with the provider, should not be counted – the RSH wants a perception survey!

It’s up to smaller RPs (ie those with less than 1000 combined socially rented and low cost home ownership stock) whether they run the perception survey every year or every two years. Larger RPs must survey annually. All RPs must provide tenant perception TSM data in the first year of reporting (2023/24).

Survey question wording and response options

Perception surveys must use the precise wording for questions and response options as set out in Table 2 of Annex 5 regardless of the collection method used (ie phone, written or face-to-face). Providers are free to use the survey collection method(s) most appropriate to them but must be able to demonstrate the rationale for their choice.

The questionnaire

RPs must ensure the purpose of the perception survey is explicitly communicated to the tenant at the start of each survey conducted, ie they must state that the survey will be used to calculate annual TSMs to be published by the landlord.

Providers can include other questions in the same survey questionnaire, eg to better understand performance or tenant characteristics, but they need to be mindful of survey fatigue. Indeed, Acuity’s principal concern with the RSH’s requirements is tenant fatigue (see What do we think? below).

Who gets surveyed?

RSH requires the 12 perception survey questions to be asked (and reported separately) for:

• Low-Cost Rental Accommodation (LCRA) stock where the landlord has more than 1,000 such units – includes general needs (GN), supported housing, Housing for Older People (HfOP), intermediate rent and temporary social housing
• Low-Cost Housing Ownership (LCHO) stock where the landlord has more than 1,000 such units – includes shared ownership homes (which have not been fully staircased)

Respective definitions of the above social housing draw on sections 69 and 70 of the Housing and Regeneration Act 2008.

However, if a RP has less than 1000 LCRA stock and less than 1000 LCHO stock, it can choose whether to report the tenant perception measures for LCRA (only), LCHO (only), LCRA and LCHO separately, or LCRA and LCHO combined. For many Acuity smaller housing association benchmarking members, it begs the question of why you would bother to report the LCHO at all given the low sample size and potential skewing effect of a combined LCRA/LCHO figure (LCHO customers would reduce a combined score as they are generally more dissatisfied!).

As noted in Part One, providers with more than 1000 LCRA and LCHO stock must report their results for all TSMs to the RSH as well as publish them to their tenants. Smaller RPs just report to their tenants.

Publication of summary of approach

When reporting TSM results to tenants, providers must publish a summary of the survey approach used. This must include:

• sample size (number of responses)
•  timing of survey
• collection method(s)
• sample method
• summary of the assessment of representativeness of the sample against the tenant population
• any weighting applied to generate the reported perception measures
• the role of any named external contractor(s) in collecting, generating, or validating the reported perception measures
• the number of tenant households that have not been included in the sample due to exceptional circumstances with a broad rationale for their removal
• reasons for any failure to meet the required sample size requirements
• type and amount of any incentives offered to tenants to encourage survey completion
• any other methodological issues likely to have a material impact on the tenant perception measures reported

This summary must be proportionate to the complexity of the sampling, eg the level of detail required from a large provider applying stratified sampling and weighting of responses is significantly greater than that required from a small provider which might employ a simple census approach.

On this last point, the RSH is aware that smaller providers are likely to have practical difficulties applying its sampling requirements and that a census rather than a sample may be required. If a smaller provider conducts a census survey and still falls short of the sample size responses required, the RSH will consider it to have met its requirements.

Annex 5 provides detailed sampling requirements to meet minimum levels of statistical accuracy as well as expectations on ensuring the sample is representative (ie that various client groups are reflected), and if not, how the results should be weighted.

Representativeness is an important issue as bias has the potential to significantly skew results. Again, smaller providers are able to take a more proportionate approach – they are only expected to do a high-level assessment of representativeness. Annex A of Annex 5 (yes, the annex has an annex) provides a useful summary of the expectations for smaller providers.

Related to representativeness is the need to take reasonable steps to ensure that barriers to participating in the survey are overcome such as those experienced by tenants who fall into one or more of the protected characteristics. Those RPs with high numbers of supported housing will be familiar with this challenge.

Where there are material year-on-year changes in the provider’s approach to the survey, eg survey collection method, a summary of these changes must be included with the reason for the change. Similarly, any analysis of year-on-year changes in performance must refer to any material changes in survey methodology that are likely to have significantly affected satisfaction scores.

What do we think?

Acuity has decades of experience in developing and conducting satisfaction surveys and provided comprehensive insight to the RSH’s development of the TSMs. Our thoughts in a nutshell:

A return to collecting national performance data reinstates an important source of transparency and accountability that was lost to tenants a decade ago. However, a commitment to data quality must underpin the entire exercise. All providers need to be thinking about how they ensure data quality now. This isn’t just about compliance; it’s about reputation.

The requirements are a big ask of tenants and landlords. There are 15 questions, including pre-qualifiers (and some of them aren’t great). This limits the capacity for you to ask the questions you want to tease out service issues, tenant preferences, etc, ie the vital intelligence that actually helps you provide better services.

Furthermore, not including a key customer service metric is a missed opportunity. For some time, in the commercial sector, ‘easy to deal with’ has been a major indicator of an organisation’s standing with their customers. We have asked how easy tenants find it to deal with their landlords in many surveys over the past few years and come to the conclusion it is the single biggest driver of overall satisfaction. Think about that as you gear up for consumer regulation!

Then there’s the issue of tenant fatigue, particularly for those landlords with 1000 – 3000 stock. At the lower end of this range, you will need to sample over a quarter of your tenants annually whilst at the higher range, just under a fifth. Due to the practicalities of generating enough responses, it’s likely some tenants will be repeatedly surveyed over a number of years leading to irritation and fatigue which will impact results. For those providers with less than 1000 stock contemplating an annual survey, fatigue will be a significant challenge.

Acuity’s Satisfaction Survey team is on hand to help you with your surveys whether you want a specific task undertaking or to outsource it all from conception through to surveying and reporting. Contact: denise.raine@arap.co.uk