Participation and prevalence research

Executive summary - participation and prevalence research

From December 2020-February 2021, we consulted on proposals to change the research methodology we use to collect adult gambling participation and problem gambling prevalence statistics. This is a data collection duty set by the Gambling Act 2005 and the data is also official statistics.

The consultation offset a limited scope related to this duty, with a focus on core participation and prevalence measurement. The aim is to provide the strongest possible foundation which we, and other interested parties can base further work on. We recognise that deficiencies currently exist in the wider system for measuring gambling harms and will continue our commitment to play our part in addressing these, through a combination of our own wider research programme and external research by partner organisations.

Our consultation outlined proposals to improve the quality, robustness and timeliness of our participation and prevalence official statistics. Specifically, the proposals sought to:

• develop a single, gold standard population survey for the whole of Great Britain
• consolidate current surveys into one population survey
• review and refresh the gambling activities included in the participation questions
• improve the frequency and turnaround time of the survey data
• explore more future proof data collection methods
• implement a new methodology from 2022, subject to a satisfactory pilot.

The consultation outlined the aim to develop a single, high quality methodology to measure participation and prevalence which will be more efficient, cost effective and timely, helping us to respond quickly to emerging consumer issues. We believe that a new approach will enable us to set the standard for authoritative research into gambling.

We received 62 valid responses to the consultation which ran for 10 weeks. This document summarises the responses received and explains our position on each proposal.

Introduction - participation and prevalence research

The Gambling Commission

The Gambling Commission exists to safeguard consumers and the wider public by ensuring that gambling is fair and safe. We place consumers at the heart of regulation and maintain the integrity of the gambling industry.

Our work is underpinned by two main pieces of legislation:

• the Gambling Act 2005 (opens in a new tab) which sets the framework for the regulation of gambling in Britain
• the National Lottery etc. Act 1993 which sets out the framework within which we regulate the National Lottery.

Under section 26 of the Gambling Act 2005 (opens in a new tab), the Commission has responsibility for collecting and disseminating information relating to the extent and impact of gambling in Great Britain. In order to do this, we collect gambling participation and problem gambling prevalence data via surveys of adults in Great Britain and the data is published as official statistics – meaning they are produced in accordance with the standards set out by the Government Statistical Service in the Code of Practice for Statistics.

We are also responsible for a wider programme of both quantitative and qualitative research which does not constitute core official statistics, but which helps us to build a better understanding of consumer gambling behaviour, attitudes and harms. Although some of this work is outside the scope of this consultation, we remain committed to driving this forward.

Consultation proposals and background

The Commission is ambitious about improving the quality, robustness and timeliness of our statistics. Timely, robust measurement of consumer gambling behaviour is critical to identify trends to help prioritise focus (for example take-up of new gambling and gambling-style products), to measure levels of risk and harm to consumers (via problem gambling screens) and to monitor the impact of policy changes. We therefore set out a commitment in our 2020/21 Business Plan to ‘review our approach to measuring participation and prevalence and publish conclusions’.

Our intention is to develop a single, high quality methodology to measure participation and prevalence which will be more efficient, cost effective and timely, helping to better inform us of the impact of policy changes and provide an ongoing foundation for evidence-based decision making. We believe that a new approach will enable us to set the standard for authoritative research into gambling.

Recognising gambling as a public health issue, there are a number of important areas which are linked to the methodology review but are considered out of scope of this consultation:

Gambling related harms:

Our National Strategy to Reduce Gambling Harms sets out the aim to identify a robust means of measuring harm and we are committed to doing what we can (with the resources we have available) to build and contribute to this evidence base. We have started a pilot of new survey questions on harm. As this work is currently in progress, the specific means of measuring harm is not within the consultation scope.

Longitudinal research:

In 2019 we commissioned NatCen to conduct a scoping review and recommend potential approaches to setting up a longitudinal survey to better understand movement in and out of experiencing gambling issues over the longer term and we are considering next steps. However longitudinal research is not in the scope of this consultation as the research aims of the two projects are not sufficiently aligned to be taken forward as one piece of work.

Reaching specific populations:

Surveys which are sampled from residential addresses can exclude certain population groups. Sufficient coverage of all groups without residential addresses would be very expensive and is also considered out of scope.

In December 2020 we launched a consultation to gather views on proposals to move towards a new method of data collection for adult gambling participation and problem gambling prevalence statistics in Great Britain. The consultation opened on 18 December 2020 and closed on 26 February 2021.

We received 62 valid responses to the consultation, from the following categories of respondents, details of which can be seen in Annex A.

We received 11 responses which had been submitted in relation to another consultation that was active at the time, these have been removed from the responses considered as part of our consultation response.

Additional context: GambleAware methodology review

In addition to the consultation responses, the following position statements also take into account GambleAware’s recently published Gambling Prevalence Methodology Review (opens in a new tab).

The GambleAware review intends to help refine its approach to collecting data on the measurement of people seeking treatment but has similar objectives to our own consultation in terms of understanding the best way to collect data on the prevalence of gambling harms in Great Britain. GambleAware published their review on 14 May 2021.

The GambleAware review is based on research it commissioned to determine best practices for estimating the level of gambling participation and prevalence of gambling harms in Great Britain. The research analysed eight surveys – including existing Health Survey data and further surveys commissioned by GambleAware – to investigate how differences in survey methodology affect the accuracy of estimates of ‘gambling harms’.

Analysis indicated that surveys conducted wholly or primarily online tend to overestimate the prevalence of gambling harms, however given the high and rising cost of in person surveys, the review recommended that future measurement of gambling prevalence and harm should move to online surveying. Further recommendations were made that the move to online data collection should be combined with a programme of methodological testing and development to mitigate selection bias, and that in person surveying should not be ceased completely; probability sampling and face to face interviewing should be used to provide periodic benchmarks.

Next steps

Based on the high level of agreement with our proposals, and the fact that GambleAware’s methodology review also supports a move away from in-person towards online surveying1, we intend to proceed with our plans to pilot a new approach for our official statistics on participation and problem gambling prevalence in 2021/22. We will work with our advisory groups and other key stakeholders to establish an evaluation framework for the success of the pilot. We will be transparent about our process and publish findings from the pilot survey. Subject to its successful completion, we will move to implementing a new methodology on a permanent basis from 2022/23.

Alongside the participation and prevalence strand of work, we will continue with a wider programme of research and will seek to make progress in the areas identified as out of scope of the core participation and prevalence survey. We will also explore other vehicles, such as the Public Health Outcomes Framework, to which gambling questions could helpfully be added.

Summary of responses - participation and prevalence research

Introduction to summary of responses

The consultation forms part of a package of work to develop a single high quality methodology to measure gambling participation and prevalence of problem gambling with the aim of setting the standard for authoritative research into gambling.

We consulted on six proposals in this consultation which were:

• to develop a single gold standard population survey for the whole of Great Britain
• to consolidate current surveys into one population survey
• to review and refresh the gambling activities included in the participation questions
• to improve the frequency and turnaround time of the survey data
• to explore more future proof data collection methods
• to pilot a new methodology and subject to a satisfactory pilot, to implement a new methodology from 2022.

Figure 1 provides a breakdown of the responses to the consultation questions. In Figure 1 and throughout the rest of the document we have reported the number of responses received in favour or against each proposal rather than the percentage, this is due to the relatively low base size that percentages would be based on (for example, below 100).

Proposal 5 was asked as a yes/no question rather than a 5-point Likert scale question. The green bar represents those who answered yes, the grey bar represents those who were undecided and the orange bar represents those who answered no.

Data for chart

	Proposal 1	Proposal 2	Proposal 3	Proposal 4	Proposal 5	Proposal 6
Strongly agree	18	11	20	14	0	14
Agree	24	31	26	28	34	26
Neither agree or disagree	9	7	5	10	20	14
Disagree	4	7	5	5	8	2
Strongly disagree	7	5	6	5	0	5

For each of the proposals we asked respondents to provide their views. We have summarised responses to these questions under respondent views and provided our response in the 'our position box'.

We acknowledge that there is a degree of overlap between the proposals in this consultation and this was evident in some respondents’ comments. For each proposal we have summarised the responses that are relevant to the proposal itself.

Proposal 1: Single survey for Great Britain

Issue

Lack of control over the inclusion of our questions on the Health Surveys limits our ability to report representative data for the whole of Great Britain.

Proposal

To replace our current usage of the separate Health Surveys for England and Scotland and equivalent survey in Wales with a new ‘gold standard’ population survey which covers the whole of Great Britain via a large and robust sample. We will consider both existing population surveys that we can access, and new surveys, which would be designed for this purpose, as a means of meeting this objective.

Respondents’ views

The level of agreement with this proposal was high (42 out of 62 responses), with 18 respondents strongly agreeing with the proposal. The highest level of agreement was amongst academics, all seven respondents who were from academia agreed with the proposal, with five out of seven strongly agreeing. Of the 11 respondents who disagreed with the proposal, eight were members of the public of which six said that they strongly disagreed.

Amongst those who agreed, it was felt that a single survey for Great Britain was a logical idea. They thought a large sample of respondents with consistency and coverage across all three nations would produce a more representative and useful dataset. One respondent representing academia welcomed the proposal, saying that the “inability to understand changes in gambling behaviour across GB (currently) hampers regulatory and policy decision making”.

However, respondents also thought it would be advantageous if the survey design had the potential to be flexed and adapted to meet the needs of the rapidly changing environment.

Some respondents thought it was unlikely that an existing survey vehicle could accommodate the needs of our research and therefore a bespoke population survey would need to be commissioned specifically for the Gambling Commission. One academic stated therefore that their agreement with the proposal was conditional on the new methodology utilising a ‘gold standard’ approach.

Whilst data at lower geographical levels was welcomed by respondents, there was some concern about what would be possible. One respondent made a comparison to the Active Lives Survey which has 500 responses per Local Authority area, commenting that this would be insufficient for a robust measurement of the prevalence of problem gambling at this low level geography. This led some respondents to comment that the investment required for a new bespoke population survey would be substantial, with an operator calling for the Commission’s research budget to be set out publicly as part of the Commission’s annual report and accounts.

One respondent from a trade association thought that the proposals for a new survey were only at the formative stage and welcomed the opportunity for external stakeholders to be more involved before any decisions are made. Similarly, others thought there was merit in further reviewing the current methodology before making any changes, with suggestions to push for secured space on the Scottish Health Survey and to implement a comparable Health Survey in Wales.

When asked if it is important to adopt a survey approach which covers the whole of Great Britain, the majority of respondents said that it was. 50 respondents said yes, six said no and six were undecided. Respondents thought the approach will help to provide a single set of results (i.e. one version of the truth) for Great Britain, and the ability to accurately compare results between nations. However, a survey vehicle which provided the ability to make regional adjustments to questions would also be seen as advantageous. Two respondents also thought that consideration should be given to a UK wide survey vehicle, especially they said given plans to modernise gambling laws in Northern Ireland.

Figure 2: Is important to adopt a survey approach which covers the whole of Great Britain?

Base: 62 responses

Data for chart

	Yes	No	Undecided
Number of responses	50	6	6

Proposal 2: Consolidation of current surveys

Issue:

Data from the different surveys is not directly comparable due to mode effects.

Proposal:

To reduce the number of surveys the Commission currently uses to produce official statistics on participation and prevalence to provide a single set of trusted metrics. As part of this, to absorb content from our existing surveys into the new population survey.

Respondents’ views

More than two thirds of respondents agreed with the consolidation of surveys outlined in Proposal 2, although respondents were more likely to agree than strongly agree (31 and 11 respectively) suggesting some degree of reservation. 12 out of 61 respondents disagreed with the proposal, with most disagreement coming from members of the public.

Respondents felt that consolidating the surveys would simplify the data and make it easier to use. By removing the issue of mode effects it would make the results easier to communicate and easier for the end user to interpret the results. Having one headline measure would also mean the results could be reported with confidence. Respondents also thought a consolidated approach may be more cost effective.

However, there was some hesitation amongst respondents about the amount of content in each of the surveys. Respondents felt it may be difficult to consolidate all the questions into one survey which may lead to content having to be left out, although a modular approach to questionnaire design could be considered.

By moving away from the Health Surveys there was also a degree of concern about the access to health data that might be lost and specifically the ability to cross reference gambling data with health data.

Another concern raised was about using survey data alone to understand gambling participation, one respondent from a gambling charity felt that administrative data on gambling spend and behaviour should be used to externally validate the claimed behaviour in survey findings.

When asked if the current reporting of data via different surveys is an issue, 28 out of 62 respondents said it was, 13 out of 62 said that it wasn’t an issue and 21 said they were undecided about whether it was an issue or not. All respondents from academic organisations and charity/not for profit organisations said that the way the data is currently reported causes an issue for them.

The main issues identified were the confusion the statistics can cause both in terms of publishing data with lots of caveats to explain the impact of mode and for the end user in interpreting results. One respondent also thought there could be a risk that results are cherry picked. The other issue identified was the inability to make direct comparisons between survey results.

Figure 3: Is the current reporting of data via different surveys an issue?

Base: 62 responses

Data for pie chart

	Yes	No	Undecided
Number of responses	28	13	21

Proposal 3: Participation questions

Issue:

Different participation questions on different surveys generate multiple figures.

Proposal:

Via a single preferred methodology to gather more granular data on gambling participation and frequency. Also, to review and refresh the list of gambling activities included in the survey so that it better reflects the current diversity of gambling products and better facilitates analysis of problem gambling prevalence at a product level.

Respondents’ views

This proposal received the highest level of overall agreement from respondents and the highest number of respondents who ‘strongly agreed’ with the proposal. 46 out of 62 respondents either strongly agreed or agreed that the participation questions should be refreshed (20 respondents strongly agreed and 26 agreed). All respondents from the charity/not for profit sector, those representing gambling operators and those from trade associations agreed with the proposal, six out of seven respondents from academia agreed with the proposal.

The opportunity to update the list of gambling activities to reflect current gambling behaviour was the main reason for agreeing with this proposal and it was felt that this would provide more credibility to the findings. One respondent stated that the benefit of updating the list of activities would outweigh the loss of losing trend series data. One operator would like to see operators involved in developing a current list of gambling activities.

One respondent with a research background who was in favour of refreshing the list of activities said that for net participation in an activity such as gambling to be accurate, it should be based on participation in individual activities which are current and relevant now rather than being based on participation in activities which have become outdated or irrelevant.

Whilst the idea of being able to expand the list of gambling activities and improve the granularity of the data was appealing to respondents, there was an acknowledgement that this would need to be balanced with survey length.

Finally, another respondent reiterated the importance of our commitment outlined in the consultation document to produce statistics in accordance with the standards set out by the Government Statistical Service in the Code of Practice for Statistics around Trustworthiness, Quality and Value. They encouraged the Commission to engage with stakeholders including the UK Statistics Authority on the development of a new survey approach.

26 respondents said there are other factors that need to be considered when developing participation questions and data to meet the needs of users. One of these factors was the requirement for cognitive testing of the questions and another was the development of an advisory group who could advise on the development of the questionnaire to include representatives from academia, industry and people with lived experience.

Specific topics or question areas that respondents thought should be included in a new survey were gambling related harms, affordability, expenditure, consumer benefits of gambling, online gaming (and the overlap with gambling) and also people’s understanding of the risks associated with gambling. Another respondent from the charity sector also thought a review of problem gambling screens, namely the PGSI and DSM-IV, should be undertaken to make sure they are still the most effective way of measuring problem gambling.

Proposal 4: Frequency and turnaround time

Issue:

The infrequency and long turnaround time of the Health Surveys from inception to reporting.

Proposal:

To explore surveys (including existing external surveys) which we would be able to access more frequently than the Health Surveys and which have a shorter turnaround time. To move towards at least annual publication of participation and prevalence metrics.

Respondents’ views

The majority of respondents agreed with the proposal to explore more timely survey options, although respondents had a stronger tendency to agree (28 respondents) rather than strongly agree (14 respondents). 10 respondents neither agreed nor disagreed with the proposal and 10 respondents disagreed (5 disagree and 5 strongly disagree).

More timely access to surveys and results was desirable amongst respondents, with a regular annual publication seen as a good idea which would keep up with the pace of change and make sure policy decisions are based on up to date evidence. One respondent from a trade association thought it would be good if the publication of the data could align with the financial year to help with business planning.

Respondents were however keen to understand the practical benefits the Commission hopes to get from faster reporting, querying what issues the current time lag presents and what policy decisions the Commission thinks might have been taken or done differently with more up to date information, especially given that gambling behaviours appear to change at a modest rate.

Respondents were also keen to stress that whilst a faster turnaround time sounds good, it should not be at the expense of robustness or comparability. If, however we were able to maintain an appropriately robust methodology and reduce the time lag between data collection and publication, this would be welcomed and would meet best practice standards for official statistics.

One of the issues highlighted in the consultation document was that survey content cannot be adapted quickly to reflect new gambling behaviours and whilst respondents thought that a bespoke survey would provide the flexibility to make survey design changes, others were concerned about comparability of trends over time if questions were constantly changing. A modular approach to questionnaire design, whereby pressing policy issues could be explored through ad hoc questions added on a modular basis, would however be advantageous.

One respondent asked if options to speed up access to the Health Survey data had been explored fully. Another commented on an increasing move in health research to develop swifter and more flexible approaches including continuous research which have been accelerated by the Covid-19 pandemic. Examples include the Better Health Programme Wider Impacts of COVID-19 (opens in a new tab).

When asked how often it would be appropriate to report gambling participation and prevalence data, most respondents were satisfied with annual reporting or as in the example given in the consultation document, annual reporting with shorter quarterly updates.

Figure 4: Frequency of reporting

Base: 36 responses

Data for chart

	Annual	Quarterly	Biannual	Monthly
Number of responses	19	13	3	1

Most respondents thought it was important to release data on gambling participation and prevalence of problem gambling quickly after its collection, 23 respondents said it was very important and 17 said it was fairly important. However, respondents were also keen to stress that the quick turnaround of the data needs to be balanced alongside the need for rigour and statistical accuracy. There was also a suggestion to build in time to peer review any publication.

Figure 5: Importance of releasing data more quickly after collection

Base: 59 responses

Data for chart

	Very important	Fairly important	Important	Slightly important	Not important
Number of responses	23	17	9	5	5

Proposal 5: Explore more future proof methods

Issue

Traditional research methods, already in decline are now under greater threat due to Covid-19.

Proposal

To explore more ‘future proof’ methodologies for ongoing measurement which will be able to withstand threats posed to more ‘traditional’ research approaches. These methods include online, 'push to web' and mixed-mode surveys.

Respondents’ views

Over half (34 out of 62) of respondents supported the proposal to use an alternative, non-interviewer-led methodology in future surveys. Whilst 8 respondents did not support the proposal, 20 respondents were undecided on whether to support it or not.

Those who were in favour of a non-interviewer-led approach thought that a push-to-web approach offered the most viable alternative to face to face interviewing, and the most logical approach for the future especially given the Covid-19 pandemic. For a standalone bespoke gambling survey, respondents also thought a push to web survey would be the most cost effective solution. An online approach would also meet the Government’s digital transformation strategy.

Some respondents thought a non-interviewer-led approach would also reduce the impact of social desirability bias which might be an issue with some methodologies (e.g. telephone survey) given the sensitive nature of gambling questions.

There were some notes of caution however, the first about the importance of a random probability sampling approach rather than relying on self-selected online panels. Some were also cautious about an online only approach and stressed the importance of a mixed mode methodology to ensure representation across the population, including the idea of boosting response rates amongst some population groups, or even recruiting sub panels to take part in rapid response research.

Respondents provided some good examples of large scale surveys that had already made a similar methodology switch or used a mixed mode approach such as the Active Lives Survey, the OPSS Consumer Tracker Survey which consists of 10,000 online and 500 telephone surveys with those with low or no internet access, the Legal Needs of Individuals Study (LSB/Law Society) with 30,000 online responses and an offline calibration to determine weighting and general population surveys in Massachusetts which have used address based sampling and mixed mode methods of completion since 2013. Reference was also made to one research agency’s ‘Knowledge Panel’ which is a random probability online panel with over 10,000 panellists.

An acknowledgement that all research methodologies have pros and cons was made in the consultation response and for this reason a pilot study and the opportunity to run surveys using different methodologies side by side seemed like a good idea. A respondent also repeated their suggestion that the Commission should make more use of administrative data to validate survey responses, and another thought we should be considering incorporating alternative, less traditional methodologies such as netnography (Twitter scraping), ethnography and videography into our research programme.

Those who were not in favour of the proposal considered face to face interviewer led research to be superior to other methodologies and were not as confident in an online approach. There was also a concern about the loss of comparability of data over time due to a methodological change.

Whilst Covid-19 is currently prohibiting face to face interviewing, some felt that it was rash to make a methodology change when things may go back to normal in the future. Another respondent requested that the Commission consults on this topic again once an alternative methodology has been proposed.

Proposal 6: Survey pilot

Issue

Changing the survey method could result in changes to the data.

Proposal

• to pilot questions using a potential new methodology in 2021 so that we can compare the results of the pilot with the telephone and online surveys that take place over a similar time frame, and with the most recent Health Survey data (2018)
• to take steps to ensure that the survey does not encourage an over-representation of gamblers, by taking care in the way the survey is branded and introduced to participants
• to analyse and report on comparability of trend data
• subject to satisfactory pilot study data, to begin our new survey methodology in 2022.

Respondents’ views

Most respondents agreed with Proposal 6; to pilot questions using a new methodology in 2021. 40 out of 61 respondents agreed with the proposal, of which 14 strongly agreed and 26 agreed. Five respondents who strongly agreed with the proposal were from academia. Just seven respondents disagreed with the proposal, with more strongly disagreeing (5) than disagreeing (2). A total of 14 respondents neither agreed nor disagreed with the proposal, with 10 of these respondents being members of the public.

Those who agreed thought that a pilot was a sensible approach to take, with one respondent saying that the Commission should take advice on the size and the scale of the pilot required to be able to robustly compare results with previous methods. The aim of the pilot should be to give users trust and confidence in the accuracy of the data going forward.

There was some concern about the timing of the pilot given the impact of Covid-19 on gambling behaviours. Respondents thought that results from a 2021 pilot which are compared to results collected pre-Covid (e.g. the Health Survey England 2018) need to be used with caution as differences may be due to changes in the gambling environment as a result of Covid rather than methodological differences. Similarly, comparisons to the 2016 Health Surveys should also be made with caution, given that the fieldwork took place five years ago and the gambling environment has changed since then. Respondents seemed to think that the solution should be comparing the results of the pilot study to a variety of other data sources rather than just one. This should include Gamble Aware's methodology review (opens in a new tab) or the 2021 Health Survey, although it is acknowledged this would cause a delay for the Commission’s methodology review.

One of the respondents who disagreed with the pilot approach thought that rather than piloting a new survey approach we should be overhauling the whole research programme to include the use of administrative data for survey validation.

Finally, a request for the outcome of the pilot to be published was made by an operator.

When asked if a break in the time series of data would impact their work, most respondents were of the mind that it would not. As one respondent said, they have managed with the uncertainty of prevalence estimates for some time (due to mode effects) and therefore if a break in time series means that the data quality and reliability is improved it is worth doing. Another said that they do not use data from the Commission’s quarterly telephone survey because the full dataset is not currently publicly available so for them a break in time series would not be an issue.

One respondent who thought the break in time series would impact their work was concerned that it would impact their work in relation to safer gambling and could lead to an increase in the misuse of data.

Figure 6: Would a break in the time series impact your work?

Base: 59 responses

Data from pie chart

	Yes	No	Undecided
Number of responses	7	36	16

Annex A: List of consultation responses

We received 62 valid responses to the consultation, from the following categories of respondents:

• Member of the public: 37 responses
• Academia: 7 responses
• Gambling operator: 5 responses
• Charity/not for profit: 3 responses
• Trade association: 2 responses
• Other: 8 responses (including representatives from Ipsos MORI, NatCen, YouGov and the Office for National Statistics)

We received 11 responses which had been submitted in relation to another consultation that was active at the time, these have been removed from the responses considered as part of our consultation response.