With this document you can:

This box is not visible in the printed version.

Participation and prevalence research

From December 2020 to February 2021 we consulted on the proposals to change how we collect adult gambling participation and problem gambling prevalence statistics.

Published: 23 June 2021

Last updated: 23 June 2021

This version was printed or saved on: 8 August 2022

Online version: https://www.gamblingcommission.gov.uk/consultation-response/participation-and-prevalence-research

Executive summary - participation and prevalence research

From December 2020-February 2021, we consulted on proposals to change the research methodology we use to collect adult gambling participation and problem gambling prevalence statistics. This is a data collection duty set by the Gambling Act 2005 and the data is also official statistics.

The consultation offset a limited scope related to this duty, with a focus on core participation and prevalence measurement. The aim is to provide the strongest possible foundation which we, and other interested parties can base further work on. We recognise that deficiencies currently exist in the wider system for measuring gambling harms and will continue our commitment to play our part in addressing these, through a combination of our own wider research programme and external research by partner organisations.

Our consultation outlined proposals to improve the quality, robustness and timeliness of our participation and prevalence official statistics. Specifically, the proposals sought to:

The consultation outlined the aim to develop a single, high quality methodology to measure participation and prevalence which will be more efficient, cost effective and timely, helping us to respond quickly to emerging consumer issues. We believe that a new approach will enable us to set the standard for authoritative research into gambling.

We received 62 valid responses to the consultation which ran for 10 weeks. This document summarises the responses received and explains our position on each proposal.

Our proposals were well supported, with an average of two-thirds of respondents agreeing with our proposals and only one in five disagreeing. On this basis, we intend to proceed with our plans to pilot a new approach in 2021/22 and subject to its successful completion, move to implementing a new methodology on a permanent basis from 2022/23.

Introduction - participation and prevalence research

The Gambling Commission

The Gambling Commission exists to safeguard consumers and the wider public by ensuring that gambling is fair and safe. We place consumers at the heart of regulation and maintain the integrity of the gambling industry.

Our work is underpinned by two main pieces of legislation:

Under section 26 of the Gambling Act 2005 (opens in a new tab), the Commission has responsibility for collecting and disseminating information relating to the extent and impact of gambling in Great Britain. In order to do this, we collect gambling participation and problem gambling prevalence data via surveys of adults in Great Britain and the data is published as official statistics – meaning they are produced in accordance with the standards set out by the Government Statistical Service in the Code of Practice for Statistics.

We are also responsible for a wider programme of both quantitative and qualitative research which does not constitute core official statistics, but which helps us to build a better understanding of consumer gambling behaviour, attitudes and harms. Although some of this work is outside the scope of this consultation, we remain committed to driving this forward.

Consultation proposals and background

The Commission is ambitious about improving the quality, robustness and timeliness of our statistics. Timely, robust measurement of consumer gambling behaviour is critical to identify trends to help prioritise focus (for example take-up of new gambling and gambling-style products), to measure levels of risk and harm to consumers (via problem gambling screens) and to monitor the impact of policy changes. We therefore set out a commitment in our 2020/21 Business Plan to ‘review our approach to measuring participation and prevalence and publish conclusions’.

Our intention is to develop a single, high quality methodology to measure participation and prevalence which will be more efficient, cost effective and timely, helping to better inform us of the impact of policy changes and provide an ongoing foundation for evidence-based decision making. We believe that a new approach will enable us to set the standard for authoritative research into gambling.

Recognising gambling as a public health issue, there are a number of important areas which are linked to the methodology review but are considered out of scope of this consultation:

Our National Strategy to Reduce Gambling Harms sets out the aim to identify a robust means of measuring harm and we are committed to doing what we can (with the resources we have available) to build and contribute to this evidence base. We have started a pilot of new survey questions on harm. As this work is currently in progress, the specific means of measuring harm is not within the consultation scope.

Longitudinal research:

In 2019 we commissioned NatCen to conduct a scoping review and recommend potential approaches to setting up a longitudinal survey to better understand movement in and out of experiencing gambling issues over the longer term and we are considering next steps. However longitudinal research is not in the scope of this consultation as the research aims of the two projects are not sufficiently aligned to be taken forward as one piece of work.

Reaching specific populations:

Surveys which are sampled from residential addresses can exclude certain population groups. Sufficient coverage of all groups without residential addresses would be very expensive and is also considered out of scope.

In December 2020 we launched a consultation to gather views on proposals to move towards a new method of data collection for adult gambling participation and problem gambling prevalence statistics in Great Britain. The consultation opened on 18 December 2020 and closed on 26 February 2021.

We received 62 valid responses to the consultation, from the following categories of respondents, details of which can be seen in Annex A.

  • Member of the public: 37 responses
  • Academia: 7 responses
  • Gambling operator: 5 responses
  • Charity/not for profit: 3 responses
  • Trade association: 2 responses
  • Other: 8 responses (including representatives from Ipsos MORI, NatCen, YouGov and the Office for National Statistics)

We received 11 responses which had been submitted in relation to another consultation that was active at the time, these have been removed from the responses considered as part of our consultation response.

Additional context: GambleAware methodology review

In addition to the consultation responses, the following position statements also take into account GambleAware’s recently published Gambling Prevalence Methodology Review (opens in a new tab).

The GambleAware review intends to help refine its approach to collecting data on the measurement of people seeking treatment but has similar objectives to our own consultation in terms of understanding the best way to collect data on the prevalence of gambling harms in Great Britain. GambleAware published their review on 14 May 2021.

The GambleAware review is based on research it commissioned to determine best practices for estimating the level of gambling participation and prevalence of gambling harms in Great Britain. The research analysed eight surveys – including existing Health Survey data and further surveys commissioned by GambleAware – to investigate how differences in survey methodology affect the accuracy of estimates of ‘gambling harms’.

Analysis indicated that surveys conducted wholly or primarily online tend to overestimate the prevalence of gambling harms, however given the high and rising cost of in person surveys, the review recommended that future measurement of gambling prevalence and harm should move to online surveying. Further recommendations were made that the move to online data collection should be combined with a programme of methodological testing and development to mitigate selection bias, and that in person surveying should not be ceased completely; probability sampling and face to face interviewing should be used to provide periodic benchmarks.

Next steps

Based on the high level of agreement with our proposals, and the fact that GambleAware’s methodology review also supports a move away from in-person towards online surveying1, we intend to proceed with our plans to pilot a new approach for our official statistics on participation and problem gambling prevalence in 2021/22. We will work with our advisory groups and other key stakeholders to establish an evaluation framework for the success of the pilot. We will be transparent about our process and publish findings from the pilot survey. Subject to its successful completion, we will move to implementing a new methodology on a permanent basis from 2022/23.

Alongside the participation and prevalence strand of work, we will continue with a wider programme of research and will seek to make progress in the areas identified as out of scope of the core participation and prevalence survey. We will also explore other vehicles, such as the Public Health Outcomes Framework, to which gambling questions could helpfully be added.

Summary of responses - participation and prevalence research

Introduction to summary of responses

The consultation forms part of a package of work to develop a single high quality methodology to measure gambling participation and prevalence of problem gambling with the aim of setting the standard for authoritative research into gambling.

We consulted on six proposals in this consultation which were:

Figure 1 provides a breakdown of the responses to the consultation questions. In Figure 1 and throughout the rest of the document we have reported the number of responses received in favour or against each proposal rather than the percentage, this is due to the relatively low base size that percentages would be based on (for example, below 100).

Image 1 - a breakdown of the responses to the consultation questions harms

Proposal 5 was asked as a yes/no question rather than a 5-point Likert scale question. The green bar represents those who answered yes, the grey bar represents those who were undecided and the orange bar represents those who answered no.

Data for chart

Proposal 1Proposal 2Proposal 3Proposal 4Proposal 5Proposal 6
Strongly agree18112014014
Agree243126283426
Neither agree or disagree975102014
Disagree475582
Strongly disagree756505

For each of the proposals we asked respondents to provide their views. We have summarised responses to these questions under respondent views and provided our response in the 'our position box'.

We acknowledge that there is a degree of overlap between the proposals in this consultation and this was evident in some respondents’ comments. For each proposal we have summarised the responses that are relevant to the proposal itself.

Proposal 1: Single survey for Great Britain

Issue

Lack of control over the inclusion of our questions on the Health Surveys limits our ability to report representative data for the whole of Great Britain.

Proposal

To replace our current usage of the separate Health Surveys for England and Scotland and equivalent survey in Wales with a new ‘gold standard’ population survey which covers the whole of Great Britain via a large and robust sample. We will consider both existing population surveys that we can access, and new surveys, which would be designed for this purpose, as a means of meeting this objective.

Consultation questions

  • Q5. Do you agree with this proposal?
  • Q6. Is it important to adopt a survey approach which covers the whole of Great Britain?

Respondents’ views

The level of agreement with this proposal was high (42 out of 62 responses), with 18 respondents strongly agreeing with the proposal. The highest level of agreement was amongst academics, all seven respondents who were from academia agreed with the proposal, with five out of seven strongly agreeing. Of the 11 respondents who disagreed with the proposal, eight were members of the public of which six said that they strongly disagreed.

Amongst those who agreed, it was felt that a single survey for Great Britain was a logical idea. They thought a large sample of respondents with consistency and coverage across all three nations would produce a more representative and useful dataset. One respondent representing academia welcomed the proposal, saying that the “inability to understand changes in gambling behaviour across GB (currently) hampers regulatory and policy decision making”.

However, respondents also thought it would be advantageous if the survey design had the potential to be flexed and adapted to meet the needs of the rapidly changing environment.

Some respondents thought it was unlikely that an existing survey vehicle could accommodate the needs of our research and therefore a bespoke population survey would need to be commissioned specifically for the Gambling Commission. One academic stated therefore that their agreement with the proposal was conditional on the new methodology utilising a ‘gold standard’ approach.

Whilst data at lower geographical levels was welcomed by respondents, there was some concern about what would be possible. One respondent made a comparison to the Active Lives Survey which has 500 responses per Local Authority area, commenting that this would be insufficient for a robust measurement of the prevalence of problem gambling at this low level geography. This led some respondents to comment that the investment required for a new bespoke population survey would be substantial, with an operator calling for the Commission’s research budget to be set out publicly as part of the Commission’s annual report and accounts.

One respondent from a trade association thought that the proposals for a new survey were only at the formative stage and welcomed the opportunity for external stakeholders to be more involved before any decisions are made. Similarly, others thought there was merit in further reviewing the current methodology before making any changes, with suggestions to push for secured space on the Scottish Health Survey and to implement a comparable Health Survey in Wales.

When asked if it is important to adopt a survey approach which covers the whole of Great Britain, the majority of respondents said that it was. 50 respondents said yes, six said no and six were undecided. Respondents thought the approach will help to provide a single set of results (i.e. one version of the truth) for Great Britain, and the ability to accurately compare results between nations. However, a survey vehicle which provided the ability to make regional adjustments to questions would also be seen as advantageous. Two respondents also thought that consideration should be given to a UK wide survey vehicle, especially they said given plans to modernise gambling laws in Northern Ireland.

Figure 2: Is important to adopt a survey approach which covers the whole of Great Britain?

Base: 62 responses

Figure 2 -  A pie chart showing the 62 responses to the question Is important to adopt a survey approach which covers the whole of Great Britain?

Data for chart

YesNoUndecided
Number of responses5066

Our position

Our proposal was to develop a new population survey which covers the whole of Great Britain via a large and robust sample to overcome the lack of control we have on the questions included in the Health Surveys and the inability to report GB-wide data on a regular basis. The proposal was met with a high level of agreement from respondents.

We therefore plan to progress with Proposal 1 and develop a GB-wide survey for measuring gambling participation and the prevalence of problem gambling.

Based on the feedback received we will explore options and costs for a bespoke survey commissioned by the Gambling Commission for the purposes of understanding gambling behaviour. However, we will also consider if there are any existing survey vehicles that might be suitable.

If a bespoke survey is commissioned, we will ensure the questionnaire is designed in a way that ensures a core set of participation and prevalence questions are asked regularly alongside modules of questions asked on a less frequent basis.

We note that respondents think it is desirable for data to be available at lower geographical levels and we look to ensure that the sample can be increased to accommodate this if budget allows. However, we acknowledge that the budget may need to increase considerably to allow for analysis at a lower geographical level.

As our remit is to regulate gambling in Great Britain our initial focus will be on rolling out a GB-wide survey. However, we welcome the opportunity to align research with colleagues in Northern Ireland so that useful comparisons can be drawn between the nations and will explore approaches which would be flexible enough to be extended to Northern Ireland in the future if that is considered a good idea.

Proposal 2: Consolidation of current surveys

Issue:

Data from the different surveys is not directly comparable due to mode effects.

Proposal:

To reduce the number of surveys the Commission currently uses to produce official statistics on participation and prevalence to provide a single set of trusted metrics. As part of this, to absorb content from our existing surveys into the new population survey.

Consultation questions

  • Q7. Do you agree with this proposal?
  • Q8. To what extent is the current reporting of data via different surveys an issue?

Respondents’ views

More than two thirds of respondents agreed with the consolidation of surveys outlined in Proposal 2, although respondents were more likely to agree than strongly agree (31 and 11 respectively) suggesting some degree of reservation. 12 out of 61 respondents disagreed with the proposal, with most disagreement coming from members of the public.

Respondents felt that consolidating the surveys would simplify the data and make it easier to use. By removing the issue of mode effects it would make the results easier to communicate and easier for the end user to interpret the results. Having one headline measure would also mean the results could be reported with confidence. Respondents also thought a consolidated approach may be more cost effective.

However, there was some hesitation amongst respondents about the amount of content in each of the surveys. Respondents felt it may be difficult to consolidate all the questions into one survey which may lead to content having to be left out, although a modular approach to questionnaire design could be considered.

By moving away from the Health Surveys there was also a degree of concern about the access to health data that might be lost and specifically the ability to cross reference gambling data with health data.

Another concern raised was about using survey data alone to understand gambling participation, one respondent from a gambling charity felt that administrative data on gambling spend and behaviour should be used to externally validate the claimed behaviour in survey findings.

When asked if the current reporting of data via different surveys is an issue, 28 out of 62 respondents said it was, 13 out of 62 said that it wasn’t an issue and 21 said they were undecided about whether it was an issue or not. All respondents from academic organisations and charity/not for profit organisations said that the way the data is currently reported causes an issue for them.

The main issues identified were the confusion the statistics can cause both in terms of publishing data with lots of caveats to explain the impact of mode and for the end user in interpreting results. One respondent also thought there could be a risk that results are cherry picked. The other issue identified was the inability to make direct comparisons between survey results.

Figure 3: Is the current reporting of data via different surveys an issue?

Base: 62 responses

Figure 3 -  A pie chart showing the 62 responses to the question Is the current reporting of data via different surveys an issue? Data for pie chart

YesNoUndecided
Number of responses281321

Our position

We will aim to reduce the number of surveys we use to produce official statistics on participation and prevalence to produce a single set of trusted metrics.

We will focus on consolidating questions from the Health Surveys and the Commission’s current quarterly telephone survey. As the questions in the Commission’s online tracker survey are no longer deemed to be official statistics, we do not intend to consolidate all the questions from the online survey into a single GB wide survey.

We will undertake a thorough review of the current questionnaires to determine which questions should be included in a consolidated survey design. We will also consider which questions from the Health Survey we may wish to retain to be able to continue to cross tabulate gambling with health-related metrics.

We understand the desire amongst respondents to be able to access data from the surveys and we will aim to publish data from a new consolidated GB-wide survey on our website and in the UK data archive.

Proposal 3: Participation questions

Issue:

Different participation questions on different surveys generate multiple figures.

Proposal:

Via a single preferred methodology to gather more granular data on gambling participation and frequency. Also, to review and refresh the list of gambling activities included in the survey so that it better reflects the current diversity of gambling products and better facilitates analysis of problem gambling prevalence at a product level.

Consultation questions

  • Q9. Do you agree with this proposal?
  • Q10. What other factors that should be considered in developing participation questions and data to meet the needs of users?

Respondents’ views

This proposal received the highest level of overall agreement from respondents and the highest number of respondents who ‘strongly agreed’ with the proposal. 46 out of 62 respondents either strongly agreed or agreed that the participation questions should be refreshed (20 respondents strongly agreed and 26 agreed). All respondents from the charity/not for profit sector, those representing gambling operators and those from trade associations agreed with the proposal, six out of seven respondents from academia agreed with the proposal.

The opportunity to update the list of gambling activities to reflect current gambling behaviour was the main reason for agreeing with this proposal and it was felt that this would provide more credibility to the findings. One respondent stated that the benefit of updating the list of activities would outweigh the loss of losing trend series data. One operator would like to see operators involved in developing a current list of gambling activities.

One respondent with a research background who was in favour of refreshing the list of activities said that for net participation in an activity such as gambling to be accurate, it should be based on participation in individual activities which are current and relevant now rather than being based on participation in activities which have become outdated or irrelevant.

Whilst the idea of being able to expand the list of gambling activities and improve the granularity of the data was appealing to respondents, there was an acknowledgement that this would need to be balanced with survey length.

Finally, another respondent reiterated the importance of our commitment outlined in the consultation document to produce statistics in accordance with the standards set out by the Government Statistical Service in the Code of Practice for Statistics around Trustworthiness, Quality and Value. They encouraged the Commission to engage with stakeholders including the UK Statistics Authority on the development of a new survey approach.

26 respondents said there are other factors that need to be considered when developing participation questions and data to meet the needs of users. One of these factors was the requirement for cognitive testing of the questions and another was the development of an advisory group who could advise on the development of the questionnaire to include representatives from academia, industry and people with lived experience.

Specific topics or question areas that respondents thought should be included in a new survey were gambling related harms, affordability, expenditure, consumer benefits of gambling, online gaming (and the overlap with gambling) and also people’s understanding of the risks associated with gambling. Another respondent from the charity sector also thought a review of problem gambling screens, namely the PGSI and DSM-IV, should be undertaken to make sure they are still the most effective way of measuring problem gambling.

Our position

Based on the positive feedback about this proposal we will review and refresh the gambling activities included in the survey to better reflect the current repertoire of gambling.

We will set up a working group with representatives from external stakeholders to advise on questionnaire design and specifically a refreshed set of gambling activities to help measure gambling participation. The requirement to establish a working group will be included as part of the specification when we are looking for a research partner to support us with the pilot.

Cognitive testing of any new questions will be included as part of the pilot.

Whilst the specific means of measuring harms via survey questions is not in scope within this consultation, it is intended that the current pilot of harms questions which is taking place outside of this consultation will result in a set of questions which can be added to our core survey(s) in the future to measure the extent to which gambling related harms are experienced by gamblers and affected others. We see the harms questions being run alongside existing screens to measure problem gambling (PGSI and DSM-IV).

As part of our review leading up to the consultation we have engaged with the ONS, DCMS, research experts and our Advisory Board for Safer Gambling (ABSG). We also reached out to other organisations which produce official or National statistics. Our statistics are subject to the Code of Practice for Statistics set out by the UK Statistics Authority (UKSA) and must abide by the principles of trustworthiness, quality and value. Our pre-consultation discussions and the consultation responses have not raised any concerns that the proposals outlined in this document will pose any risk to these principles, and in fact are more likely to strengthen our commitment towards them.

Proposal 4: Frequency and turnaround time

Issue:

The infrequency and long turnaround time of the Health Surveys from inception to reporting.

Proposal:

To explore surveys (including existing external surveys) which we would be able to access more frequently than the Health Surveys and which have a shorter turnaround time. To move towards at least annual publication of participation and prevalence metrics.

Consultation questions

  • Q11. Do you agree with this proposal?
  • Q12. What is an appropriate frequency for reporting?
  • Q13. How important is it to release data more quickly after its collection and do you have any views on what sort of timescales would be acceptable or preferable?

Respondents’ views

The majority of respondents agreed with the proposal to explore more timely survey options, although respondents had a stronger tendency to agree (28 respondents) rather than strongly agree (14 respondents). 10 respondents neither agreed nor disagreed with the proposal and 10 respondents disagreed (5 disagree and 5 strongly disagree).

More timely access to surveys and results was desirable amongst respondents, with a regular annual publication seen as a good idea which would keep up with the pace of change and make sure policy decisions are based on up to date evidence. One respondent from a trade association thought it would be good if the publication of the data could align with the financial year to help with business planning.

Respondents were however keen to understand the practical benefits the Commission hopes to get from faster reporting, querying what issues the current time lag presents and what policy decisions the Commission thinks might have been taken or done differently with more up to date information, especially given that gambling behaviours appear to change at a modest rate.

Respondents were also keen to stress that whilst a faster turnaround time sounds good, it should not be at the expense of robustness or comparability. If, however we were able to maintain an appropriately robust methodology and reduce the time lag between data collection and publication, this would be welcomed and would meet best practice standards for official statistics.

One of the issues highlighted in the consultation document was that survey content cannot be adapted quickly to reflect new gambling behaviours and whilst respondents thought that a bespoke survey would provide the flexibility to make survey design changes, others were concerned about comparability of trends over time if questions were constantly changing. A modular approach to questionnaire design, whereby pressing policy issues could be explored through ad hoc questions added on a modular basis, would however be advantageous.

One respondent asked if options to speed up access to the Health Survey data had been explored fully. Another commented on an increasing move in health research to develop swifter and more flexible approaches including continuous research which have been accelerated by the Covid-19 pandemic. Examples include the Better Health Programme Wider Impacts of COVID-19 (opens in a new tab).

When asked how often it would be appropriate to report gambling participation and prevalence data, most respondents were satisfied with annual reporting or as in the example given in the consultation document, annual reporting with shorter quarterly updates.

Figure 4: Frequency of reporting

Base: 36 responses

Figure 4 -  A chart showing the frequency of reporting

Data for chart

AnnualQuarterlyBiannualMonthly
Number of responses191331

Most respondents thought it was important to release data on gambling participation and prevalence of problem gambling quickly after its collection, 23 respondents said it was very important and 17 said it was fairly important. However, respondents were also keen to stress that the quick turnaround of the data needs to be balanced alongside the need for rigour and statistical accuracy. There was also a suggestion to build in time to peer review any publication.

Figure 5: Importance of releasing data more quickly after collection

Base: 59 responses

Figure 5 -  A chart showing how the respondents viewed the important of releasing data more quickly

Data for chart

Very importantFairly importantImportantSlightly importantNot important
Number of responses2317955

Our position

We will seek to establish a survey which enables us to publish participation and prevalence statistics on at least an annual basis and quarterly if possible, whilst ensuring this more frequent publication does not impact the robustness or the accuracy of the research.

We will also consider the timing of the publication, possibly to align with the financial year.

As part of our commitment to refresh the gambling activities included with the participation question in the survey, we will also establish a set of core participation and prevalence metrics which are asked on every wave of the research alongside a modular design for other topic areas which can be asked on a less frequent basis.

We will explore examples of continuous research in health research.

We will seek advice on a periodic external peer review process for publication.

Proposal 5: Explore more future proof methods

Issue

Traditional research methods, already in decline are now under greater threat due to Covid-19.

Proposal

To explore more ‘future proof’ methodologies for ongoing measurement which will be able to withstand threats posed to more ‘traditional’ research approaches. These methods include online, 'push to web' and mixed-mode surveys.

Consultation question

  • Q14. Do you support the proposal to use an alternative, non-interviewer-led methodology?

Respondents’ views

Over half (34 out of 62) of respondents supported the proposal to use an alternative, non-interviewer-led methodology in future surveys. Whilst 8 respondents did not support the proposal, 20 respondents were undecided on whether to support it or not.

Those who were in favour of a non-interviewer-led approach thought that a push-to-web approach offered the most viable alternative to face to face interviewing, and the most logical approach for the future especially given the Covid-19 pandemic. For a standalone bespoke gambling survey, respondents also thought a push to web survey would be the most cost effective solution. An online approach would also meet the Government’s digital transformation strategy.

Some respondents thought a non-interviewer-led approach would also reduce the impact of social desirability bias which might be an issue with some methodologies (e.g. telephone survey) given the sensitive nature of gambling questions.

There were some notes of caution however, the first about the importance of a random probability sampling approach rather than relying on self-selected online panels. Some were also cautious about an online only approach and stressed the importance of a mixed mode methodology to ensure representation across the population, including the idea of boosting response rates amongst some population groups, or even recruiting sub panels to take part in rapid response research.

Respondents provided some good examples of large scale surveys that had already made a similar methodology switch or used a mixed mode approach such as the Active Lives Survey, the OPSS Consumer Tracker Survey which consists of 10,000 online and 500 telephone surveys with those with low or no internet access, the Legal Needs of Individuals Study (LSB/Law Society) with 30,000 online responses and an offline calibration to determine weighting and general population surveys in Massachusetts which have used address based sampling and mixed mode methods of completion since 2013. Reference was also made to one research agency’s ‘Knowledge Panel’ which is a random probability online panel with over 10,000 panellists.

An acknowledgement that all research methodologies have pros and cons was made in the consultation response and for this reason a pilot study and the opportunity to run surveys using different methodologies side by side seemed like a good idea. A respondent also repeated their suggestion that the Commission should make more use of administrative data to validate survey responses, and another thought we should be considering incorporating alternative, less traditional methodologies such as netnography (Twitter scraping), ethnography and videography into our research programme.

Those who were not in favour of the proposal considered face to face interviewer led research to be superior to other methodologies and were not as confident in an online approach. There was also a concern about the loss of comparability of data over time due to a methodological change.

Whilst Covid-19 is currently prohibiting face to face interviewing, some felt that it was rash to make a methodology change when things may go back to normal in the future. Another respondent requested that the Commission consults on this topic again once an alternative methodology has been proposed.

Our position

We will explore more future proof methodologies for the ongoing measurement of gambling participation and prevalence which are able to withstand the threats that are posed to more traditional methodologies.

It is very important to maintain the robustness and accuracy of the research. Therefore, in commissioning a new survey, we will aim to use a random probability sampling approach if possible.

In commissioning a new survey, we will be open to both fully online and ‘push to web’ approaches (supplemented by offline data collection for periodic benchmarks) and mixed methodology approaches which will maximise sample representativeness.

We will continue to build our knowledge of other national surveys that have switched methodology to build on best practice.

We acknowledge the suggestion that our research programme should utilise a wider range of research methodologies. Whilst we do not think that all of these are suitable to produce official statistics about gambling participation and prevalence as per this consultation, subject to budget we continue to be open to exploring new and innovative research methodologies in our wider programme of research. We also welcome the use of new and varied approaches by partners conducting research in support of the National Strategy for Reducing Gambling Harm and will continue to ensure that learnings from the wider research space are integrated into our evidence base.

Proposal 6: Survey pilot

Issue

Changing the survey method could result in changes to the data.

Proposal

Consultation question boxes:

  • Q15. Do you agree with this proposal?
  • Q16. Would a break in the time series impact your work?

Respondents’ views

Most respondents agreed with Proposal 6; to pilot questions using a new methodology in 2021. 40 out of 61 respondents agreed with the proposal, of which 14 strongly agreed and 26 agreed. Five respondents who strongly agreed with the proposal were from academia. Just seven respondents disagreed with the proposal, with more strongly disagreeing (5) than disagreeing (2). A total of 14 respondents neither agreed nor disagreed with the proposal, with 10 of these respondents being members of the public.

Those who agreed thought that a pilot was a sensible approach to take, with one respondent saying that the Commission should take advice on the size and the scale of the pilot required to be able to robustly compare results with previous methods. The aim of the pilot should be to give users trust and confidence in the accuracy of the data going forward.

There was some concern about the timing of the pilot given the impact of Covid-19 on gambling behaviours. Respondents thought that results from a 2021 pilot which are compared to results collected pre-Covid (e.g. the Health Survey England 2018) need to be used with caution as differences may be due to changes in the gambling environment as a result of Covid rather than methodological differences. Similarly, comparisons to the 2016 Health Surveys should also be made with caution, given that the fieldwork took place five years ago and the gambling environment has changed since then. Respondents seemed to think that the solution should be comparing the results of the pilot study to a variety of other data sources rather than just one. This should include Gamble Aware's methodology review (opens in a new tab) or the 2021 Health Survey, although it is acknowledged this would cause a delay for the Commission’s methodology review.

One of the respondents who disagreed with the pilot approach thought that rather than piloting a new survey approach we should be overhauling the whole research programme to include the use of administrative data for survey validation.

Finally, a request for the outcome of the pilot to be published was made by an operator.

When asked if a break in the time series of data would impact their work, most respondents were of the mind that it would not. As one respondent said, they have managed with the uncertainty of prevalence estimates for some time (due to mode effects) and therefore if a break in time series means that the data quality and reliability is improved it is worth doing. Another said that they do not use data from the Commission’s quarterly telephone survey because the full dataset is not currently publicly available so for them a break in time series would not be an issue.

One respondent who thought the break in time series would impact their work was concerned that it would impact their work in relation to safer gambling and could lead to an increase in the misuse of data.

Figure 6: Would a break in the time series impact your work?

Base: 59 responses

Figure 6 -  A pie chart showing the 59 responses to the question would a break in the time series impact your work. 36 said no, 7 said yes, and 16 are unsure

Data from pie chart

YesNoUndecided
Number of responses73616

Our position

Given the level of agreement with our proposals in this consultation document we plan to pilot a new approach in 2021/22.

We will run a procurement exercise to find a suitable research partner to lead the pilot, with the pilot having two aims:

  • to test current participation and prevalence questions using a new methodology against data from a range of existing surveys
  • to test new questions aimed at measuring participation using a refreshed and updated list of gambling activities.

We will consider options for the best way for the pilot to achieve the above aims including running a two-stage pilot or a split sample within one pilot survey.

We will take on board findings from the GambleAware research methodology project.

We will publish the outcome of the pilot and our plan for a future survey methodology.

Annex A: List of consultation responses

We received 62 valid responses to the consultation, from the following categories of respondents:

We received 11 responses which had been submitted in relation to another consultation that was active at the time, these have been removed from the responses considered as part of our consultation response.

The following respondents indicated that they gave consent for their name and organisation to be published to indicate their response to the consultation.

  • Roger Nobbs - Member of the public
  • Joe Tupman - Member of the public
  • Charles Ritchie - Gambling with Lives
  • Conor Irving - Member of the public
  • Christina Thakor-Rankin - 1710 Gaming Ltd
  • Liza Hermansen - CEGO A/S
  • John Leake - Member of the public
  • Philip Ellison - Member of the public
  • Keith Brown - Member of the public
  • Alex Guiford - Member of the public
  • Jim Orford - University of Birmingham
  • Toby Daeche - Member of the public
  • Andy Prestidge - Member of the public
  • Andrew Fowles - Member of the public
  • Gary Anderson - Member of the public
  • Stuart Ashman - Member of the public
  • Trevor Pitman - Member of the public
  • Jamie Benson - Member of the public
  • Matt Frith - iBetweb
  • Jon Hughes - Member of the public
  • Stephen Jones - Member of the public
  • Nicola Moss - Market Research Consultancy
  • Miles Baron - Bingo Association
  • Dr Amy Sweet - GambleAware
  • Heather Wardle - University of Glasgow
  • Mari Toomse-Smith - Social research organisation
  • Fiona Dobbie - University of Edinburgh
  • Brian Wallace - Flutter
  • Wes Himes - The Betting and Gaming Council
  • Lauren Hilton - Tote
  • Matt Zarb-Cousin - Clean Up Gambling
  • Bradley Canham - Member of the public
  • Stephen Russell - Member of the public
  • Killian O'Leary - Lancaster University
  • Martin Paterson - Member of the public
  • Patrick Sturgis - London School of Economics and Political Science
  • Michael Driver - Jumbo Interactive
  • Gavin Ellison - YouGov
  • Charles Lound - Individual researcher working at ONS (not representing departmental views)
  • Rachel Volberg - School of Public Health and Health Sciences, University of Massachusetts Amherst