Gambling participation and the prevalence of problem gambling survey: Experimental statistics stage

Executive summary

Experimentation work

The Gambling Commission is undertaking a project to improve the way it collects data on adult gambling participation and the prevalence of problem gambling. The project started in 2021 with a pilot survey and this report summarises the findings from the experimental stage of the project.

The purpose of the experimental statistics phase was to build on the pilot and conduct further testing and refinement, to ensure the survey design and questionnaire content is robust for official statistics continuous data collection. The experimental statistics phase involved three steps:

1. Step 1 - experiments on participant selection and gambling-related harm questions.
2. Step 2 - testing different approaches to asking about gambling participation.
3. Step 3 - final test of agreed approach and content taking on board recommendations from step 1 and step 2 (not yet started).

Subject to the success of the experimental phase, the survey will move, in summer 2023, to continuous official statistics data collection.

This report outlines the experimental stage methodology and analysis for steps 1 and 2. The report also makes recommendations for the next phases.

Response

For step 1, 21,953 addresses were sent an invitation to take part in the survey and a maximum of two or four adults (aged 18 years and over) were invited to take part. In total, 3,563 addresses were productive, yielding 5,275 individual responses.

In total, 44 percent of the responding unweighted sample were men and 56 percent were women. In terms of mode of questionnaire completion, 3,312 (63 percent) were completed online and 1,959 (37 percent) were completed on paper and posted back.

For step 2, invitation letters were sent to 14,982 addresses. In total, 3,549 adults in 2,405 addresses completed the survey. 45 percent of the responding unweighted sample were men and 55 percent were women. 2,228 (63 percent) were completed online and 1,315 (37 percent) were completed on paper and posted back. These are the same proportions found for the step 1 sample.

Assessment of experiment results

In step 1, the analysis looked to establish whether there were significant differences between two experimental conditions – where a maximum of two or a maximum of four adults invited to take part. There was no discernible experimental condition effect on household response rates, duplications nor gambling participation rates. There was evidence of significant clustering of gambling behaviours among households with three or four participants. As this can impact on the accuracy of the gambling participation data the recommendation was that going forward, up two adults per household should be invited to take part in the survey.

A further experiment in step 1 looked at gambling-related harms questions. Those responding 'yes' in condition B (binary answer option) produced similar rates of participants classified as experiencing harms to those responding 'fairly often' or 'very often' in condition A (scaled answer options). Those answering 'yes' on the harms from others questions displayed less consistent patterns of association with personal wellbeing, contrary to expectations. For these reasons combined, we recommend retaining the refined four-point answer scale for the next step of the survey.

When it comes to analysing the data from the scaled answer options, the Gambling Commission will need to carefully assess the extent to which endorsement represents the experience of harm. We concur with Williams and Volberg¹ that those answering 'occasionally' are most likely representing the potential for harm rather than experience of it. If the Commission wanted to explore this further, qualitative follow-up interviews could be undertaken with participants to this survey to explore their experiences.

Step 2 aimed to assess the impact of asking questions about gambling participation in different ways on survey estimates of gambling participation rates. The following three different ways of asking questions about gambling activities participated in over the last 12 months and the last four weeks were assessed:

• long-list approach
• chunked-list approach
• hierarchical-list approach.

No clear picture emerged as to which of the three approaches (long-list, chunked-list and hierarchical-list) performed best in capturing information about gambling participation. Whilst all three approaches had limitations, those of the hierarchical-list and chunked-list approaches were considered more problematic than those of the long-list approach.

On balance, the recommendation was that the long-list approach should be used going forward but that the routing instructions and visual layout of the postal version should be improved. The impact of changes to the postal questionnaire should then be assessed by comparing results of the next phase of data collection with the data from this experiment. Specifically, the goal would be to see a reduction in the levels of item non-response to follow-up questions in the postal questionnaire.

In step 2, two Quick Response (QR) codes were included in the invitation letter and reminders. The QR codes, when scanned, took participants straight into the online questionnaire, bypassing the need to manually enter any access information. Uptake of the QR codes was high: 52 percent of those completing the survey online did so via the QR code though usage decreased with increasing age.

---

¹ Developing survey questions capturing gambling-related harms - Part A - Expert review of question development by Professor Robert Williams and Dr Rachel Volberg, Heather Wardle, Viktorija Kesaite, Robert Williams, Rachel Volberg (2022).

Recommendations

Household selection

There was no discernible experimental condition effect on household response rates, duplications nor gambling participation rates. There was evidence of significant clustering of gambling behaviours among households with three or four participants. As this can impact on the accuracy of the gambling participation data our recommendation is that inviting up to two adults per household to take part, is the preferred option going forward.

Within-household selection bias is an issue across push-to-web survey methodologies (for example, see Community Life Survey: Investigating the feasibility of sampling all adults in the household (opens in new tab)). However, the current gambling survey weighting strategy addresses this by modelling the number of responses per household (for households with more than one eligible adult). This step in the weighting strategy tests a range of variables for association with a number of key survey responses, including the number of eligible adults in the household, to control for bias arising from within-household non-response. In the next phase, we recommend that analysis is carried out to identify if patterns in gambling behaviour are different between those households where no household selection took place and those where there was a within-household selection.

It is important to note that weights do not overcome the bias towards those who gamble identified in the pilot. To do this, one of two sources of information would be needed, either the true number of gamblers in a household or an external source for the number of gamblers which could be calibrated to. As neither of these are available, the focus for reducing the bias is on making the invite and reminder letters more appealing to those who do not gamble.

Gambling-related harms

Those responding 'yes' in condition B (binary answer option) produced similar rates of participants classified as experiencing harms to those responding 'fairly often' or 'very often' in condition A. Those answering 'yes' on the harms from others questions displayed less consistent patterns of association with personal wellbeing, contrary to expectations. For these reasons combined, we recommend retaining the refined four-point answer scale for the next step of the survey.

When it comes to analysing the data from the scaled answer options, the Gambling Commission will need to carefully assess the extent to which endorsement represents the experience of harm. We concur with Williams and Volberg² that those answering 'occasionally' are most likely representing the potential for harm rather than experience of it. If the Commission wanted to explore this further, qualitative follow-up interviews could be undertaken with participants to this survey to explore their experiences.

Further recommendations related to the analysis of gambling harms are:

• the changes to the screening questions to route people into the harms from others questions have made improvements and should be retained
• in the harms from others questions, the answer options for questions asking about harms to health, borrowing money or feeling guilty and/or embarrassed or ashamed should match the answer options for the other questions
• the Commission should consider how it wants to analyse and present the gambling suicidality questions and think about steps to reduce analytical ambiguity
• if the harms to self questions are ever included in a survey which is not also using the Problem Gambling Severity Index (PGSI) instrument, answer options to these three items should also be changed for consistency.

Gambling participation

No clear picture emerged as to which of the three approaches (long-list, chunked-list and hierarchical-list) performed best in capturing information about gambling participation. Whilst all three approaches had limitations, the limitations of the hierarchical-list and chunked-list approaches were considered more problematic than those of the long-list approach.

On balance, the recommendation was that the long-list approach should be used going forward but that the structure of the postal version of the questionnaire and the routing instructions within it should be reviewed and where possible improved.

The impact of changes to the postal questionnaire should be assessed by comparing results of the next phase of data collection with the data from this experiment. Specifically, the goal would be to see a reduction in the levels of item non-response to follow-up questions in the postal questionnaire.

Use of QR codes to access the online survey

In step 2, two Quick Response (QR) codes were included in the invitation letter and reminders. The QR codes, when scanned, took participants straight into the online questionnaire, bypassing the need to manually enter any access information.

Uptake of the QR codes was high: 52 percent of those completing the survey online did so via the QR code though usage decreased with increasing age.

It is recommended that QR codes are retained as an alternative way for participants to access the survey in future phases.

---

² Developing survey questions capturing gambling-related harms - Part A - Expert review of question development by Professor Robert Williams and Dr Rachel Volberg, Heather Wardle, Viktorija Kesaite, Robert Williams, Rachel Volberg (2022).

Introduction

Background

The Gambling Commission exists to safeguard consumers and the wider public by ensuring that gambling is fair and safe. The Commission’s work is underpinned by two main pieces of legislation:

• Gambling Act 2005 (opens in new tab), which sets the framework for the regulation of gambling in Britain
• National Lottery etc. Act 1993 (opens in new tab), which sets out the framework within which the Commission regulates the National Lottery.

Under section 26 of the Gambling Act 2005, the Commission has responsibility for collecting and disseminating information relating to the extent and impact of gambling in Great Britain. In order to do this, the Commission collects gambling participation and problem gambling prevalence data via surveys of adults in Great Britain. The data is published as official statistics, that is produced in accordance with the standards set out by the Government Statistical Service in the Code of Practice for Statistics (opens in new tab).

To date, a variety of data collection approaches have been used to meet this requirement, including as a:

• section of separate Health Surveys for England, Scotland and Wales³
• quarterly telephone survey which supplements the Health Surveys by providing a more regular measure of participation and problem gambling prevalence.

In December 2020, the Commission launched a consultation on gambling participation and prevalence research (opens in new tab) to gather views on proposals to develop a single, high quality methodology to measure gambling participation and prevalence of problem gambling. The aim was to have a more efficient, cost effective data source providing robust and timely insight and the flexibility to swiftly provide information on emerging trends. The results of the gambling participation and prevalence research consultation were published in June 2021.

In October 2021 NatCen Social Research (NatCen), working with the University of Glasgow and Bryson Purdon Social Research, was commissioned to take on the pilot project to test the new data collection methodology in 2021 to 2022.

The pilot was successful in attracting participants and exceeded response rate expectations. Estimates of gambling participation and problem gambling were somewhat higher than those based on the Health Survey for England (HSE) 2018 (opens in new tab), but were lower than those typically generated by online panel surveys and thus broadly commensurate with expectations at this stage.

The analysis highlighted two potential causes of differences. Firstly, it was possible that response rates were higher among those who gambled than those who did not gamble, which in turn may have led to somewhat higher estimates of gambling participation and problematic gambling. Secondly, it appeared that there were differences between the two surveys in the way that survey participants completed the Problem Gambling Severity Index (PGSI), with the differences greatest for women.

Upon its successful evaluation at the pilot stage, the methodology was rolled out in summer 2022 for data collection under experimental statistics. Experimental statistics (opens in new tab) are a subset of newly developed or innovative official statistics undergoing evaluation. The experimental statistics phase was contracted to NatCen, working with the University of Glasgow. Subject to the success of the experimental phase, the survey will move, in July 2023, to continuous official statistics data collection.

---

³ These are large scale face-to-face population surveys where gambling questions are included approximately every two years.

Aims and overview

The purpose of the experimental statistics phase was to build on the pilot and conduct further testing and refinement, to ensure the survey design and questionnaire content is robust for official statistics continuous data collection. The experimental statistics phase involved three steps:

1. Step 1 - experiments on participant selection and gambling-related harm questions.
2. Step 2 - testing different approaches to asking about gambling participation.
3. Step 3 - final test of agreed approach and content taking on board recommendations from step 1 and step 2.

Step 1

In the pilot study, there was some concern that those who gambled were more likely to self-select into the study compared with those who did not gamble. Therefore, the first experiment in step 1 examined the impact (adherence rates) of allowing up to a maximum of four adults aged 18 years and over to take part compared with restricting participation to two adults per household. Further detail and analysis is provided in the Completion rates in responding households section of this report.

The second experiment of step 1 was to assess how the gambling-related harm questions with binary and scaled-answer options performed. In the 2021 pilot study severity answer option scales ('not at all', 'a little' and 'a lot') were used for a number of the gambling-related harm questions. The distribution of responses across the answer options was not as expected. Usually, fewer people would be expected to select answer options linked to higher frequency, with the proportion selecting the answer reducing as the frequency increases. This was not the case in the pilot for these gambling-related harm questions where more participants selected 'a lot' than 'a little' for each question⁴.

Therefore, for this experiment, alternative answer options were tested. Households were pre-assigned to answer either the binary (yes or no) or the frequency scaled-answer options ('very often', 'fairly often', 'occasionally' and 'never') gambling-related harms questions. Further detail and analysis is provided in the Measuring gambling-related harms section of this report.

Step 2

As supported in the gambling participation and prevalence research consultation, one of the original aims for the survey was to review and refresh the gambling participation question that had been commonly used on surveys. The intention was to better reflect the current diversity of gambling products and to facilitate analysis of problem gambling prevalence at a product level. The pilot stage of the project began work reviewing the gambling participation questions by consulting stakeholders and conducting cognitive testing to explore the ways in which people understood the descriptions of gambling activities used in survey questions, identifying any misunderstandings, ambiguities and missing activities.

Step 2 of the experimental statistics stage then examined the best approach for asking questions about gambling participation and whether the way in which participants are asked about gambling participation affects the estimated gambling participation rates. To do this the list of gambling activities used to identify which gambling activities participants took part in the last 12 months was updated and the following three different approaches for asking questions about these activities were tested:

• long-list approach
• chunked-list approach
• hierarchical approach.

Further detail of the approaches and the analysis conducted are provided in the Testing different approaches to asking questions about gambling participation section of this report.

In step 2, the opportunity was also taken to test the use of Quick Response (QR) codes on advance letters whereby participants could scan the QR code to directly access the questionnaire. The aim of the latter was to look at the proportion and profile of online questionnaires completed via QR codes. Further detail is provided in the Methodology and response section of this report.

This report outlines the methodology used in steps 1 and 2 of this experimental statistics phase, provides detail on the response rates achieved and the results from the methodological experiments conducted. The report concludes with recommendations for the next phase.

Following a review of steps 1 and 2, the final step of the experimental statistics phase will be the launch of step 3. Step 3 will provide the opportunity to roll out and test the finalised methodology prior to the official statistics phase. This third and final step will also include planning and setting up the reporting infrastructure for the new survey.

---

⁴ Developing survey questions capturing gambling-related harms, Heather Wardle, Viktorija Kesaite, Robert Williams, Rachel Volberg (2022).

Methodology and response

Survey design

Sampling

A high-quality sample is essential for meeting the Gambling Commission’s aim of creating a robust and nationally representative new survey. To achieve this, a stratified random probability sample of addresses in Great Britain was used. The target population of the survey was adults aged 18 years and over, living in private households within Great Britain, and the aim was to achieve a sample size of 10,000 individuals (6,000 for step 1 and 4,000 for step 2)⁵.

There is no publicly available list of adults that could be used for sampling individuals. However, the Postcode Address File (PAF), compiled by the Post Office, provides a list of postal addresses (or postcode delivery points) which can be used as a sampling frame. The sampling process had two stages:

• selection of addresses from the PAF
• selection of adults within addresses.

Prior to selection, the sampling frame was stratified (ordered), this can help to reduce sampling error and thus increase the precision of estimates, as well as ensuring representativeness with respect to the measures used. The following measures for stratification (in order) were:

• country and English region
• population density at Local Authority level
• overall Index of Multiple Deprivation (IMD) score⁶.

Addresses were split into two selection types (‘conditions’), which specified the maximum number of adults (aged 18 and over) to be selected – by a householder – from each address to complete the survey. In condition 1 addresses, up to two adults were selected; in condition 2 addresses, up to four adults were selected. These two experimental groups are referred to as ‘C1 (up to two adults)’ and ‘C2 (up to four adults)’ in this report. A maximum of four adults was chosen as a practical cut-off given that according to Labour Force Survey data from 2022, one percent of households in Great Britain contain five or more adults.

At each sampled address, there may have been more than one dwelling and/or household. However, a random selection of households is very difficult to operationalise without an interviewer and there was no control over which household opened the invitation letter. As a result, in multi-occupied addresses no formal household selection took place and the selection of which household took part was left to chance (that is whichever household opened the letter). The overall proportion of multi-occupied addresses for PAF samples is very small (around one percent), and it is therefore unlikely to lead to any systematic bias in the responding sample.

Step 1

The aim was to have a total achieved sample size of 6,000 adults aged 18 years and over for step 1. Power calculations assumed that in C1 households where two adults were invited to take part, an average of 1.4 participants would do so. In C2 households where up to four adults were invited to take, an average of 1.5 would do so. The estimate of 1.4 participants per responding household was taken from the pilot survey.

The estimate of 1.5 participants per responding household was taken from the Community Life Survey 2020/21 (opens in new tab). These calculations further assumed a small amount of clustering of gambling behaviour within each participating household. This clustering reduced the effective target sample size from 6,000 to 5,714 (DEFF⁷ 1.05) but this would be large enough to detect differences in gambling behaviour between the two conditions (C1 (up to two adults) and C2 (up to four adults)) of 4.5 percent for past year gambling prevalence covering overall gambling rates, 3.6 percent for online gambling excluding the National Lottery rates, and 4.6 percent for overall gambling excluding the National Lottery rates.

For step 1, the issued sample comprised 21,953 addresses, each randomly allocated to one of three, similar-sized groups as shown in 'Figure 1: Step 1 issued and target achieved sample sizes for experimental conditions and type of gambling-related harms question asked' as follows.

Figure 1: Step 1 issued and target achieved sample sizes for experimental conditions and type of gambling-related harms question asked

Figure 1: Step 1 issued and target achieved sample sizes for experimental conditions and type of gambling-related harms question asked.

Gambling-related harms question asked	Experimental condition C1 (up to two adults) (number)	Experimental condition C2 (up to four adults) (number)	Total (number)
Scaled	Issued addresses: 7,305 Target achieved individuals: 1,953	Issued addresses: 7,343 Target achieved individuals: 2,094	Issued addresses: 14,648 Target achieved individuals: 4,047
Binary	Issued addresses: 7,305 Target achieved individuals: 1,953	Not applicable	Issued addresses: 7,305 Target achieved individuals: 1,953
Total	Issued addresses: 14,610 Target achieved individuals: 3,906	Issued addresses: 7,343 Target achieved individuals: 2,094	Issued addresses: 21,953 Target achieved individuals: 6,000

Step 2

For step 2, 14,982 addresses were issued with the aim of achieving 4,000 productive individual questionnaires. The sampled addresses were randomly allocated to one of three equal sized groups. Each group had 4,994 issued addresses with the aim of achieving 1,333 productive individual questionnaires. The three groups were then asked a different set of gambling questions (a long-list approach, a hierarchical approach and a chunked-list approach).

Figure 2: Step 2 issued and target achieved sample sizes for experimental approaches to asking about gambling participation

Figure 2: Step 2 issued and target achieved sample sizes for experimental approaches to asking about gambling participation.

Sample	Experimental approach: long-list (number)	Experimental approach: hierarchical (number)	Experimental approach: chunked-list (number)
Issued addresses	4,994	4,994	4,994
Target achieved individuals	1,333	1,333	1,333

---

⁵ The target achieved sample size for step 3 of the experimental stage, the soft launch, will be 4,000 individuals.

⁶ Indices of Multiple Deprivation (IMD) is a measure of relative deprivation for small, fixed geographic areas of the United Kingdom (UK). Separate indices are produced for each UK country. IMD classifies these areas into five quintiles based on relative disadvantage, with quintile one being the most deprived and quintile five being the least deprived.

⁷ The Design Effect (DEFF) is a measure that summarises the degree of clustering that has occurred. It is the ratio of the variance between individuals in the (clustered) sample, compared with the variance that would be expected from a simple random sample.

Questionnaire content and design

The online mode was supplemented by a postal questionnaire follow up to enable less technologically literate people, those without internet access and those who preferred an alternative approach, to respond.

This step is essential for the new gambling survey as some gambling behaviours, notably the propensity to gamble online, is correlated to the probability to take part in an online survey and would therefore lead to biased results⁸. The pilot survey also demonstrated the importance of the postal option as 43 percent of completions were by this mode.

In addition to the questions being tested as part of the previously mentioned experiments, the questionnaires included content on:

• leisure activities, internet access and use
• online games and gambling adjacent activities (step 2 only)
• Problem Gambling Severity Index (PGSI)
• why people gamble and how it makes them feel (step 1 and online only for step 2)
• attitudes towards gambling (ATGS-8), gambling management tools and complaints (step 2 only)
• advertising and social media
• health and wellbeing, including general health, smoking and drinking status, impulsivity scale, Short Warwick-Edinburgh Mental Well-Being Scale (SWEMWBS) and suicidality questions
• demographic questions.

The online and postal questionnaires for step 1 of the survey are provided in Appendix A - Step 1 web questionnaire (PDF) and Appendix B - Step 1 paper questionnaires (PDF) of this report. The online and postal questionnaires for step 2 are provided in Appendix D - Step 2 web questionnaire (PDF) and Appendix E - Step 2 paper questionnaires (PDF) of this report.

Analysis of some of the previous questions will be presented in short topic reports, to be published in summer 2023.

Mailing strategy

The following overall participant engagement strategy was used, each item was sent to selected addresses in the post:

• invitation letter with the URL (Uniform Resource Locator) and sets of login details (two or four according to the assigned experimental condition) needed to access the survey online. Step 2 letters also contained Quick Response (QR) codes as an alternative method of accessing the survey. A Welsh version of the letter was also sent to addresses in Wales
• first reminder letter
• second reminder letter with two postal questionnaires⁹ and return envelopes
• third reminder letter.

The invitation letter and reminders, provided in Appendix C - Step 1 invitation and reminder letters (PDF) and Appendix F - Step 2 invitation and reminder letters (PDF), were the main levers to convince people to take part. The documents used in the pilot stage were reviewed and updated with the relevant participant selection criteria and also, the wording was strengthened to emphasise that those who did not gamble were invited to complete the survey.

All were carefully designed following the latest best practice and following the participant engagement guidance for online surveys published by the Office for National Statistics (ONS) (opens in new tab), drawing on their extensive testing in this area.

Experience shows that most people complete a survey within few days of receiving the request. The time between each mailing was therefore kept as short as possible, to ensure that the request was fresh in people’s mind. A gap of around 10 days between mailings was introduced, to allow removal of responding participants from the sample for the reminders. The day of the week of the mailing was varied to allow for the fact that different people may have time for survey participation on different days of the week.

A study website, freephone number and dedicated email address were set up for participants to contact with issues or queries. A £10 pounds completion incentive per individual questionnaire was offered. All online responders were emailed a 'Love2Shop' voucher code and postal responders were posted a voucher¹⁰.

Data preparation and checks

As described in previous sections, data was collected from two sources: an online questionnaire and a postal questionnaire. The online questionnaire included built-in routing and checks, whereas the postal questionnaire relied on correct navigation by participants and there was no constraint on the answers they could give.

The online questionnaire data in its raw form were available immediately to the research team. However, the postal questionnaire data had to be manually recorded as part of a separate process.

A number of rigorous quality assurance processes were utilised when preparing the survey data. These included checks that variables from the two data collection modes had merged correctly into one dataset. As up to four adults per household could answer demographic questions relating to the whole household (for example, household size and information about income), there was potential for differing responses between individuals.

The following rules for harmonising household responses were followed, in priority order:

• taking the most common valid answer (such as excluding 'don’t know', 'refusal')
• taking the valid answer from the oldest household member: or where this was not clear, the response of the first household member to complete a questionnaire (online completions first then postal completions).

A further step involved identifying and removing duplicate responses. For this, questionnaires were checked to see if responses to up to two or up to four questionnaires (depending on the experimental condition) were very likely to be from the same individual in a household (based on exact matches for the age, sex and name provided). Suspected duplicates were removed so that only one completed questionnaire from that individual was retained.

Where a household had more than the maximum number of records (two or four depending on the experimental condition), any extra cases were removed according to the following rules:

• fully completed online questionnaires took priority over postal questionnaires
• fully completed postal questionnaires took priority over partially completed online questionnaires
• partially completed online questionnaires took priority over partially completed postal questionnaires
• identifying and removing ‘speeders’ (individuals who completed the online questionnaire in an unrealistic amount of time for them to have properly engaged with the questions)¹¹.

The data were then weighted to allow for comparisons with other data sources. The weighting strategy is outlined in Appendix G - Weighting technical summary (PDF).

---

⁸ How survey mode affects estimates of the prevalence of gambling harm: a multisurvey study (opens in new tab), Public Health, 204, 63-69, P. Sturgis and J. Kuha (2022).

⁹ This included C2 addresses where up to four adults were asked to participate.

¹⁰ 'Love2Shop' vouchers cannot be exchanged for cash and cannot be used for gambling, so do not pose ethical problems for this survey.

¹¹ 'Speeders' were identified by calculating the median time it took to answer each question among all those who answered. From this an expected time was calculated for each participant dependent on the questions that they answered. A ratio of actual time compared with expected time was produced and any statistical outliers on this ratio measure were removed.

Response to the survey

You can view tables referenced in this section by downloading the file Tables A1 to A48 - Gambling Survey - Experimental statistics stage (XLSX)

Step 1 response

Address-level response rates

Table A.1 (Step 1 address-level response) summarises step 1 address-level response rates.

In total 21,953 addresses were issued. In remote surveys (that is where participants complete the survey independently without any involvement from an interviewer), no information is known about the reason for non-response in individual addresses. However, it was assumed that around 9 percent of addresses in the sample (1,976) were not residential and were therefore ineligible to complete the survey¹².

In total, 3,563 addresses were productive. The adjusted address-level response rate, that is the proportion of eligible addresses where a questionnaire was completed by at least one adult in eligible addresses, was 18 percent (lower than the target of 22 percent). There was no response from 16,226 of the addresses assumed to be eligible and an individual from a further 188 addresses contacted the office to say they did not wish or were unable to take part.

Table A.2 (Step 1 address-level response, by experimental condition) summarises the response rate for households, split by the experimental condition (such as whether a maximum of two or four adults were invited to take part).

In total, 2,396 C1 (up to two adults) addresses were productive. This gives a household response rate of 18 percent for the assumed eligible addresses, under the target of 22 percent. Of these 2,396 productive households, 1,276 (53 percent) yielded one fully productive adult and 1,120 (47 percent) yielded two fully productive adults.

In terms of C2 (up to four adults) addresses, 1,167 were productive. This gives a household response rate of 17 percent for the assumed eligible addresses, similar to that achieved for the C1 (up to two adults) addresses. Of these 1,167 productive households, 668 (57 percent) yielded one fully productive individual participant. 408 (35 percent) yielded two, 89 (8 percent) yielded three and two households (less than 0.5 percent) yielded four fully productive adults.

Table A.3 (Step 1 address-level response, by country) shows the breakdown of the issued step 1 sample in England, Scotland and Wales and the household response rate in each country.

In total, 86 percent of the issued addresses were in England, 9 percent in Scotland and 5 percent in Wales. Household responding rates were very similar across all three countries: 18 per cent in England, 18 per cent in Scotland and 19 per cent in Wales.

In terms of the English regions, the highest response rates were achieved in the South West (21 percent) and East Midlands (20 percent) and the lowest in London (13 percent) (Table A.4, Step 1 address-level response, by Government Office Region).

As stated previously, power calculations assumed that in C1 households where two adults were invited to take part, an average of 1.4 participants would do so. In C2 households where up to four adults were invited to take, an average of 1.5 would do so. As can be seen in Table A.5 (Step 1 experimental condition means), these assumptions were borne out in step 1: C1 (up to two adults) households had an average of 1.47 completions whilst C2 (up to four adults) had an average of 1.51.

Individual-level response rates

Following the process of removing duplicate responses, cases deemed to have completed the online questionnaire too quickly to have properly engaged with the questions and cases above the maximum defined number of completions (two or four) for that household, it was assumed that all responses in the dataset were from 5,275 unique individuals who had completed the questionnaire.

Table A.6 (Step 1 individual response, by age and sex) shows the age and sex profile of the 5,275 responding adults who completed the step 1 questionnaire (2,336 men, 2,935 women and four participants who did not respond to the age and/or sex questions).

In total, 44 percent of the responding unweighted sample were men and 56 percent were women. This under-representation of men is similar to that seen in the latest published results for other surveys with the same completion modes, for example, the Public Confidence in Official Statistics (PCOS) 2021 by the National Centre for Social Research (opens in new tab). In total, 46 percent of the PCOS 2021 unweighted main sample were men and 54 percent women.

Those in the younger age groups were less likely to take part than their older counterparts: 6 percent of responding adults were aged 18 to 24 years (this age group makes up 10 percent of the adult population of Great Britain) and 19 percent were aged 65 to 74 years (this age group makes up 13 percent of the adult population of Great Britain).

This difference was particularly pronounced for men: 5 percent of the male sample were aged 18 to 24 years (this age-sex group makes up 11 percent of the male adult population of Great Britain). The equivalent proportions for women aged 18 to 24 years were 7 percent and 10 percent. The national percentages are based on the 2021 mid-year population estimates for Great Britain: Estimates of the population for the UK, England, Wales, Scotland and Northern Ireland by the Office for National Statistics (ONS) (opens in new tab).

As recommended from the pilot, the minimum age of completion was raised from 16 to 18 years of age. Whilst this youngest age group are still underrepresented; it was less so than was seen in the pilot.

Table A.7 (Step 1 individual response by mode of completion, experimental condition and sex) summarises the individual response rate for the two completion modes (online and postal), split by experimental condition (that is whether up to two or up to four adults were asked to take part in the survey), and by sex.

Of the 5,275 questionnaires included in this analysis: 3,316 (63 percent) were completed online and 1,959 (37 percent) were completed on paper and posted back. The proportion of participants completing the questionnaire online was very similar across the two experimental conditions, 63 percent in the C1 (up to two adults) addresses and 62 percent in the C2 (up to four adults) addresses.

Step 2 address-level response rates

Address-level response rates

Table A.8 (Step 2 address-level response) summarises the address-level response rates for step 2. At this step up to two adults were asked to take part in all 14,982 issued addresses.

In total, 1,348 addresses (9 percent of the issued sample) were assumed to be ineligible (non-residential) and 2,405 addresses were productive. The adjusted address-level response rate, that is the proportion of eligible addresses where a questionnaire was completed by at least one adult in eligible addresses, was 18 percent (same as the response rate achieved at step 1 but lower than the target of 22 percent).

There was no response from 11,057 of the addresses assumed to be eligible and an individual from a further 172 addresses contacted the office to say they did not wish or were unable to take part.

Table A.9 (Step 2 address-level response, by country) shows the breakdown of the issued step 2 sample in England, Scotland and Wales and the household response rate in each country.

In total, 86 percent of the issued addresses were in England, 9 percent in Scotland and 5 percent in Wales. Household responding rates were similar across all three countries: 18 per cent in England, 19 per cent in Scotland and 16 per cent in Wales.

In terms of the English regions, the highest response rates were achieved in the South East (20 percent) and South West (20 percent) and the lowest in London (13 percent) (Table A.10 Step 2 address-level response, by Government Office Region).

Individual-level response rates

Following the processes of removing duplicate responses, cases deemed to have completed the online questionnaire too quickly to have properly engaged with the questions and cases above the maximum defined number of completions (two adults), it was assumed that all responses in the dataset were from 3,549 unique individuals who had completed the questionnaire.

Table A.11 (Step 2 individual response, by age and sex) shows the age and sex profile of the 3,549 responding adults (1,594 men, 1,949 women and six participants who did not respond to the question) who completed the questionnaire.

In total, 45 percent of the responding unweighted sample were men and 55 percent were women. Those in the younger age groups were less likely to take part than their older counterparts: 6 percent of responding adults were aged 18 to 24 years (this age group makes up 10 percent of the adult population of Great Britain) and 19 percent were aged 65 to 74 years (this age groups makes up 13 percent of the adult population of Great Britain).

The Great Britain population estimates are based on the 2021 mid-year data from the Estimates of the population for the UK, England, Wales, Scotland and Northern Ireland by the Office for National Statistics (ONS). This difference was again particularly pronounced for men: 5 percent of the male sample were aged 18 to 24 years (this age-sex group makes up 11 percent of the male adult population of Great Britain). The equivalent proportions for women aged 18 to 24 years were 6 percent and 10 percent.

Table A.12 (Step 2 individual response, by mode of completion and sex) summarises the individual response rate for the two completion modes (online and postal).

Of the 3,543 questionnaires included in this analysis: 2,228 (63 percent) were completed online and 1,315 (37 percent) were completed on paper and posted back. These are the same proportions found for the step 1 sample.

There was no difference between men and women in the mode of completion (63 percent for online completions and 37 percent for postal completions for both groups). However, there was a marked difference according to age with the percentage of those completing the survey online decreasing with age (and hence the percentage completing the postal questionnaire increasing with age).

In all age groups, except the oldest two (aged 65 to 74 years and aged 75 years and over), a higher proportion completed the survey online rather than filling in the postal questionnaire. In total, 29 percent of those aged 75 years and over and 40 percent of those aged 65 to 74 years completed the survey online compared with between 58 percent and 90 percent of those in the younger age groups (Table A.13 Step 2 individual response, by mode of completion and age).

Use of Quick Response (QR) codes to facilitate online participation

The invitation and reminder letters for the 2021 pilot and step 1 of this experimental statistics phase included a short survey Uniform Resource Locator (URL) and individual eight-character access codes for participants to manually enter into a browser and access the survey.

In step 2, two QR codes were also included as an alternative way for participants to access the survey. The QR codes, when scanned, took participants straight into the online questionnaire, bypassing the need to manually enter any access information.

Uptake of the QR codes was high: 52 percent of those completing the survey online did so via the QR code and 48 percent accessed the survey via manually entering the survey URL and typing in the access code.

As discussed in the previous section, the percentage of those completing the survey online decreased with age. Within online completions, this pattern was replicated for the use of QR codes: 73 percent of those aged 18 to 24 years accessed the survey via a QR code compared with 19 percent of those aged 75 years and over ('Figure 3: Age profile and access mode of those completing the survey online' as follows).

Figure 3: Age profile and access mode of those completing the survey online

Figure 3: Age profile and access mode of those completing the survey online.

Access and completion mode	18 to 24 years of age (percentage)	25 to 34 years of age (percentage)	35 to 44 years of age (percentage)	45 to 54 years of age (percentage)	55 to 64 years of age (percentage)	64 to 75 years of age (percentage)	75 plus years of age (percentage)	Total (percentage)
Via QR codes	73%	66%	65%	56%	40%	20%	19%	52%
Via URL and access code	27%	34%	35%	44%	60%	80%	81%	48%
Bases (unweighted) (number)	179	436	466	400	363	264	121	2,229

---

¹² When estimating the proportion of ineligible addresses on an online survey, it is best practice to assume the same ineligibility rate as a recent face-to-face survey which uses the same sampling frame and sampling approach and for which detailed outcomes are known for the entire issued sample. Ineligibility rates in Postcode Address File (PAF) face-to-face surveys tend to fall between 8 percent and 10 per cent. 9 percent is the rate recorded in the most recent face-to-face British Social Attitudes Survey (2019) and has been used as an appropriate default for this survey.

Testing an alternative approach to the selection of participants within households

Introduction

In the pilot study, up to two individuals per household were invited to take part. The minimum age for participation was 16 years of age in order to maintain consistency with the Health Survey for England (HSE). As only eight individuals aged 16 or 17 years took part in the pilot, a recommendation was made to increase the minimum age to 18 years of age for the experimental stage.

Results from the pilot suggested that within responding households, those who gambled were more likely to take part than those who did not gamble, possibly due to the salience of the survey topic (that is because the survey is about gambling, it is disproportionately attractive to those who gamble).

A further source of suspected non-response bias towards those who gambled related to household participation having been restricted to a maximum of two adults, potentially creating a within-household selection bias whereby those most interested in gambling took part.

A recommendation was made to undertake further work to better understand this suspected non-response bias and make changes to attempt to reduce it. This section provides results from a methodological experiment, conducted in step 1, which investigated whether asking up to four, rather than two, adults in a household would reduce the potential for within-household selection bias (and capture more of those who did not gamble).

Issued addresses were randomly allocated to one of two experimental conditions. In the first experimental condition, up to two adults per household were invited to complete the survey. This was the method used for the pilot but in an attempt to reduce selection bias and improve random selection, instructions were added to the invitation letter stating that where the household contained three or more adults, the two adults with the most recent birthdays should complete the survey. This group is referred to as ‘C1 (up to two adults)’ in this report.

In the second experimental condition group, up to four adults per household were asked to complete the survey. No participant selection instructions were given so if the household contained five or more eligible adults, any four could take part. This group is referred to as ‘C2 (up to four adults)’ in this report.

The analysis looked to establish whether there were significant differences between the two experimental conditions which could reduce survey data quality, impact key survey statistics and contribute towards selection bias. Hence, the analysis carried out looked at whether there were differences between the two conditions in terms of:

• completion rates in responding households
• adherence to participant-selection instructions
• rates of duplicate cases
• rates of in-household clustering of gambling behaviours
• the prevalence of certain gambling behaviours (gambling participation and problem gambling).

Completion rates in responding households

You can view tables referenced in this section by downloading the file Tables A1 to A48 - Gambling Survey - Experimental statistics stage (XLSX)

The two experimental conditions had different types of participant burden. In C1 (up to two adults) households, the person opening the initial invitation letter was asked to complete a within-household selection and invite the selected (up to two) adults to take part. In C2 (up to four adults) households, they were also potentially asked to distribute the survey details to more (up to four) adults in the household.

The previous Sampling section of this report provides full detail on the response rates achieved in step 1, overall and between the two experimental conditions. As was shown, there was no significant difference between the two experimental conditions in terms of the household-level response rate (18 percent of assumed eligible addresses for C1 (up to two adults) and 17 percent of assumed eligible addresses for C2 (up to four adults)). This was not unexpected given that whilst the types of participant burden may have differed, it is not clear that one condition should be more burdensome than the other.

For both experimental conditions, the majority of responding households had a single participant: 53 percent of C1 (up to two adults) and 57 percent of C2 (up to four adults) households. The larger percentage of single-participant households for C2 (up to four adults) could indicate that the instructions are counterproductive and are failing to persuade more people within the household to take part (Table A.14 Completions per responding household, by experimental condition).

A further indication that increasing the number of adults who may take part in the survey does not necessarily mean those adults will take part is shown in Table A.15 (Responding adults within productive households, by household size and experimental condition). This table shows that as the household size increased, response to the survey did not equally increase. This was most notable for C2 (up to four adults) households where 0 percent of four eligible adult households completed four questionnaires.

Adherence to participant-selection instructions

You can view tables referenced in this section by downloading the file Tables A1 to A48 - Gambling Survey - Experimental statistics stage (XLSX)

Adherence refers to whether the household had correctly followed instructions on who should complete the survey. This was established by comparing the number of adults within the household completing the survey with the number of stated and eligible adults. In a single-adult household we would expect one participant and in a two-adult household we would expect two participants. For C2 (up to four adults) addresses only, in a three-adult household, three participants would be expected and in a four-plus adult household, four participants would be expected.

Non-adherence is where fewer participants than the number of intended participants (for example only one participant completed in a two-adult household) were recorded¹³. In households with multiple (potential) participants it is harder to get all eligible individuals to take part and for that reason we might expect adherence to be greater in C1 (up to two adults) households.

For C1 (up to two adults) households, 74 percent of households were adherent meaning that all eligible participants took part. In C2 (up to four adults) households, 61 per cent were adherent. This difference was statistically significant (Table A.16 Whether the household adhered to participant selection instructions, by experimental condition).

---

¹³ As birthdays for all household members were not collected in the questionnaire, it was not possible to conduct any analysis for adherence to the instruction that those with the two most recent birthdays were the ones to take part within the household selection for C1 (up to two adults) addresses.

Duplicate cases

On occasion, a household member completed the survey more than once, either by the same or a different mode. These cases are called duplicates. Potential duplicates were manually reviewed to determine whether responses to multiple questionnaires were very likely to be from the same individual in a household (based on exact matches for the age, sex and name provided).

Each responding adult, up to the maximum number permitted for the experimental condition, was given a £10 pounds incentive. Arguably this incentivises participants to make up household members and/or questionnaires to gain an additional incentive.

In total 78 duplications (1.5 percent of total completions) were identified. Duplications accounted for 1.4 percent of C1 (up to two adults) completions and 1.7 per cent of C2 (up to four adults) completions but the difference was not statistically significant. Even with the increased opportunity for C2 households to complete more interviews and claim an additional incentive, this did not seem to be the case (Table not shown).

Household clustering

You can view tables referenced in this section by downloading the file Tables A1 to A48 - Gambling Survey - Experimental statistics stage (XLSX)

Measures of household clustering are sensitive to differences in the numbers of completions per household. If more adults take part in multi-adult households, the degree of clustering will increase because some of the additional participants will have similar behaviours to their fellow participating householders. This is important to bear in mind when interpreting these results. As shown previously, C1 (up to two adults) households had an average of 1.47 completions whilst C2 (up to four adults) had an average of 1.51.

The Design Effect (DEFF) is a measure that summarises the degree of clustering that has occurred. In technical terms it is the ratio of the variance between individuals in the (clustered) sample, compared with the variance that would be expected from a simple random sample. A higher DEFF shows a greater degree of clustering of gambling behaviour within households that is more similarity between participants in multi-response households (other things remaining equal).

For each of the activities there was a greater degree of clustering for C2 (up to four adults) households compared with C1 (up to two adults) households. This would be expected given that C2 (up to four adults) households had a larger mean of responses per household (1.51 completions compared with 1.47 for C1 (up to two adults) households). This increased clustering is most notable for those who took part in online gambling (Table A.17 (Design Effect (DEFF) for gambling activities in the last 12 months, by experimental condition) shows the DEFF for gambling activities by the two experimental conditions).

One aim of the experiment was to investigate whether allowing more than two participants per household would attract more of those who did not gamble to complete the survey, particularly in households with at least one person who gambled.

Clustering was much higher in these larger households than it was in the sample as whole. This suggests that when more completions were allowed, there was a tendency for more people with similar gambling behaviours to take part. Notably, the largest difference in the degree of clustering compared with the sample as a whole is for 'any online gambling other than National Lottery within the last 12 months', demonstrating that those who took part in this activity had a higher propensity to respond to the survey (Table A.18 (Design Effect (DEFF) for gambling activities in the last 12 months) shows the DEFF for households with three or four participants (these are only from experimental condition C2).

Prevalence of gambling behaviours

The 2021 pilot survey reported a higher prevalence of gambling relative to the Health Survey for England (HSE) 2018. There appeared to be a response bias with the pilot study being more attractive to those who gambled, overestimating the prevalence within the population. This part of the analysis looked to establish whether there were significant differences between the two experimental conditions which could impact key survey statistics on gambling participation in the last 12 months.

There was no statistically significant difference between C1 (up to two adults) and C2 (up to four adults) households in the three last 12 months participation measures considered (any gambling, any gambling excluding the national lottery only; any online gambling) ('Figure 4: Gambling participation in the last 12 months, by experimental condition' as follows).

Figure 4: Gambling participation in the last 12 months, by experimental condition

Figure 4: Gambling participation in the last 12 months, by experimental condition.

Gambling participation in the last 12 months	Experimental condition C1 (up to two adults) (percentage)	Experimental condition C2 (up to four adults) (percentage)
Any gambling	62.1%	62.0%
Any gambling other than the National Lottery	47.4%	47.9%
Any online gambling other than the National Lottery	13.7%	15.2%

Figure 4 information

Note: Responses in the chart do not add up to 100 percent as they represent NET percentages of those participating in a number of different activities.

'Figure 5: Gambling activities in the last 12 months by experimental condition and adherence to participation-selection instructions' as follows, expanded this analysis to include adherence (such as whether adherence to participant-selection instructions further impacts gambling participation rates). Those from non-adherent households (irrespective of experimental condition) had higher gambling participation rates than those from adherent households but the differences were not statistically significant.

Figure 5: Gambling activities in the last 12 months by experimental condition and adherence to participation-selection instructions

Figure 5: Gambling activities in the last 12 months by experimental condition and adherence to participation-selection instructions.

Gambling participation in the last 12 months	Experimental condition C1 (up to two adults): adherent (percentage)	Experimental condition C2 (up to four adults): adherent (percentage)	Experimental condition C1 (up to two adults): non-adherent (percentage)	Experimental condition C2 (up to four adults): non-adherent (percentage)
Any gambling	62.4%	62.0%	66.5%	63.8%
Any gambling other than the National Lottery	47.4%	48.0%	51.7%	50.5%
Any online gambling other than the National Lottery	13.7%	14.0%	15.7%	19.1%

Figure 5 information

Note: Responses in the chart do not add up to 100 percent as they represent NET percentages of those participating in a number of different activities.

Conclusion

The aim was to establish whether there were significant differences between the two experimental conditions (that is whether a maximum of two or four adults from each household were invited to take part in the survey), which could have a detrimental effect on survey data quality and selection bias.

There was no discernible experimental condition effect on household response rates nor on duplications. However, there was a statistically significant difference in rates of adherence to participant-selection instructions with adherence among C2 (up to four adults) households being lower than in C1 households where up to two adults were invited to take part. This suggests that there was increased participant burden for the C2 households. However, both conditions showed that it was difficult to get all eligible adults to take part, particularly the final adult, and this was even more so when the number of eligible adults was increased.

It would be ‘worth’ this increased burden if it appeared that the C2 (up to four adults) condition performed better in minimising non-response among those who did not gamble. There is no evidence of this. In fact, to the contrary, there is evidence that those in C2 households who gambled did so on more activities (despite prevalence rates being similar) and that there is significant clustering of gambling behaviours among C2 households with three or four participants.

Recommendations for participant selection and reducing selection bias are detailed in the Recommendations section of this report.

Measuring gambling-related harms

Introduction

The Gambling Commission has been developing a series of survey questions aimed at measuring gambling-related harms. This includes measurement of harms experienced because of one’s own gambling (harms to self) and the harms experienced due to someone’s else gambling (harms from others). These questions were tested in the Participation and Prevalence: Pilot methodology review report.

The questions tested include measurement of 14 different types of harms and were split into two types.

Type 1 harms are those deemed to be so serious that experience of this even once would be detrimental to individuals, communities and society. This includes relationship breakdown, losing something of significant financial value; violence or abuse and crime.

Type 2 harms are those which if experienced frequently are likely to be harmful but where experience of this once or twice may not necessarily be harmful or may serve to indicate potential risk for future harms. This includes spending less on everyday items, increased use of credit or savings to gamble, experience of conflict within relationships, feeling isolated, and lying about the extent of gambling and poor work performance or work absences.

In the pilot, participants were directed to answer yes or no for type 1 harms. For type 2 harms, participants were asked to report if they had 'never' experienced this, if they had experienced this 'a little', or if they had experienced this 'a lot'.

Analysis of the pilot data showed issues with the type 2 (scaled) answer responses. Response patterns did not conform to expectations that endorsement would decline as severity increased, yet this pattern was not observed in the data. A subsequent expert review of the Commission’s procedures undertaken to date¹⁴ noted the unequal spacing between answer options: it is a small increment to move from 'never' to 'a little' but a much larger increment to move from 'a little' to 'a lot'. It was recommended that these answer options be changed to a more standard, and evenly spaced, response scale.

The pilot report also recommended that further experimental work be undertaken to better assess patterns of response for each type 2 harm using the revised answer options. In addition, the harms questions were updated and refined after expert review to take on board recommendations.

---

¹⁴ Developing survey questions capturing gambling-related harms, Heather Wardle, Viktorija Kesaite, Robert Williams, Rachel Volberg (2022).

Testing different ways of asking about gambling harms

Step 1 of the experimental statistics data collection included an experiment to test the performance of two different ways of asking the type 2 harms questions. This aimed to compare responses to the harms questions when: a) measured on a four-point scale ranging from never, occasionally, fairly often, very often, and, b) when measured using binary yes or no answer options.

The harms asked about were:

• reducing or cutting back on spending on everyday items such as food, bills and clothing
• using of savings or increasing use of credit to gamble
• experiencing conflict or arguments with friends, family and/or work colleagues
• feeling isolated from other people, left out or completely alone
• lying to family, or others, to hide the extent of gambling
• being absent or performing poorly at work or study.

Addresses (and responding individuals within those addresses) were randomly allocated to one of two experimental conditions:

• condition A - participants at 14,648 addresses were asked to answer the six harm questions using the four-point scaled answer options
• condition B - participants at 7,305 addresses were asked to answer the six harms questions using the binary yes or no answer options.

The issued sample size was different for the two conditions as the household selection and gambling harms experiment ran simultaneously and the aim was to achieve a total sample size of 6,000 completed questionnaires¹⁵.

Questions relating to harms experienced because of one’s own gambling were asked of those who had gambled in the last 12 months (2,183 individuals from condition A and 1,090 from condition B). Questions relating to harms experienced because of someone else’s gambling were asked of anyone who reported that someone close to them (friend, family, partner etc.) gambled (2,006 individuals from condition A and 985 from condition B).

Figure 6: Response to the gambling harms questions for each experimental condition

Figure 6: Response to the gambling harms questions for each experimental condition.

Type of harm	Experimental condition A (four-point answer option) (number)	Experimental condition B (binary answer options) (number)
Harms to self	2,183	1,090
Harms from others	2,006	985

The experiment aimed to assess the following questions:

1. Did the refined four-point answer scale give improved quality data compared with the previous three-point answer scale?
2. How did rates of response patterns compare between those who answered the four-point answer scale and those who answered the binary answer options?
3. What associations were there between reporting of these harms and known correlates of gambling harms?

In addition, the number of participants reporting that someone close to them gambled and the gambling-related suicidality questions were examined.

---

¹⁵ For detail on the step 2 sample, see previous 'Figure 1: Step 1 issued and target achieved sample sizes for experimental conditions and type of gambling-related harms question asked'.

Results

You can view tables referenced in this section by downloading the file Tables A1 to A48 - Gambling Survey - Experimental statistics stage (XLSX)

To look at the performance of the question with four answer categories (condition A), the basic patterns of responses to each question were first examined. The proportion of missing data (that is, individuals choosing not to answer this question for whatever reason) were looked at and compared with missing data for the binary answer options.

Response patterns

For harms to self questions, the response pattern was as expected, with fewer participants choosing more frequent answer categories. For example, 95 percent of participants answering these questions reported that they never felt isolated, whereas three percent reported that they occasionally felt this, one percent reported that they felt this fairly often and one percent reported they felt this very often. The same pattern was evident for men and women ('Figure 7: Responses to the harms to self-questions, condition A (scaled answer options)' as follows as well as, Table A.19, Response pattern to gambling harms to self, scaled answer options questions)).

Figure 7: Responses to the harms to self questions, condition A (scaled answer options)

Figure 7: Responses to the harms to self questions, condition A (scaled answer options).

Response pattern for scaled answer options: harms to self questions	Responses to the harms to self questions: Never (percentage)	Responses to the harms to self questions: Occasionally (percentage)	Responses to the harms to self questions: Fairly often (percentage)	Responses to the harms to self questions: Very often (percentage)
Reduced spending	92.4%	4.1%	1.6%	0.7%
Uses savings and/or borrows money to gamble	94.2%	2.9%	1.1%	0.6%
Conflict with others	94.7%	2.6%	0.8%	0.8%
Feels isolated	94.7%	2.7%	0.9%	0.6%
Lies to family	93.7%	3.5%	0.9%	0.9%
Absent from work and/or poor performance at work	95.5%	1.9%	0.6%	0.7%

Figure 7 information

Note: 'Don't know' and 'refused' options are not shown on chart or in table, hence responses do not sum to 100 percent.

Missing data

For the harms to self questions, similar proportions of participants in the two conditions did not answer the questions (between 1.1 percent and 1.7 percent of those eligible to answer the questions did not). This level of non-response is lower than observed in the pilot, where between three and four percent of eligible participants did not answer these questions.

For the harms from others questions, the extent of missing data was minimal, with just 0.1 percent to 0.2 percent of participants who were eligible to answer these questions not doing so. There were no differences by condition (Tables A.21 Response pattern to gambling harms to self, scaled answer options questions and Table A.22 Response pattern to gambling harms to self, binary answer options questions). This replicates findings from the pilot which also had very low non-response to these questions.

Comparison of rates of experiencing harms

With scaled answer options, there are different ways that participants can be categorised as either experiencing the harm or not experiencing the harm. Participants could be classified as experiencing harm when they:

1. At least occasionally experienced each harm.
2. Fairly often or very often experienced each harm.
3. Experienced each harm very often.

The proportion of participants classified as experiencing harms according to these three options were compared with those who said 'yes' when answering the binary answer options.

‘Figure 8 Prevalence of harms to self, by response options’ as follows, shows participants classified as experiencing harms across the four options for the harms to self questions. Including those who say they experience harms occasionally broadly doubles the rate of harms for each item compared with those answering 'yes' in condition B. However, the actual impact of occasionally experiencing each harm is unclear. These may represent fairly minor harms for some or, following suggestions by external reviewers, Prof Robert Williams and Dr Rachel Volberg¹⁶, could indicate the potential for harm rather than the experience of it.

Classifying harms as experiencing each one either fairly often or very often gives similar rates of endorsement to those responding yes in condition B. For example, in condition A, 2.4 percent of those who gambled said they 'fairly often' or 'very often' reduced their spending on other things to gamble compared with 2.8 percent of those who gambled and answered 'yes' to this question in condition B.

The largest difference between these two definitions was observed for lying to family and/or others to hide the extent of one’s gambling. In condition B, 3.1 percent of eligible participants reported this compared with 1.5 percent reporting this 'fairly often/very often' in condition A. For this harm, it appears the 'yes' answer option includes more individuals who might otherwise report doing this 'occasionally'. For the other harms considered, data suggest that the majority of those who may have otherwise reported doing this occasionally instead selected 'no' when faced with a binary answer option.

Finally, prevalence of harms was lowest when looking at those who reported experiencing each 'very often', falling between 0.6 percent and 0.9 percent of adults who had gambled in the past year. When compared with those responding 'yes' in condition B, the greatest difference was seen for lying to family or others (where the 'yes' group appears to include some people who may have said occasionally otherwise) and the smallest difference was for poor performance at work (0.7 percent 'very often' compared with 1.0 percent 'yes').

Figure 8: Prevalence of harms to self, by endorsement options

Figure 8: Prevalence of harms to self, by endorsement options.

Response pattern for scaled answer options: harms to self questions	Option 1: experience this at least occasionally (percentage)	Option 2: experience this at least fairly often (percentage)	Option 3: experience this very often (percentage)	Option 4 (condition b): Yes, experiences this (percentage)
Reduced spending	6.5%	2.4%	0.7%	2.8%
Uses savings and/or borrows money to gamble	4.7%	1.8%	0.6%	1.9%
Conflict with others	4.2%	1.6%	0.8%	1.7%
Feels isolated	4.2%	1.5%	0.6%	1.1%
Lies to family	5.3%	1.5%	0.9%	3.1%
Absent from work and/or poor performance at work	1.9%	1.3%	0.7%	1.0%

Figure 8 information

Note: 'Don't know' and 'refused' options are not shown on chart or in table, hence responses do not sum to 100 percent.

Similar patterns were observed for the harms from others questions. Option 1 (at least occasionally experiencing each harm) resulted in the highest prevalence of harms, being between two to three times higher than those answering 'yes' in condition B. Rates between those answering 'fairly often' or 'very often' were similar to those answering 'yes', with the exception of conflict with others and lies to conceal the extent of someone else’s gambling.

For both, those who may otherwise have answered that they 'occasionally' experienced this seemed more likely to have answered 'yes' when given a binary choice. Finally, as before, those answering 'very often' had the lowest prevalence rates ('Figure 9: Prevalence of harms from others, by response options' as follows).

Figure 9: Prevalence of harms from others, by endorsement options

Figure 9: Prevalence of harms from others, by endorsement options.

Response pattern for scaled answer options: harms from others questions	Option 1: experience this at least occasionally (percentage)	Option 2: experience this at least fairly often (percentage)	Option 3: experience this very often (percentage)	Option 4 (condition b): Yes, experiences this (percentage)
Reduced spending	4.3%	1.6%	0.5%	2.0%
Uses savings and/or borrows money to gamble	4.8%	1.7%	0.7%	2.2%
Conflict with others	8.7%	2.8%	1.0%	4.5%
Feels isolated	5.1%	1.9%	0.9%	2.4%
Lies to family	6.7%	1.5%	1.0%	3.6%
Absent from work and/or poor performance at work	3.6%	1.3%	0.5%	1.1%

Figure 9 information

Note: The table shows the percentages of respondents who answered at least 'occasionally' in response to this question. The table does not show the percentage of respondents who answered never, so the responses shown will not add up to 100 percent.

How experience of harms correlates with other factors

In order to better understand the impact of different ways of calculating rates of harms, the analysis looked at how well these options associated with other measures where a relationship would be expected. For example, with the harms to self questions, a strong relationship between the rate of participants classified as experiencing each harm and Problem Gambling Severity Index (PGSI) status would be expected. Further, those with lower levels of wellbeing and those engaged in other risky health behaviours (higher risk alcohol consumption and cigarette smoking), have poorer general health and have higher rates of impulsivity would be more likely to experience harms.

To assess this, a series of unadjusted binary logistic regression models were produced. The models looked at how each of these factors (wellbeing, general health, other high-risk health behaviours etc.) were associated with harms across the following three different outcome measures:

• outcome 1 - experienced this harm occasionally compared with never
• outcome 2 - experienced this harm fairly often or very often compared with occasionally or never
• outcome 3: yes, had experienced this harm compared with no, had not.

Harms to self

For reducing spending, gambling causing isolation and lying to others to conceal extent of gambling, all three outcome measures were significantly associated with PGSI status, cigarette smoking status, general health status, impulsivity, wellbeing scores and alcohol consumption. The patterns were as expected. For example, participants had greater odds of saying that they had reduced their spending or that gambling caused isolation, or they had lied to people about their gambling (irrespective of how endorsement was defined) if they had PGSI scores of eight or higher (compared with those with a PGSI score of 0). In short, whichever way endorsement was defined, these harms were associated with a range of health and wellbeing measures in the way expected.

This pattern broadly held true for using savings to fund gambling; experiencing conflict and being absent from work, with some minor exceptions.

For using savings to fund gambling and being absent from work, only endorsement derived from scaled answer responses were significantly associated with alcohol consumption ('Figure 10: Odds ratios of using savings or borrowed money to gamble, by response options' as follows).

Likewise, for conflict with others, only answer options from the scaled responses were significantly associated with cigarette smoking status. In short, for these harms, the yes or no answer options did not have the same range of associations with health behaviours as the scaled answer options, though these were fairly minor differences.

Figure 10: Odds ratio of high risk alcohol consumption for using savings or borrowed money to gamble, by endorsement options

Figure 10: Odds ratio of high risk alcohol consumption for using savings or borrowed money to gamble, by endorsement options.

Odds ratio for using savings to gamble	Odds ratio	95 percent Confidence interval (lower)	95 percent confidence interval (higher)
Option 1: Yes versus No
No high-risk alcohol consumption	1 (reference)	Not applicable	Not applicable
High-risk alcohol consumption	1.8	0.7	4.4
Option 2: at least occasionally versus never
No high-risk alcohol consumption	1 (reference)	Not applicable	Not applicable
High-risk alcohol consumption	3.0	2.0	4.5
Option 3: at least fairly often versus occasionally and/or never
No high-risk alcohol consumption	1 (reference)	Not applicable	Not applicable
High-risk alcohol consumption	3.8	2.0	7.1

Harms from others

When looking at responses to the harms from others questions, the same consistency between condition A (scaled answer options) and condition B (yes or no) responses and their association with health and wellbeing measures was less evident.

Condition B responses performed less well across the different harms asked about. For example, an association between experiencing harms from other people’s gambling and the participant’s own personal wellbeing would be expected. This was observed for all harms when endorsement was measured using the scaled answer options (condition A), but not for reducing spending, experiencing conflict or lying to hide the extent of someone else’s gambling in condition B (yes or no answer options).

Likewise, each harm measured using the scaled answer options had a relatively consistent association with alcohol and cigarette smoking status (whereby an individual was more likely to report experiencing each harm if they smoked or consumed alcohol at higher risk levels). But when endorsement was measured using binary responses, most harms were not associated with smoking status, and lying about other people’s gambling or being absent from work was not associated with alcohol consumption.

In short, when harms from other people’s gambling were measured using yes or no answer options, the experience did not correlate as well with measures of wellbeing or other health behaviours.

Harms from others questions: routing and answer options

Only those who reported that someone who is close to them gambled were routed to questions asking about harms from other people’s gambling. In the pilot there appeared to be a systematic underreporting of gambling by close others when compared with known gambling prevalence rates: only 28.5 percent of participants said that someone close to them gambled. This could potentially impact on the accuracy of any prevalence estimates produced from these questions, leading to underreporting of harms from others.

For this experimental statistics phase, the screening question was refined to remind participants about the types of gambling to include and to answer this question even if people close to them gambled only occasionally. In total, 57 percent of participants reported that someone close to them gambled, closer to the past year participation rates. Thus, refinements to this question appears to have addressed this underreporting.

Gambling-related suicidality questions

Using questions developed for the Adult Psychiatric Morbidity Survey (APMS), participants were asked to report if they had experienced suicidal thoughts in the last 12 months and whether they had attempted suicide in this period. Anyone who answered 'yes' to either of these questions were asked the extent to which this was related to their gambling. Answer options were 'not at all', 'a little' or 'a lot'.

Overall, 11.8 percent of participants reported thinking about suicide and 1.2 percent reported attempting suicide in the last 12 months. These estimates are higher than rates of suicidal thoughts and suicide attempts reported within the Adult Psychiatric Morbidity Survey: Mental Health and Wellbeing, England, 2014 (opens in new tab)(5.4 percent and 0.7 percent respectively).

Whilst the APMS 2014 survey notes an upward trend in the prevalence of suicidal thoughts, comparison with more up-to-date data is unlikely to fully explain these differences. This over-estimation should be borne in mind when using this data for future analysis.

Of the 668 individuals who reported any suicidal thoughts or attempts eight reported this was related to their own gambling 'a lot' and a further 25 participants reported this was due to their own gambling 'a little'. With regards to attributing these behaviours to gambling, it is unclear if these participants were thinking about their suicidal thoughts, suicide attempts or both when answering the gambling question. This creates the potential for analytical ambiguity in terms of what this data is representing, especially where people may have had multiple experiences of suicidal ideation and attempts in the past year.

---

¹⁶ Developing survey questions capturing gambling-related harms - Part A - Expert review of question development by Professor Robert Williams and Dr Rachel Volberg, Heather Wardle, Viktorija Kesaite, Robert Williams, Rachel Volberg (2022).

Conclusion

The four-point answer scale (condition A) showed improvements over the prior three-point scale, with more logical sequences of endorsement and reduced item non-response. It also provides more analytical opportunities than the binary 'yes' or “no” option (condition B).

For most harms, those responding yes' in condition B (binary answer option) produced similar endorsement rates to those responding 'fairly often or very often' in condition A. However, for some harms (such as lying to family and others about the extent of one’s gambling) those answering 'yes' likely included people who only experienced this once or twice.

Contrary to expectations, those answering 'yes' at the harms from others questions displayed less consistent patterns of association with personal wellbeing.

The changes to the screening questions to route people into the harms from others questions have resulted in improvements, matching the rate closer to the gambling participation rate.

Some of the harms from others answer options differed, potentially confusing participants by switching between different scales.

Some analytical ambiguity was identified in the gambling suicidality questions. The wording and routing for these questions should be determined by the data needs.

Testing different approaches to asking questions about gambling participation

Introduction

One of the original aims of the survey was to review and refresh the gambling participation question commonly used on surveys, such as the Health Survey for England (HSE) 2018: Survey documentation (opens in new tab). The intention was to better reflect the current diversity of gambling products and to facilitate analysis of problem gambling prevalence at a product level.

One of the proposals that was consulted on by the Gambling Commission at the beginning of this project was to review the way respondents are asked about their participation in different gambling activities within the gambling participation and prevalence research consultation. The pilot stage of the project then began with gathering ideas from stakeholders about how best to update and refresh the list of activities asked about, and conducting cognitive testing to explore the ways in which people understood the descriptions of gambling activities used in survey questions, identifying any misunderstandings, ambiguities and missing activities.

Step 2 of the experimental statistics stage then aimed to assess the impact of asking questions about gambling participation in different ways on survey estimates of gambling participation rates.

Three different ways for asking questions about gambling activities participated in over the last 12 months and the last four weeks were assessed:

• a long-list approach
• a hierarchical-list approach
• a chunked-list approach.

The two time frames were asked about in order to capture recent activity (such as in the last four weeks) and also, less frequent gambling activity (such as over the last 12 months). The 12 months’ time frame was also used to correctly route into the Problem Gambling Severity Index (PGSI) items.

Survey participants were randomly assigned to one approach. For all three approaches, participants were asked whether they had spent money on any of the listed activities in the past 12 months. If they said yes, they were then asked the same question (and presented with the same answer codes) but about the last four weeks.

The wording of two questions was the same for all three approaches, but the answer options differed as follows.

Long-list approach

For the long-list approach, participants were presented with a redefined version of the list of activities used on the HSE, which distinguished between taking part in activities online and in-person. A total of 20 answer options, including ‘other’, were presented and participants were instructed to select 'yes' or 'no' for each activity.

Hierarchical-list approach

For the hierarchical-list approach, participants were presented with a list of 10 high-level activities; more granular-level activities then sat within the overarching high-level categories. As with the long-list approach, participants were instructed to select 'yes' or 'no' for each activity.

Chunked-list approach

For the chunked-list approach, the long-list was combined into five different categories or 'chunked' lists in which participants were asked about each group sequentially. Participants were asked to select all that applied (or ‘none of these’ if none of the five categories applied).

For each approach, those who had participated in a particular gambling activity in the last four weeks were asked a series of follow-up questions. Chunked-list participants were asked a shorter list of similar in nature activities at a time and if they had not spent money on the listed activities, they were routed to the next category of activities. For the long-list and hierarchical-list approaches, additional follow-up questions were asked including those to distinguish between online and offline participation and to capture frequency of play and spend. The questionnaire specifications are provided in Appendix D - Step 2 web questionnaire (PDF) and Appendix E - Step 2 paper questionnaires (PDF).

For step 2, 14,982 addresses were issued with the aim of achieving 4,000 productive individual questionnaires. The sampled addresses were randomly allocated to one of three equal sized groups.

Random allocation was used to eliminate the potential for systematic self-selection bias. Minimising differences between participants in each group that could be responsible for any differences seen in estimates between the three question approaches ensures that any observed differences are due to the way the question was asked¹⁷. In total, 4,994 addresses were issued to each of the survey approaches with the aim of achieving 1,333 productive individual questionnaires from each.

A total of 3,492 individual questionnaires were completed:

• 1,230 in the long-list approach group (35 percent of completions)
• 1,184 in the hierarchical-list approach group (34 percent of completions)
• 1,078 in the chunked-list approach group (31 percent of completions).

---

¹⁷ Experiments for Evaluating Survey Questions, Jon Krosnick (2011). In Question Evaluation Methods: Contributing to the Science of Data Quality. Hoboken, New Jersey; John Willey & Sons; pp. 215-238, Jennifer Madans, Kristen Miller, Aaron Maitland, Gordon Willis [Eds].

Comparison of gambling participation across the three approaches

You can view tables referenced in this section by downloading the file Tables A1 to A48 - Gambling Survey - Experimental statistics stage (XLSX)

This section looks at whether there was a difference in the reporting of gambling participation across the three approaches. It also looks at whether the approaches performed differently across the two data collection modes (online and postal, see the Questionnaire content and design section of this report).

The types of gambling participation reported, the number of activities those who gambled had participated in and which activities were most commonly mentioned were also reviewed.

Differences which are statistically significant at the five percent level are identified as such in the text. Some observed differences which did not reach this level of statistical significance are nonetheless included if they indicate a pattern; these are clearly identified in the text as not statistically significant.

Participation in any gambling activity in the last 12 months

In total, 63 percent of participants reported that they had participated in any gambling activity in the last 12 months. Those asked the hierarchical-list approach were less likely to report that they had spent money gambling in the last 12 months compared with those asked the long-list or the chunked-list approaches: 59 percent compared with 64 and 65 percent respectively. These differences were statistically significant.

For past year gambling, there was a higher participation rate among those who completed the survey online (64 percent) compared with those completing a postal questionnaire (59 percent). The lowest participation rate occurred amongst the hierarchical-list approach (61 percent for online completions and 54 percent for postal completions compared with 67 percent and 60 percent for the long-list approach and, for the chunked-list approach, 65 percent and 64 percent). These differences were statistically significant (Table A.35, Proportion having spent money on gambling activities in the last 12 months, by questions approach and mode of completion).

A statistically significant higher proportion of men (66 percent) than women (60 percent) reported having gambled in the last 12 months. For men, 68 per cent of those asked the chunked-list, 67 percent of those asked the long-list and 62 percent of those asked the hierarchical-list reported gambling in the last 12 months. The percentages for women were 62 percent, 61 percent and 55 percent respectively. However, differences between men and women for the types of participation questions they were asked did not reach statistical significance.

The lower reported rate of gambling participation amongst those asked the hierarchical-list set of questions was anticipated. When individual gambling activities are grouped together, the loss of visual prompts about specific types of gambling activity results in lower reporting of gambling activities (Table A.36, Proportion having spent money on gambling activities in the last 12 months, by questions approach and sex).

During analysis, individual gambling activities were grouped into eight higher-level categories:

• lotteries
• scratchcards
• instant win games
• betting
• bingo
• casino games
• fruit and/or slot machine games
• other gambling activities.

There were no statistically significant differences between the three approaches for participation (such as having spent money on) in any grouped activities in the last 12 months, but the most commonly mentioned activities for all three approaches were lotteries (between 48 percent and 52 percent), scratchcards (between 20 percent and 26 percent) and betting (between 15 percent and 20 percent) (Table A.37, Proportion having spent money on grouped gambling activities in the last 12 months, by questions approach).

Money spent on gambling activity

Those who had gambled in the last 12 months were asked which gambling activities (grouped) they had spent money on in the last four weeks. Again, there were no statistically significant differences between the approaches but the two most commonly mentioned activities across all three were the same as those mentioned for the last 12 months: lotteries (between 81 percent and 83 percent) and scratchcards (between 26 percent and 33 percent). The third most frequently mentioned activity among those asked the hierarchical-list approach was instant wins (mentioned by 20 percent). For the other two approaches, the third most frequently mentioned activity participated in in the last four weeks was betting (14 percent for the long-list approach and 18 percent for the chunked-list approach).

In the long-list and chunked-list list approaches, participants were presented with descriptions of different types of betting as part of the question. These descriptions did not form part of the question presented to participants in the hierarchical-list approach (Table A.38, Proportion having spent money on grouped gambling activities in the last four weeks, by questions approach).

The average number of gambling activities participated in over the last 12 months was 1.3, with significant differences between the three approaches (1.2 for those asked the hierarchical-list, 1.2 for those asked the chunked-list and 1.4 for those asked the long-list). A higher proportion of long-list participants reported taking part in three or more grouped gambling activities (21 percent compared with 18 percent for the hierarchical list and 16 percent for the chunked-list).

One potential reason for the lower number of grouped gambling activities for the chunked-list is that the continual switching between answering questions about gambling participation in the last 12 months and the last four weeks may have caused confusion and resulted in the lower reporting of some gambling activities (Table A.39, Number of grouped gambling activities spent money on in the last 12 months, by questions approach).

Potential primacy effect with the long-list approach

You can view tables referenced in this section by downloading the file Tables A1 to A48 - Gambling Survey - Experimental statistics stage (XLSX)

Questions with a long-list of answer options are known to be at greater risk of a type of systematic error called primacy bias when presented visually¹⁸. Response options visually presented at the start of a long-list are more likely to be selected than those further down the list. One method of combatting this is to break up longer lists into chunks, hence the inclusion of this approach in the experiment. It might be expected that if a primacy effect is evident in the long-list approach, gambling activities further down the list would be selected by a smaller proportion of participants compared with the shorter chunked-list approach.

Another method to combat the risk of primacy bias is to require participants to answer yes or no to each item presented in the list; this was used in the long-list approach. Randomising the list would have been another method but whilst this would have been possible in the online questionnaire, it could not be done in the postal version.

The analysis compared rates of gambling participation for the long-list and chunked-list approaches. The hierarchical-list approach was not included due to the different way activities were presented to participants.

Rates of gambling participation for the two approaches, the long-list and chunked-list, were similar across most gambling activities. The greatest differences was seen for activities nearer the top of the list: National Lottery scratchcards (24 percent for the long-list and 18 percent for the chunked-list, a statistically significant difference), tickets for other charity lotteries (23 percent and 18 percent respectively, not statistically significant), and National Lottery online instant win games (10 percent and five percent respectively a statistically significant difference).

Activities in the latter half of the participation list differed by up to three percentage points across the two approaches. This as well as the higher endorsement of activities overall suggests that the long-list was not subject to primacy bias.

A similar (but again, not statistically significant) pattern and absence of evidence of primary bias was found for gambling activities in the last four weeks. This may be because the use of yes or no response options in the long-list was more effective at encouraging participants to engage with each item on the list, than chunking the list (Table A.41: Gambling activities spent money on in the last 12 months, by questions approach; Table A.42: Gambling activities spent money on in the last four weeks, by questions approach).

---

¹⁸ See for example: Retrospective reports: The impact of response alternatives, Norbert Schwarz, Hans-J. Hippler, Elisabeth Noelle-Neumann (1994). In Autobiographical memory and the validity of retrospective reports, New York,: Springer-Velag, pp187-202, Norbert Schwarz and Seymour Sudman [Eds].

Additional considerations for the postal questionnaire

You can view tables referenced in this section by downloading the file Tables A1 to A48 - Gambling Survey - Experimental statistics stage (XLSX)

Completion of postal questionnaires relies on correct navigation by participants. Whilst routing instructions are provided, there is no constraint on which questions are completed nor on the answers that can be given and hence data is subject to errors when routing instructions are not followed. These type of errors are not possible in the online questionnaire, where the routing is handled by the questionnaire program.

The three sets of gambling participation questions took up the same amount of space (nine pages) in the postal questionnaire. However, the layout, routing instructions and order of follow-up questions differed, and this could have impacted on how participants answered the questions.

For the long-list and hierarchical-list approaches participants were routed to later pages to record further detail about each activity recorded. To an extent, this relied on participants remembering which activities they initially recorded and going to the right page and activity to record the further detail. The chunked-list approach had less complex routing instructions with follow-up questions following directly after each activity type.

Adherence to routing instructions for Problem Gambling Severity Index (PGSI) items

Those who had gambled in the last 12 months were routed to the PGSI items. Those who had not gambled in the last 12 months were routed past the PGSI items.

Looking at those who had gambled in the last 12 months who were routed to but did not answer the first PGSI item, non-response was 37 percent for the long-list, 31 percent for the hierarchical-list and 21 percent for the chunked-list approach. The same pattern and similar proportions were seen for the final PGSI item where non-response was 38 percent, 34 percent and 24 percent respectively. All approaches had higher levels of item non-response to the PGSI items than was seen in the pilot and in step 1.

The highest level of non-response for the long-list approach is likely because the routing instructions were more complex compared with the chunked-list approach, leading to more participants missing the questions (Figure 11: Postal questionnaire PGSI non-response amongst those who had gambled in the last 12 months, by questions approach as follows).

Figure 11: Postal questionnaire Problem Gambling Severity Index (PGSI) non-response amongst those who had gambled in the last 12 months, by questions approach

Figure 11: Postal questionnaire PGSI non-response amongst those who had gambled in the last 12 months, by questions approach. Routing error at PGSI-1: Bet more than could really afford to lose.

Routing error at PGSI1: Bet more than could really afford to lose	Gambling participation questions approach: Long-list (percentage)	Gambling participation questions approach: Hierarchical-list (percentage)	Gambling participation questions approach: Chunked-list (percentage)
Proportion of those who had gambled in the last 12 months that did not answer the question	37%	31%	21%
Bases (unweighted): those who had gambled in the last 12 months (number)	264	234	227

Figure 12: Postal questionnaire PGSI non-response amongst those who had gambled in the last 12 months, by questions approach. Routing error at PGSI-9: Felt guilty about the way gamble or what happens when gamble.

Routing error at PGSI9: Felt guilty about the way gamble or what happens when gamble	Gambling participation questions approach: Long-list (percentage)	Gambling participation questions approach: Hierarchical-list (percentage)	Gambling participation questions approach: Chunked-list (percentage)
Proportion of those who had gambled in the last 12 months that did not answer the question	38%	34%	24%
Bases (unweighted): those who had gambled in the last 12 months (number)	264	234	227

Figure 11 information

Note: The table shows the percentage of respondents who were eligible to answer the question but did not answer it across the three different versions of the questionnaire, so the responses shown will not add up to 100 percent.

Those who had not gambled in the last 12 months should not have answered the PGSI items. The proportion who nonetheless did so was highest in the chunked-list approach (44 percent) and lower in the hierarchical and long-list approaches (11 percent and 18 percent respectively). These differences were statistically significant. This lower ‘routing error’ rate on the long-list and hierarchical-list approaches suggests that the instructions were clearer and easier to navigate for those who had not gambled in the last 12 months ('Figure 13: Postal completions: proportion of those who had not gambled in the last year with PGSI routing errors, by questions approach' as follows).

Figure 12: Postal completions: proportion of those who had not gambled in the last year with Problem Gambling Severity Index (PGSI) routing errors, by questions approach

Figure 12: Postal completions: proportion of those who had not gambled in the last year with Problem Gambling Severity Index (PGSI) routing errors, by questions approach. Routing error at PGSI-1: Bet more than could really afford to lose.

Routing error at PGSI1: Bet more than could really afford to lose	Gambling participation questions approach: Long-list (percentage)	Gambling participation questions approach: Hierarchical-list (percentage)	Gambling participation questions approach: Chunked-list (percentage)
Proportion of those who had not gambled in the last 12 months that answered question	18%	11%	44%
Bases (unweighted): Those who had not gambled in the last 12 months (number)	199	220	176

Figure 13: Postal completions: proportion of those who had not gambled in the last year with Problem Gambling Severity Index (PGSI) routing errors, by questions approach. Routing error at PGSI-9: Felt guilty about the way gamble or what happens when gamble.

Routing error at PGSI9: Felt guilty about the way gamble or what happens when gamble	Gambling participation questions approach: Long-list (percentage)	Gambling participation questions approach: Hierarchical-list (percentage)	Gambling participation questions approach: Chunked-list (percentage)
Proportion of those who had not gambled in the last 12 months that answered question	17%	10%	39%
Bases (unweighted): those who had gambled in the last 12 months (number)	199	220	176

Figure 12 information

Note: The table shows the percentage of respondents who answered the question even though they had not gambled in the past 12 months across the three different versions of the questionnaire, so the responses shown will not add up to 100 percent.

Regression modelling was carried out to look at the effects of the demographic variables used in the weighting strategy on the first and last PGSI items, PGSI1 and PGSI9. The aim was to see the relative impact of the weighting variables compared with the impact of the question approach (that is whether the participant was asked the long-list, hierarchical-list or chunked-list set of gambling participation questions). The independent variables included were, from the weighting strategy: sex, age group, ethnicity, household income (grouped), tenure and highest education level and also, the question approach. The dependent variables in the analysis were PGSI1 and PGSI9.

When accounting for the possible effect of variables used in weighting, the proportion of non-response to PGSI1 was lower in the chunked-list approach than for the other two approaches but there was no difference between the approaches for PGSI9. Hence, question approach had a significant impact on response which confirms the initial analyses (Tables A.43: Multivariate analysis with PGSI1 and A.44: Multivariate analysis with PGSI9).

Collecting detail about gambling activities in the last four weeks

When participants reported that they had gambled (that is spent money) on a certain activity in the last four weeks they were instructed to answer a set of follow-up questions on that activity. For the long-list and hierarchical-list approaches postal participants were first asked what gambling activities they participated in and then routed to later pages to record further detail. This relied on participants remembering which activities they initially recorded and going to the right page and activity to record the further detail.

The chunked-list approach had less complex routing instructions with follow-up questions following directly after each activity type. However, for some activities, the routing instructions in the hierarchical-list better matched the category listed at the previous question. As an example, the lotteries follow-up questions in all approaches began with an instruction of 'If you bought lottery tickets in the last four weeks answer the following questions', an instruction that was likely clearer and easier to navigate for hierarchical-list participants as it matched the category previously listed (in the other two approaches ‘lotteries’ was separated into two categories so there was not a direct match).

There was a lower non-response to the first lotteries follow-up question amongst hierarchical-list approach participants (21 percent compared with 30 percent in the long-list approach and 32 percent in the chunked-list approach). A lower proportion of participants who had not bought a lottery ticket in the last four weeks also mistakenly answered these questions (14 percent). None of these differences reached statistical significance (Table A.45 Routing errors to first lottery follow-up question, postal completions).

Missing responses

The long-list and hierarchical-list approaches asked participants to code 'yes' or 'no' for each listed activity. The chunked-list approach had multi code 'tick all that apply' answer options (with 'None of these' if they had not participated in any gambling activity during the recording period).

There was little difference between the long-list and hierarchical-list approaches in terms of non-response (between two percent and six percent). For the chunked-list, non-response was higher with between 11 percent and 27 percent of participants not selecting any of the activities nor the 'None of these' answer option.

This suggests that the yes or no grids used on the long and hierarchical-lists worked better than the multi-coded question in the chunked-list approach. It should also be borne in mind that switching between time periods in the chunked-list approach may have confused participants and confounded the rate of non-response (Table A.46 Missing responses to gambling participation questions, postal completions).

Other considerations

You can view tables referenced in this section by downloading the file Tables A1 to A48 - Gambling Survey - Experimental statistics stage (XLSX)

Online questionnaire break off rates

Online questionnaire break-off rates (that is participants exiting the questionnaire) during the gambling participation questions were very low: just seven hierarchical-list participants, six long-list approach participants and one chunked-list participant (Data not shown).

Inconsistent responses

A review was carried out into inconsistent responses to the gambling participation questions. Inconsistent responses were defined as instances where a participant did not report gambling on a particular activity in the last 12 months, but then reported gambling on that activity in the last four weeks.

As the hierarchical approach was more complex to navigate on paper, it might have been expected that for this approach there would have greater inconsistency between activities reported at follow-up questions and those reported at the initial participation. This did not appear to be the case: the number of inconsistent responses ranged from one to 10 per questionnaire for this approach compared with one to 11 for the long-list and one to 17 for the chunked-list¹⁹.

No discernible pattern in response inconsistencies was found across the three question approaches (Table A.47 Inconsistency in responses: Proportion of participations reporting gambling on each activity in the last four weeks but not in the last 12 months).

Exclusiveness of gambling activities listed

In order to capture the full range of gambling activities, the final item in the participation lists for all three approaches was 'another form of gambling'. Online participants who selected this answer option were asked to record what this other form of gambling was.

The proportion of participants selecting the 'another form of gambling' option was similarly low across the three approaches (between two percent and three percent for gambling in the last 12 months and between one percent and two percent for gambling in the last four weeks, none of these differences were statistically significant), which suggests that the updated gambling participation list successfully captures all types of gambling (Table A.48, Number and proportion of participants selecting the “another form of gambling” answer option).

Around half of 'another form of gambling' responses could be back coded into the pre-existing code frames. This included 16 participants who described their ‘another activity’ as ‘the Grand National’’ or ‘horse racing’, rather using the pre-existing code 'Betting on sports and racing'.

Most of the remaining responses were valid forms of gambling but the participant did not provide enough information for it to be back coded into one of the existing answer options. For example, responses such as 'Football results' or 'Betting' could not be back coded for the long-list and chunked-list approaches as the participant did not state whether this was in-person or online.

---

¹⁹ Significance testing was not carried out due to small numbers and the differences in how groups of activities were presented to participants, according to the question approach.

Conclusion

No clear picture emerged as to which of the three approaches performed best in capturing information about gambling participation, each had advantages and disadvantages, as summarised as follows in 'Figure 13: Advantages and disadvantages of the three approaches to asking about gambling participation'.

Summary

The hierarchical-list approach appears to lead to lower reporting of gambling participation. However, it should be considered whether the combined categories of lotteries, scratchcards and online instant win games should be incorporated into the long-list and chunked-list approaches as the presentation of these categories in the hierarchical approach had the lowest non-response in the postal questionnaire format of the three approaches.

The chunked-list approach generated higher non-response in the postal questionnaire format. The experiment was not designed to identify the reasons for this, but it is possible that this was due to participant-fatigue and satisficing, whereby participants learn that if they select a gambling activity this results in them being asked additional, follow up questions.

The long-list approach generally performed well except for routing postal participants who had gambled in the last 12 months into the Problem Gambling Severity Index (PGSI), a key survey variable. The chunked-list approach performed better in routing into the PGSI those who had gambled in the last 12 months.

Recommendations are provided in the Recommendations section of this report.

Figure 13: Advantages and disadvantages of the three approaches to asking about gambling participation

Past-year gambling participation rate

The advantages and disadvantages of the long-list, hierarchical-list, and chunked-list approaches to the past-year gambling participation rate are as follows.

Long-list approach

Advantage - similar gambling participation rate to the pilot survey and chunked-list approach indicating it is capturing all forms of gambling.

Hierarchical-list approach

Disadvantage - a lower gambling participation rate than the pilot survey and than the other two approaches.

Chunked-list approach

Advantage - similar gambling participation rate to the pilot survey and long-list approach indicating it is capturing all forms of gambling.

---

Past-year gambling participation rate - types of activities (grouped)

The advantages and disadvantages of the long-list, hierarchical-list, and chunked-list approaches to the past-year gambling participation rate - types of activities (grouped) are as follows.

Long-list approach

Advantage - highest participation rate for lotteries, casino games, fruit and/or slot machine games. Did not have the lowest participation rate for any activity.

Hierarchical-list approach

Advantage - highest participation rate for scratchcards, instant win games.

Disadvantage - lowest participation rate for lotteries, betting, bingo, casino games.

Chunked-list approach

Advantage - highest participation rate for betting.

Disadvantage - lowest participation rate for scratchcards, instant win games and fruit and/or slot machine games.

---

Past-year gambling participation rate - number of grouped gambling activities

The advantages and disadvantages of the long-list, hierarchical-list, and chunked-list approaches to the past-year gambling participation rate - number of grouped gambling activities are as follows.

Long-list approach

Advantage - highest average number of gambling activities. Highest proportion of participants reporting three or more gambling activities.

Hierarchical-list approach

Disadvantage - lower proportion of participants reporting three or more gambling activities indicating some gambling activities potentially missed.

Chunked-list approach

Disadvantage - lowest proportion of participants reporting three or more gambling activities indicating some gambling activities potentially missed.

---

Primacy effect

The advantages and disadvantages of the long-list, hierarchical-list, and chunked-list approaches to the primacy effect are as follows.

Long-list approach

Advantage - no evidence of primacy effect.

Hierarchical-list approach

Not applicable.

Chunked-list approach

Not applicable.

---

Postal questionnaire routing: Problem Gambling Severity Index (PGSI) (based on past-year activity)

The advantages and disadvantages of the long-list, hierarchical-list, and chunked-list approaches to the postal questionnaire routing: PGSI (based on past-year activity) are as follows.

Long-list approach

Advantage - lower level of routing error for those who had not gambled.

Disadvantage - high level of routing error for those who had gambled.

Hierarchical-list approach

Advantage - lower level of routing error for those who had not gambled.

Disadvantage - high level of routing error for those who had gambled.

Chunked-list approach

Advantage - lower level of routing error for those who had gambled.

Disadvantage - high level of routing error for those who had not gambled.

---

Postal questionnaire routing: lotteries follow-up questions (based on past 4 weeks activity)

The advantages and disadvantages of the long-list, hierarchical-list, and chunked-list approaches to the postal questionnaire routing: lotteries follow-up questions (based on past 4 weeks activity) are as follows.

Long-list approach

Disadvantage - similarly high level of routing error for lottery players and non-lottery players.

Hierarchical-list approach

Advantage - lowest level of routing error for lottery players.

Chunked-list approach

Advantage - lowest level of routing error for non-lottery players.

Disadvantage - highest level of routing error for lottery players.

---

Non-response to past year gambling list

The advantages and disadvantages of the long-list, hierarchical-list, and chunked-list approaches to the non-response to past year gambling list are as follows.

Long-list approach

Advantage - low proportion of participants not answering the participation list.

Hierarchical-list approach

Advantage - low proportion of participants not answering the participation list.

Chunked-list approach

Disadvantage - high proportion of participants not answering some of the participation lists (later chunked sets).