Report
Gambling Survey for Great Britain - technical report
Gambling Survey for Great Britain - technical report
Data analysis and reporting
Accuracy and reliability of survey estimates
The Gambling Survey for Great Britain (GSGB), in common with other surveys, collects information from a sample of the population. The sample is designed to represent the whole population of adults aged 18 years and over living in private households in Great Britain, as accurately as possible within practical constraints, such as time and cost. Consequently, statistics based on the survey are estimates, rather than precise figures, and are subject to a margin of error, also known as a 95 percent confidence interval. For example, the survey estimate might be 15 percent with a 95 percent confidence interval of 13 percent to 17 percent. A different sample might have given a different estimate, but it would be expected that the true value of the statistic in the population would be within the range given by the 95 percent confidence interval in 95 cases out of 100. Confidence intervals are affected by the size of the sample on which the estimate is based. Generally, the larger the sample, the smaller the confidence interval, which results in a more precise the estimate.
Confidence intervals are quoted for key statistics within GSGB reports. Where differences are commented on, these reflect the same degree of certainty that these differences are real, and not just within the margins of sampling error. These differences can be described as statistically significant10.
Strengths and limitations
When commissioning the GSGB, the Gambling Commission (the Commission) weighed a range of strengths and limitations of the new approach. These are summarised in this section.
Strengths
The Commission’s information needs are consolidated into a single survey (rather than several surveys as previously) which ensures consistency and efficiency.
The collection of data on a rolling basis and producing annual datasets and trends reduces the impact seasonal events (such as the FIFA World Cup) may have on key variables (for example, gambling participation rates).
The survey has undergone a comprehensive development stage, led by experts in the field. Development included cognitive testing, pilot testing, experimental testing, and stakeholder engagement.
The survey design and large, representative samples (per wave) allow the Commission to report on key results on a quarterly basis as well as to conduct more detailed analyses.
The push-to-web methodology is more cost effective when compared with face-to-face collection methods.
The methodology also allows increased numbers of people to be interviewed at relatively lower cost, something that is important for the analysis of gambling harms.
A postal alternative to the online questionnaire enabled the recruitment of adults who may have been less technologically literate, not have access to the Internet, or preferred an alternative option, and so increasing the representativeness of the sample.
The survey includes a broad range of content on gambling that is relevant to both gamblers and non-gamblers.
The self-administered data collection methods are likely to mitigate social desirability in responses to questions about sensitive topics (for example, about their gambling behaviour).
As the survey is ‘gambling focused’, it means more detail can be collected about gambling behaviours than is possible in a more general survey, where the number of questions that can be included is limited.
Limitations
With a push-to-web methodology, interviewers are not present to collect the data in person and accuracy of answers relies on participants understanding the questions asked and following the instructions.
Similarly, there is a risk that some participants (although a small proportion) will not follow the routing instructions correctly on the postal version of the questionnaire. To minimise the risk, the postal questionnaire was designed with simple routing instructions and further, routing errors were checked and corrected during the office-based data editing process.
Compared with face-to-face interviewing methods, remote data collection methods typically have lower response rates, meaning they are potentially more susceptible to non-response bias. However, response rates for face-to-face interviews are also declining meaning these studies are also subject to non-response bias11. Furthermore, survey methodologists have found that the correlation between response rate and non-response bias is considerably weaker than conventionally assumed (Groves and Peytcheva 2008; Sturgis et al. 2017)12.
As the GSGB is ‘gambling focused’, it is possible that the survey disproportionately attracts those who gamble, so that this group may be over-represented.
Caveats for interpreting estimates generated by the Problem Gambling Severity Index (PGSI)
The GSGB will produce new estimates of those scoring 1 to 2, 3 to 7 and 8 or higher on the Problem Gambling Severity Index (PGSI). No survey methodology is perfect; different surveys measuring the same phenomena will provide different estimates because variances in survey design and administration can affect both who takes part and how people answer these questions. Until 2010, data on gambling was captured through the bespoke British Gambling Prevalence Survey (BGPS) series (conducted in 1999, 2007 and 2010). Originally intended to be a tri-annual survey, funding for the BGPS was cut in 2011. The Commission then sought different ways to capture information about gambling within available budgets.
Between 2012 and 2021, the primary method of measuring scores according to the PGSI (as well as a second measurement instrument, the DSM-IV) was through the Health Survey for England (HSE series) and the Scottish Health Survey. The GSGB picks up where the BGPS left off by being a bespoke gambling survey that captures a wide range of information about gambling across the whole of Great Britain. However, the methodology for the new GSGB differs from the BGPS and the health survey series in a number of ways. The remainder of this section considers a range of issues affecting all surveys, that may either serve to under-estimate or over-estimate the PGSI estimates.
Factors which may mean PGSI estimates are under-estimated in household-based surveys
Coverage error
Using the PAF as a sample frame is common on large-scale surveys, including the BGPS, the GSGB and the health survey series. This means that only those living in private households are eligible to be included in the survey. People living in student halls of residence, military barracks, hospitals, prisons and other institutions are excluded. Some of these populations may have higher rates of gambling and higher PGSI scores. All studies using the PAF as a sample frame inherit this source of bias.
Social desirability bias
This bias in founded on the idea that there are social norms that govern certain behaviours and attitudes, and that people may misrepresent themselves so as to appear to conform to these norms13. In the survey context, this misrepresentation may involve participants explicitly deciding to give false information or modifying their in-mind answer. However, it can also involve participants giving information that they believe to be true but is in fact inaccurate14. It is a potential risk for all surveys that collect information on sensitive topics, including the health survey series and the GSGB. Sensitive topics include those that:
- may be perceived as an invasion of privacy (for example, asking about frequency of gambling)
- involve a disclosure risk where there could be repercussions for the participant as a result of responding (for example, asking about criminal behaviour), or
- have to admit to breaking a perceived social norm (for example, asking about alcohol consumption).
One strategy to reduce the risk of social desirability bias is to use self-completion methods. These methods include online and postal questionnaires, which are completed by the participant. Self-completion methods are used on both the health survey series and the GSGB to collect information on gambling. However, the surveys differ in the way in which self-completion methods are used, which may affect resulting estimates. The health survey series is an interviewer-administered survey that includes a paper self-completion questionnaire to ask about gambling behaviour. This is typically completed by participants in the presence of the interviewer and potentially other household members, who also take part in the survey.
Sturgis and Kuha15noted that it is possible that the presence of an interviewer or other household members might lead to underreporting of gambling in the self-completion questionnaire. Their analysis did not find a statistically significant difference in the proportion of people with a PGSI score of 1 or more within Health Survey for England (HSE) data, depending on whether other people were present at the time the gambling questions were being completed. However, subsequent analysis of HSE 2018 data conducted for the GSGB pilot, using multi-variate regression models, found that the odds of having an PGSI of 1 or more were 1.5 times higher among those who did not have other household members present at the point of interview16. The authors concluded that the online methods of GSGB may offer greater privacy to participants, and so reduce social desirability bias. A recent review of the GSGB development has suggested that additional research could be undertaken to better understand the role of socially desirable responding as the driver of the difference in gambling estimates between in-person and self-completion surveys17.
Non-response and/or selection bias
During the stakeholder engagement sessions conducted for the GSGB, those with lived experience of gambling harms stated that they would have been unlikely to participate in a survey when they were experiencing gambling difficulties. This was also highlighted by Sturgis in his review of the GSGB methods. Evidence supporting this is provided by analysis on non-response of the 2007 and 2010 BGPS series. In 2007, Scholes et al demonstrated a strong relationship between the factors predicting household non-response and gambling frequency: area and household-level factors which predicted lower household response were associated with higher gambling frequency. This suggests that those households less likely to take part in surveys were more likely to contain frequent gamblers18. Similar analysis conducted for the BGPS 2010 (reported in Wardle et al, 2014)19 demonstrated that households which either:
a) required multiple attempts to contact
b) were reissued after multiple follow-up attempts, or
c) were followed-up by telephone interviewers after the face-to-face interviewer had been unable to make contact were more likely to contain people who gambled.
This supports the notion that very engaged gamblers may be less likely to take part in surveys overall. This is likely to apply to all surveys. (However, both the health survey series and the GSGB are likely subject to different selection biases.)
Question instruments to measure the negative impacts of gambling
The measurement of experience of so-called problem gambling is via a series of questions known as “screens”. Multiple different screens to measure the experience of problem gambling exist. No screen is perfect. In the BGPS and health survey series, two different screening instruments have been used: the DSM-IV and the PGSI Problem gambling screens.
Analysis of these screens shows that they capture different groups of people with potentially different types of problems. Orford et al suggested that the PGSI, especially among women, may underestimate certain forms of gambling harms which the DSM-IV is better suited to measure20. For this reason, the BGPS and health survey series always included both the DSM-IV and PGSI screens. The rates of problem gambling reported by the PGSI are lower than those reported by the DSM-IV. Since the BGPS was developed, the PGSI has become one of the most widely used screens, particularly because it presents scores on a spectrum of severity. In addition, there is now greater focus on the wider range of negative consequences associated with gambling which are not captured by the PGSI or the DSM-IV. During consultation on the GSGB questionnaire content, stakeholders strongly suggested that it would be appropriate to include only one screen for problem gambling and to use additional questionnaire space to capture other important aspects of gambling experiences. As a result, the GSGB only includes the PGSI screen, which generates lower estimates of problem gambling than the DSM-IV.
Factors which may mean PGSI estimates are over-estimated within bespoke gambling studies
Non-response bias/selection bias
How surveys are presented to potential participants can influence who takes part. Williams and Volberg21 conducted an experiment presenting the same survey to potential participants but varying its description – introducing it either as a health and recreation survey or a gambling survey. The found that rates of problem gambling were higher in the latter. This is maybe because people who gamble may potentially be more likely to take part in a gambling survey because it is relevant to them. The GSGB likely suffers from this selection bias compared with the health survey series. Ethically, the GSGB invite letter has to inform people what the study is about which may make it more attractive to those who gamble. Despite best efforts to reduce this possibility, it is likely that some selection bias remains and so that rates of past-year gambling participation and PGSI scores are higher in the GSGB compared with the health survey series (when compared using the exact same questions, as was the case for the pilot and stage 1 of the experimental statistics phase).
In addition, analysis conducted by Sturgis and Kaha22 and also, Ashford and others (the latter for the GSGB pilot) detailed in the Participation and Prevalence: Pilot methodology review report found that those who completed PGSI questions online had higher PGSI scores than those who completed the questions via an alternative mode. In short, online surveys may overestimate the proportion of online gamblers, which may in turn overestimate gambling harm because online and frequent gambling are independently associated with a higher probability of gambling harm.
However, evidence suggests that those experiencing harms from gambling are less likely to take part in surveys overall and have poorer health outcomes. Given this, there is also the possibility that these people may be less likely to take part in a health-focused survey, which would also impact on the results obtained by health surveys. This is a theoretical possibility that needs further empirical examination, as recently recommended by Sturgis in his review of the GSGB development23.
References
10Statistical significance does not imply substantive importance; differences that are statistically significant are not necessarily meaningful or relevant.
11For example, in 2021, the Health Survey for England (HSE) household response rate was 32% compared with 60% in 2015 and 59% in 2018.
12Sturgis, Patrick, Joel Williams, Ian Brunton-Smith, and Jamie Moore. 2017. ‘Fieldwork Effort, Response Rate, and the Distribution of Survey Outcomes: A Multilevel Meta-Analysis’. Public Opinion Quarterly 81(2): 523–42; Groves, Robert M., and Emilia Peytcheva. 2008. ‘The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis’. Public Opinion Quarterly 72(2): 167–89.
13Kreuter, F., Presser, S. and Tourangeau, R. (2008) ‘Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity’, Public Opinion Quarterly, 72(5), pp. 847–865.
14Tourangeau, R. and Yan, T. (2007) ‘Sensitive Questions in Surveys’, Psychological Bulletin, 133(5), pp. 859–883.
15Sturgis, P., & Kuha, J. (2022). How survey mode affects estimates of the prevalence of gambling harm: a multisurvey study. Public Health, 204, 63-69.
16Ashford, R., Bates, B., Bergli C et al (2022) Gambling participation and the prevalence of problem gambling survey: Pilot stage Methodology review report. National Centre for Social Research: London.
17Sturgis, Patrick (2024) Assessment of the Gambling Survey for Great Britain (GSGB). London School of Economics and Political Science, London, UK Assessment of the Gambling Survey for Great Britain (GSGB) - LSE Research Online (opens in new tab).
18Scholes, S., Wardle, H., Sproston, K., et al (2008) Understanding non-response to the British Gambling Prevalence Survey 2007. Technical Report. National Centre for Social Research, London.
19Wardle, H., Seabury, C., Ahmed, H et al (2014). Gambling Behaviour in England and Scotland. Findings from the Health Survey for England 2012 and the Scottish Health Survey 2012. National Centre for Social Research: London. Available at: Gambling behaviour in England and Scotland -Findings from the Health Survey for England 2012 and Scottish Health Survey 2012 (opens in new tab).
20Orford, J., Wardle, H., Griffiths, M., Sproston, M., Erens, B. (2010) PGSI and DSM-IV in the 2007 British Gambling Prevalence Survey: reliability, item response, factor structure and inter-scale agreement, International Gambling Studies, 10:1, 31-44.
21Williams, R. J., & Volberg, R. A. (2009). Impact of survey description, administration format, and exclusionary criteria on population prevalence rates of problem gambling. International Gambling Studies, 9(2), 101–117.
22Sturgis, P., & Kuha, J. (2022). How survey mode affects estimates of the prevalence of gambling harm: a multisurvey study. Public Health, 204, 63-69.
23Sturgis, Patrick (2024) Assessment of the Gambling Survey for Great Britain (GSGB). London School of Economics and Political Science, London, UK Assessment of the Gambling Survey for Great Britain (GSGB) - LSE Research Online (opens in new tab).
GSGB 2024 technical report - Methodology
Last updated: 2 May 2024
Show updates to this content
Formatting changes