GSGB – Removing the ‘official statistics in development’ label

GSGB – Removing the ‘official statistics in development’ label: Approach

The label ‘official statistics in development’ is used to alert users to statistics that may be affected by a development, for example in our case the design of a new survey and new methodology to administer the survey. However the label should only be a temporary measure and it was always our intention to remove the label following the developmental stage of the project, providing it was deemed appropriate.

Guidance published by the Office for Statistics Regulation (OSR) (opens in new tab) states that the decision to remove the label should be based on one of three outcomes:

1. The statistics are of sufficient quality and value to be used in a meaningful way and so be published as official statistics.
2. The statistics are of insufficient quality or do not meet the required need and so production will be stopped.
3. There is insufficient evidence to reach a conclusion, or further refinement is required – producers would continue the development and be clear about extent of further testing.

We believe that the GSGB is collecting statistics which meet outcome 1.

In addition, advice from the Government Statistical Service (opens in new tab) outlines factors that should be considered when removing an ‘official statistics in development’ label:

• whether user feedback indicates that statistics are trustworthy, of high quality, and valuable
• whether the methods employed have proved sufficiently robust to suit the variety of circumstances material to the use of the statistics
• whether coverage has reached a sufficient level
• whether the defined development phase has ended
• whether it is judged that the statistics fully comply with the standards of the Code of Practice.

We have set out our evidence of meeting these factors in the Evidence of meeting these factors section.

GSGB – Removing the ‘official statistics in development’ label : Evidence of meeting these factors

Whether user feedback indicates that statistics are trustworthy, of high quality, and valuable

User feedback has been an important aspect in the development of the GSGB. Over 60 respondents completed the original 2020 consultation, a further 70 completed a stakeholder engagement survey at the end of 2021. We have held 9 stakeholder engagement panel sessions where we have updated on project progress and we hosted a workshop session about the new survey at our Evidence Conference in March 2023 which over 60 people attended. This level of engagement demonstrates how valuable stakeholders consider the statistics to be. The Government’s white paper on High Stakes: gambling reform for the digital age, published in April 2023, also highlighted the value of regular and reliable statistics to inform policy making.

We commissioned an independent review of the GSGB approach which was undertaken by Professor Sturgis, Professor of Quantitative Social Science at the London School of Economics. Professor Sturgis concludes in his report that in his opinion it was the right decision to have moved to an online self completion methodology and that the GSGB will yield high quality and timely estimates of gambling prevalence in Great Britain. Professor Sturgis also highlights the benefits that the survey will bring in terms of the better detection and understanding of patterns and trends in gambling behaviours, both within population sub groups and over time – emphasising the value that these statistics will bring.

Some stakeholders do remain critical of the GSGB methodology because some estimates are higher than those generated by other surveys and they believe previous methods of collecting the data (for example, face to face Health Surveys) were more reliable. All survey methodologies have strengths and limitations associated with them. We will review the recommendations in Professor Sturgis’s report to help us to continue to understand the impact of the changes made to both the survey design and the methodology and where we can test further improvements. We think the key is to be open and transparent about the strengths and limitations of the GSGB methodology approach, emphasise the nature of the statistics being an estimate and continue to refine our approach to improve that estimate over time.

Whether the methods employed have proved sufficiently robust to suit the variety of circumstances material to the use of the statistics

The objective of collecting data through the GSGB is to understand patterns and trends in gambling behaviour in Great Britain. The Commission has a statutory to provide advice to the government on the incidence of gambling, understand changes in the market and player behaviour and appreciate the impact of policy change.

A push-to-web methodology has been used for the GSGB as previously used methods are no longer robust enough to suit the variety of circumstances material to the use of the statistics.

We have used experts in the field of survey design and collection to develop our approach and had it independently assessed by a leading academic in population survey work.

We are confident that the frequent collection of data, across an improved questionnaire and timely outputs will allow us to track key statistics trends. The large sample size associated with the GSGB will also mean we can also understand gambling behaviours amongst sub groups of the population, which we have not been able to do adequately before due to smaller sample sizes. Professor Sturgis’s report reflects this view.

We do recognise that headline data, particularly those used in relation to understanding the estimate of consumers who are scoring 8+ on the PGSI need to be sensitively handled. This is especially so, given the methodology looks likely to deliver an estimate higher than that previously found in earlier surveys which the Commission used for official statistics and is reflected in Professor Sturgis’s report.

Being transparent in relation to the strengths and weaknesses of the statistics and calling out those who may misuse the statistics will be an important part of engagement as stakeholders get used to the new survey. We will consider how best to analyse and report on the data. For example, when using the statistics to understand the impact of gambling, the data will be analysed amongst gamblers rather than the whole population to negate the risk that the sample could have some bias towards gamblers.

On balance we are content that the methods employed are robust but will continue to refine the approach, in line with best practice, to improve it. We would also note that this is just one, important, perspective on the questions we seek to understand. Our aspiration is to have triangulated evidence rather than just one source.

Whether coverage has reached a sufficient level

We have developed an approach which will provide national coverage, regularity of data collection, and consistency across England, Scotland, and Wales. As we collect more data year on year, this will allow us to confidently report on trends in gambling behaviour, as well as a more detailed understanding of behaviours amongst sub-population groups.

Each wave we will collect 5,000 responses from adults across Great Britain, we will undertake 4 waves per year with a total of 20,000 responses being collected per annum from a nationally representative sample of adults aged 18 years and older in Great Britain.

We have maximised coverage of respondents by providing a paper response option alongside the web response option. This ensures responses can be captured by those who are less technologically literate, those without internet access and those who prefer an alternative approach to respond. These respondents may have otherwise been missed and not covered in the sample. We are however aware that some groups will be excluded from the survey because they are not included in the sampling frame (that is, those who do not live in private residences) and we will continue to find ways to include these groups in other research to ensure their views are not missed.

Whether the defined development phase has ended

Following the Participation and Prevalence: Pilot methodology review, we entered a 12 month experimental phase for the project. This phase allowed for continued development and refinement of the research methodology. In November 2023 we published the findings from the final step in the experimental stage of the project, signalling that our defined development phase had ended.

Whilst our defined development phase has ended and we will be removing the official statistics in development label, we will be continuing to innovate and improve our official statistics over time as per the Code of Practice for Statistics (opens in new tab). This includes taking forward the recommendations set out by Professor Sturgis in his review of the GSGB to continue to improve the quality of the estimates and ensure stakeholder confidence in the statistics.

Whether it is judged that the statistics fully comply with the standards of the Code of Practice

Working closely with our research supplier, NatCen and the University of Glasgow we are confident that the GSGB adheres to the three pillars of Trustworthiness, Value and Quality in the Code of Practice for Statistics.

The trustworthiness pillar details the requirements which support independent statistics production. For GSGB data releases are pre-announced on a regular and consistent basis, staff are free from political influence, as well as the data being managed and dealt with confidentiality with privacy of respondents remaining protected.

The value pillar necessitates that the statistics meet user needs, add value, and provide insight.

We have fully engaged with our users throughout the different stages of development, tracking our progress via a timeline on our website. Engagement sessions with stakeholders have been important in allowing us to go into detail on their expectations as users of the data and an opportunity to discuss concerns and updates. Advice and recommendations from experts in the field has been vital in the process in terms of making sure that these statistics represent good value. The data provides opportunities for granular analyses and will be publicly available on the UK data archive each year.

The quality pillar requires that “The statistics must be the best available estimate of what they aim to measure, and should not mislead… the data must be relevant, the methods must be sound and the assurance around the outputs must be clear”. We follow an internal quality assurance process which is published on our website. As well as this, a peer review by Professor Sturgis concludes that the Gambling Survey for Great Britain is robust and meets the quality and value for the modern needs of a gambling survey. This survey will provide national coverage, regularity, and consistency of approach across England, Scotland, and Wales, allowing us to confidently report on trends in gambling behaviour, as well as a more detailed understanding of behaviours amongst sub-population groups.

GSGB – Removing the ‘official statistics in development’ label : Conclusion

In conclusion, we have reached the decision to remove the official statistics in development label.

Whilst we will remove the ‘in development’ label, this does not mean that we will not continue to refine and improve the methodology over time. We will continue to undertake a programme of continuous improvement for our official statistics building on best practice and the recommendations outlined in Professor Sturgis’s report. We will also continue to be open and transparent about the strengths and limitations of our approach, describing them suitably in supporting technical documents.