Strategy
Testing strategy for compliance with remote gambling and software technical standards
Requirements for the timing and procedures for the testing of remote gambling products.
Contents
Procedure for testing
The Commission maintains and publishes a list of approved test houses authorised to perform third party testing. Licensees and their chosen test house will need to agree the scope of testing, which must be sufficient to ensure that testing will adequately assess compliance with the Commission’s standards and meet the level of testing required under this strategy. This primarily applies to the RTS requirements outlined in Table 1.
The level and scope of testing required by the Commission is as follows.
RNG testing
- review of RNG documentation to understand the implementation of RNG in the gaming system.
- research about RNG algorithm or hardware to ensure there is no publicly known weakness or vulnerabilities associated with the RNG under evaluation.
- review of source code to verify the implementation of RNG is in accordance with the RNG documentation.
- statistical testing of raw output of RNG and scaled or shuffled decks data.
- any issues or non-compliance are reported to the supplier. Once resolved, these issues are re-evaluated to confirm the non-compliance has been addressed adequately.
Game testing
- verification of game design – maths, artwork or rules as displayed to players, and theoretical RTP.
- Software testing – this involves verifying the software implementation of the above game design, artwork, maths and theoretical RTP through testing of the game on an environment which reflects the intended live environment; verification of game rules, actual RTP using simulation3, emulation4 and manual5 testing; any scaling and mapping used to convert raw RNG output to game outcomes.
The Commission supports and participates in the International Association of Gaming Regulators (IAGR) Multi-Jurisdictional Testing Framework. This framework aims to standardise technical standards and testing requirements between participating jurisdictions in order to reduce testing duplication for products deployed internationally. Further details are available on the IAGR website (opens in new tab).
Any additional integration testing required, for example when the game will be utilised with a different RNG or platform to the original game testing, is covered in the Gambling Platform and RNG changes as follows. Testing in these instances will be determined by the licensee in conjunction with the test house and will depend on the changes made to integrate the software as well as the amount of previous testing that can be relied on.
For games, the testing report should include at least:
- test house details including the test supervisor that signed off the testing
- licensee name
- date of testing
- certificate reference
- game details – including game name, return to player (RTP), software number and digital signature
- scope and approach to testing and a description of all tests applied
- platform supplier and platform version
- channels (game clients) covered by testing
- result of testing
- details of games or versions of games that the game supersedes
- where a limited scope of testing has occurred due to changes within a previously tested game, an updated games test report must be provided to the Commission, making reference to the original games test report, changes made, testing completed and new digital signatures.
For RNGs, the testing report should include:
- test house details including the test supervisor that signed off the testing
- license name
- date of testing
- certificate reference
- RNG details – brief description of the RNG and its use including RNG version, whether it is hardware and/or software and digital signature
- scope and approach to testing and a description of all tests applied
- platform supplier and platform version
- Any limitations on the use of the RNG should be cited.
Limitations on the use of the RNG might include, but not be limited, to:
- the acceptable degrees of freedom (DOF) permitted for the RNG
- whether it is suitable for use with or without replacement, and
- any dependency on operating system functionality that if modified could impact on the operation of the RNG, for example, Java SecureRandom
- results of testing.
Licensees6 must send the results of testing7 (that is, a test house’s game/RNG report) to the Commission on completion of satisfactory testing (but prior to release). All new games and RNGs can only be released once the testing has been completed and the report provided to the Commission.
The games/RNG reports should be uploaded to the licensees games register by using eServices (opens in new tab).
B2C licensees who utilise the services of a B2B for the provision of gaming content must still maintain their own up to date games register for any games offered directly or via the B2B.
Game updates
For the purposes of this document, an update that does not impact game fairness is referred to as a minor update and can be released without the need for external retesting. For illustrative purposes, a non-exhaustive list of major and minor updates is provided in Annex A.
Licensees in conjunction with their test houses will be expected to use their own judgement as to those changes that do not affect game fairness and for all updates will need to ensure they:
- adhere to the minimum change control standards, see section 68
- maintain a record of all updates in change control documentation, which must be available upon request for inspection
- ensure a relevant personal management licence (PML) holder (or in the case of a small scale licensee, the relevant qualified person), is responsible for the process.
All of the previously mentioned will be subject to an annual audit by an approved test house, see section 6.
These provisions do not affect the requirement for licensees to submit software for external testing for all new games, (or updates to existing games when changes affect game fairness), and to submit the test reports via the games register by using eServices (opens in new tab) to the Commission prior to release.
Testing environment and gambling platform/RNG changes
We expect game testing to occur using the software and environment intended for live operation. Test houses would need to perform some integration testing where there are differences in the live environment compared to the test environment which could affect the fairness of games.
There are a number of games and RNGs brought under Commission regulation via transitional arrangements from other jurisdictions. Where previous testing was deemed satisfactory (previously known as level 3 testing) these games could continue to be used without need for further testing. Where the games had not been level 3 tested then further testing was required to be completed by 31 October 2015. Updates to any transitioned games should be assessed in accordance with this testing strategy (that is updates which may impact fairness need to be externally retested as per the game updates section, as mentioned previously).
In some instances an update will be made to a remote gaming system (RGS) or an RNG which could affect the functionality, and therefore fairness, of hundreds of games served by the updated RNG or residing on the updated RGS. In this scenario licensees must ensure a representative sample of games are retested to ensure the RGS/RNG change has not affected their operation.
The sample should be wide enough to include each game type and generation (and not be restricted to RNG functionality or games with similar characteristics). The nature and size of the representative sample and the full scope of the integration testing should be decided by licensees in conjunction with an approved test house.
Details of this testing may be captured in a single test report and evidence retained by the license holder. The Commission expects testing to have been completed prior to launching the updated RNG or RGS.
New channel testing
Where a licensee wants to release a new channel for an existing game they must ensure that channel has been tested by an approved test house. Normally, when a game is first tested for release, it will be tested using the intended channel(s) it will be offered via, for example HTML 5 and native mobile app. Over time, and as newer channels become popular, existing games may be ported across to those new channels. As the channel represents the main player interface it is important that its operation is tested.
The subset of tests for a new channel will generally be limited to the user interface and player display aspects of a game, such as a manual test to see how the game client displays results. If the backend game design and functionality have not been altered to accommodate the new channel, as is usually the case, then these aspects will not need retesting. Submission of test reports for new channels added to existing games is required, as per 3.7 as previously mentioned. Reference should be made to the original game test report.
Where third party client operating systems and browsers9 are updated this generally won’t require external retesting. We would expect the licensee’s own testing to confirm satisfactory performance of existing games when new client operating systems and browser versions are released, note this is different to updates for the RGS or RNG, which underpins the game engine. Such updates may require games to be retested as per gambling platform or RNG changes in the previous section).
Where a game is designed to work on a variety of devices or browsers testing should be of the most commonly used devices and browsers, these should be identified within the test report.
Testing and audit requirements for remote lottery licensees
This section sets out the criteria that applies to remote lottery licensees10 (including external lottery managers) when determining specific testing and audit requirements.
Holders of remote lottery licences that accept no more than £250,000 worth of entries per year by means of remote communication will not be required to submit their RNG for testing by a Commission approved test house or undertake a third party annual security audit.
Instead, and in terms of RNG testing, such licensees will need to demonstrate that:
- their RNG has been tested or verified as being fair and random by an independent and suitably qualified third party. This must be supported by documentary evidence
- they have policies and procedures in place which set out how they ensure the lottery draw is fair and open and can produce evidence that these procedures are followed.
In terms of the third party security audit requirement, such lottery licensees will instead be required to demonstrate to the Commission on request that they comply with the RTS security requirements as set out in Section 5 of the RTS.
Holders of such licences that accept more than £250,000 worth of entries by remote means per year will be required to meet the full RNG testing and third party security audit requirements as set out in table 1.
References
3 Simulation (output) testing – setting the game up to play automatically for a high number of games (actual number will depend on volatility of the game as per the game maths) to verify that the actual RTP is within an acceptable range of the expected RTP. Sample data should be tester generated, unless supervised in a controlled environment for the purposes of meeting specific regulatory requirements. Software modified from the original to enable rapid play is permitted provided the tester has confidence that the modifications do not impact on the assessment of game fairness.
4 Emulation testing is used to replicate certain rare game outcomes (such as jackpot triggers, special features and maximum prizes).
5 Manual game play - actually playing the game to verify all activity observed works as expected (for example, playing a game for one hour would allow the tester to see most of the common prizes and determine whether pay lines are implemented correctly etc).
6 The following categories of licences require games and RNG testing by an independent test house (subject to a best practice declaration):
Remote general betting (standard) (virtual events), remote betting host (virtual events) remote pool betting, remote casino, remote casino (game host), remote bingo operating, remote bingo (game host) and remote lottery licences(entries greater than £250,000 per year).
7 Where a licensee relies on a B2B for the provision of games they should receive a games register reference number from the B2B to upload the relevant game to their games register, therefore alleviating the need to re-submit the same report.
8 Section 6 sets out the minimum change control requirements that licence holders will be expected to adhere to. Licence holders may adopt alternative approaches those set out in Section 6 if they have actively taken account of the requirements and can demonstrate that an alternative approach is reasonable in the particular circumstances or that to take an alternative approach would be acting in a similarly effective manner.
9 For example, updated versions of the mobile operating system provided by Apple or Google for mobile devices; or the version of the internet browser software increases.
10 By lottery licensees we mean; remote lottery operating licensees, converted lottery operating licensees (but only those licensees that run remote lotteries themselves or via a lottery manager) or remote lottery managers’ operating licensees (also known as external lottery managers) licensed under the Gambling Act 2005 (opens in new tab).
Testing strategy for compliance with remote gambling and software technical standards - Approach Next section
Testing strategy for compliance with remote gambling and software technical standards - Annual games testing
Last updated: 6 December 2023
Show updates to this content
Following an audit corrected link formatting issues only. Formatting changes