I have been involved with testing many VR and AR applications, and I can safely say that they all come with their own specific challenges when it comes to the implementation of software testing. To run a testing project in this field successfully and not interfere with development time, it is vital that technical issues that can arise with virtual reality applications are taken into account when planning test sessions. Today I will be discussing two challenges you’re quite likely to come across: thermal throttling and low-quality texture.
1. Thermal throttling
Thermal throttling, when a device slows down its CPU to prevent itself from overheating, is very specific to mobile VR. Using a VR app with demanding graphics will a lot of the time end up overheating and overloading the mobile device that you are testing on. This makes it very difficult to test an application, as devices overheat after long-period of use time.
How can it be managed?
To be honest, thermal throttling is something that you should expect. The VR apps that we test are still undergoing development and are most of the time not optimised to run properly on different combinations of hardware. To start with, the Test Engineer should always report this, as part of having a testing period is to make sure that the end product will run properly on the platforms and hardware that is being designed for.
And since this is very likely, it is best to adapt the test planning to help prevent your devices overloading and interrupting your testing schedule. Use ‘burst testing’ sessions of a shorter duration: First test the application’s different requirements on the device itself, then report the issue in the device’s ‘cooling-off’ period between testing sessions.
2. Low-quality texture
The quality of textures used in VR applications also influence the user experience and duration of testing periods. Lower-quality textures, which are often used in mobile VR or low-specification VR desktop applications, have a huge impact on the user’s capacity to focus and concentrate.
Combine this with the physical stresses of being in VR – the strain of wearing a HMD on your head, very close to your eyes, and the dissonance of moving within the app whilst sitting on a chair – and ‘VR sickness’ can quickly set in. This has a huge impact on how long a tester is able to stand being in the VR environment.
How can it be managed?
As with thermal throttling, we recommend adapting the test planning to help prevent testers from burning out physically and mentally. Use ‘burst testing’ sessions of a shorter duration and report bugs in between sessions. We also recommend testing an application in pairs to increase test coverage, allowing testers breaks between time slots spent in the VR environment to recuperate, and also log bugs they found.
What has worked for us so far?
To help limit the impact of these problems in our testing sessions, we do the VR testing in pairs. For more complex projects, we use a team of testers. As such, we try to lighten each tester’s time spent in virtual reality – and increase the potential test coverage – by having more testers available and also by recording all the testing sessions. Thus, whilst one person is testing with the VR headset, the other can go through the recordings and start logging the issues the other person found whilst wearing the HMD.
Also, it’s important that the testers follow the testing plan and at the same time are open for exploratory ‘charter-based’ testing. Having multiple people on one project helps to cover the first-time user experience multiple times over, and provide useful information back to the development team regarding their updates and fixes that need to be made.
spriteCloud will be at VR Days Europe 2017 on Thursday 26th and Friday 27th October in at the Expo Hall. If you want to hear more about our experiences testing virtual reality applications, then come and say hi at our booth and tell us about your app. We look forward to speaking with you!