blog

Let's talk about testing

Standard platform, operating system and browser recommendations

Introduction

The following post makes recommendations for platforms, operating systems and browsers to use when testing commercial marketing web sites aimed at the general public in Europe.

Our recommendations are based on usage figures widely available on the Internet, our experience and analysis of clients needs. Since modern browser versions change frequently, we review and update these recommendations regularly.

Assumptions

  • For the majority of users, Chrome and Firefox silently auto-update in the background.
  • Each new version cannibalises the market share of previous versions quickly, i.e. within a quarter.
  • Significant differences between versions are few and far between.

List of latest stable releases

Desktop browsers

  • Microsoft Internet Explorer (MS-IE) 10 for Windows 7 released to general public (February 26, 2013)
  • Chrome Stable release 25.0 (February 21, 2013)
  • Firefox (FF) Stable release 19.0 (February 19, 2013)
  • Opera Stable release 12.14 (February 5, 2013)
  • Safari Stable release 6.1.2 (February 19, 2013)

Mobile and tablet browsers

  • Android 4.2.2 “Jelly Bean” (February 11, 2013)
  • iOS 6.1.2 (February 19, 2013)
  • Windows Phone 8.0.10211.204 (Portico) (after October 29, 2012)

Recommendations

General

  • Invest in the future, not the past. Focus project time and effort on optimising web sites for future-facing operating system and browser combinations while making sure that popular older combinations are still viewable.
    • We define ‘the future’ as the next 3-6 months, i.e. 2 x internet lifetimes of three months each.
  • Use a range of mainstream devices to ensure coverage of most regular users everyday experience.
  • Avoid testing with specific minor releases unless there are compelling reasons to do so.
    • Example of a compelling reason: a third-party augmented reality (AR) component is known to work with Safari 6.0.1 but not with Safari 6.0.2.
  • Do not support a browser if it is not supported by the manufacturer.
    • E.g. Mozilla discontinued support for Firefox 3.6 on 24 April 2012. Firefox 3.6 is therefore defunct.
  • Do not support beta-release operating systems and browsers. Allow them time to be released and become mainstream first.
    • We define ‘mainstream’ as above 5% market share.

PC platform recommendations

  • Test with Windows XP.
  • Test with Windows Vista.
  • Test with Windows 7.
  • Do not test with Windows 8 – too early to adopt and not yet mainstream
  • Do not test with Linux. Market penetration is too low.

 

  • Do not test with MS-IE 6. The version is defunct plus market penetration is low and getting smaller.
    • Exception: Web sites targeted at China.
  • Do not test with MS-IE 7. The version is practically defunct plus market penetration is low and getting smaller.
  • Test with MS-IE 8.
  • Test with MS-IE 9.
  • Do not test with MS-IE 10 – too early to adopt and not yet mainstream.
  • Test with Chrome latest release only.
  • Test with Firefox (FF) latest release only.
  • Do not test with Safari on the PC platform. The PC version is defunct plus market penetration is low and getting smaller.
  • Do not test with Opera. Market penetration is too low.
    • Exception: Web sites targeted at Eastern Europe, especially Belarus.

Mac platform recommendations

  • Test with OS/X 10.6 ‘Snow Leopard’ using Safari 5.1.7.
  • Test with OS/X 10.7 ‘Lion’ using the latest stable release of Safari.
  • Test with OS/X 10.8 ‘Mountain Lion’ using the latest stable release of Safari.

 

  • Test with the latest release of OS/X using Firefox (FF) latest release only.
  • Test with the latest release of OS/X using Chrome latest release only.
  • Do not test with Opera on the Mac platform. Market penetration is too low.

Mobile and tablet platform recommendations

  • Test in both landscape & portrait modes.
  • Test with the latest release of iOS 6 using the native Safari browser only.
  • Test with a version of iOS 5 using the native Safari browser only.

 

  • Test with Android 2.3 ‘Gingerbread’, using the native browser only.
  • Test with Android 3.x ‘Honeycomb’, tablets only, using the native browser only.
  • Test with Android 4.0 ‘Ice Cream Sandwich’, using the native browser only.
  • Test with Android 4.1 ‘Jelly Bean’, using the native browser only.
  • Do not test with Android 4.2 ‘Jelly Bean’ – too early to adopt and not yet mainstream.

 

  • Do not test with Windows Phone 7.5. Market penetration is too small and will decline further.
  • Do not test with Windows Phone 8 – too early to adopt and not yet mainstream.
  • Do not test with Blackberry. Market penetration is too small.
    • Do not test with Blackberry 10 (Initial release: January 30, 2013) – too early to adopt and not yet mainstream.
  • Do not test with Firefox OS smartphones and tablet computers – too early to adopt and not yet mainstream.
  • Do not test with Firefox, Chrome, Opera or any other non-native third-party mobile browser variants on any mobile or tablet platform. Market penetration is too small.

Flash recommendations

  • Test with no Flash. At the very least, ‘no Flash’ should prompt the site visitor to download Flash.
  • Test with Flash 10.3.
  • Test with Flash 11.x (latest version).

Sources

 


Reputation. Meet spriteCloud

Find out today why startups, SMBs, enterprises, brands, digital agencies, e-commerce, and mobile clients turn to spriteCloud to help improve their customer experiences. And their reputation. With complete range of QA services, we provide a full service that includes test planning, functional testing, test automation, performance testing, consultancy, mobile testing, and security testing. We even have a test lab — open to all our clients to use — with a full range of devices and platforms.

Discover how our process can boost your reputation.