Which Android versions should I test on?

At spriteCloud we know how important it is to test on as many as physical devices as possible. We know it is best to mix up the brands, models, screen sizes and performance of those devices too. We also know that time is precious and extra time is costly. We’ve seen device-specific bugs that appear on one device but not on another, with both devices running the same version of Android. This blog post is about which Android versions are important to focus on now.

To determine which versions are important, statistics are a useful tool. There are a number of places from where we get our numbers. One of those is the statistics that Google releases each month for Android here: http://developer.android.com/about/dashboards/index.html

Unless we are testing a website/app that is only useful for a limited amount of time (let’s say a short campaign), then the latest numbers are no more than just some numbers on a page. We also need to know in which direction the numbers are going, i.e. is the trend up, down or flat? We have therefore collected the statistics for many years so we can give good advice to our clients. The 12-month graph below gives more insight than just the table and pie chart. Keep in mind that some Android version numbers are combined under a single version codename. We therefore display the 12-month overview in numbers and names to make the picture clear.

Android versions over the last 12 months

Fig.1 Overview by version number (click to enlarge)

Looking at the current numbers, it is clear we should test with Android 5.1, 5.0, 4.4, 4.2 and 4.1 to get good coverage. But depending on the kind of project, the 12-month overview can give us even more insight into what is important. We can see for example, that Android 6.0 only has 4.6% market share currently and thus does not look that important. However, based on how rapid 5.1 increased over time, we can estimate when the percentage of market share will become important enough to take into account for our project. This also works the other way around. Although Android 4.4 is still by far the largest version in use, it is decreasing over time and we can estimate how long we still need to support this version before it becomes less/not important.

For the version codenames, the same rules apply. However, certain version numbers are combined in one codename and can distort the picture. The figure below help make this clear.

Android codenames over the last 12 months

Fig.2 Overview by Android version (click to enlarge)

Just looking at the codenames alone would make every customer say we should still test on Jelly Bean, since 21,3% is a pretty big chunk. This is true, except for the fact that Android 4.3, which is also part of the Jelly Bean codename, only has 3% market share. Thus 4.3 might be not considered important enough to test on – again depending on the type of project. Our advice is to look at the version numbers, not the codenames.

Keep track of our upcoming blog posts if you don’t want to keep track of these numbers yourself. When you develop a campaign website/app, the most recent numbers might be more important but most websites and apps are build to last a while. It’s therefore wise to always build for the latest version numbers and the ones with a high percentage of Android market share. For long-term projects, focus on the latest version numbers and up, since the landscape might look quite different a year from now – and usually in the favor of the newest builds.