blog

Let's talk about testing

The Cost of Compatibility

Yesterday at spriteCloud, we discussed how Safari is the new IE, meaning that of the major browsers, Safari seems to lag further and further behind in providing an up-to-date Web experience.

Elsewhere, that article turned into a different discussion: does it even matter to end-users if there is one browser that needs to be supported with more effort than any other? Yes it does: we can even calculate how much it matters.

Cloud

Let’s take a quick step back here. When we’re talking about how one browser takes an inordinate amount of effort to support, what we’re talking about is that a cloud app isn’t running properly in that browser without extra development effort. Such incompatibilities can be very minor (albeit annoying) — you might just have to tweak the formatting a little bit to make it work. Other incompatibilities can cause major headaches, for example when you’re trying to provide rich visuals with WebGL.

What we’re trying to do here is to provide a particular kind of filter for deciding how bad an incompatibility is.

Threshold

The discussion we’re referring to included the premise that the pain of supporting a new browser is negligible to the end user if the extra cost for it lies in the millicent range.

While that statement was probably hyperbole, it is actually a fairly good assumption to start out from. A thousandth of a cent is a negligible amount to pay for just about anyone. By focusing on the millicent threshold, we can allow for a thousand special cases before the cost starts to become visible to end users in the form of an extra whole cent to pay. Let’s go with that!

Subscription

Cloud apps are usually paid via some kind of subscription, and for the end user type apps, monthly payments seem to be fairly typical. That means any costs we will look at below are also to be considered monthly costs only.

Compatibility issues often require workarounds that aren’t just implemented once. More likely than not, in order to deal with such issues, your developers will build or use an abstraction layer that takes care of the incompatibilities on the one hand, but always imposes new usage paradigms on the other. That means any new features you add later on will likely also fall subject to the new usage paradigms, and new use cases often reveal omissions or flaws in the abstraction layer.

In other words, compatibility problems haunt you forever, even if you’re being clever about dealing with them.

Calculation

So let’s look at some numbers. According to U.S.News, software developer salaries averaged around USD $96,000 per year. If you distribute that across the Dutch average of 222 working days per year, and assume a nice, comfortable 8 working hours per day, you’ll arrive at an hourly cost for a developer of around USD $54.

Next, let’s see amongst how many users we have to spread that amount in order to make it into the millicent range. USD $54 is 5,400 cents, or 5,400,000 millicents. That means spreading USD $54 across 5,400,000 paying end users will put a burden of a millicent on each of them. That’s your threshold reached.

Let that sink in: a single developer hour a month requires 5,400,000 monthly subscribers to be as cheap as a millicent for each of them.

We’re not talking about single-hour amounts of effort here, though. Even if a developer only needed an hour to fix a compatibility issue, testing needs to happen on all supported browsers. The fix, if it actually works, needs to be rolled out. Then there’s administrative overheads for everything, and so forth.

It pretty much doesn’t make sense to count effort in hours from the perspective of the end-user. Going with a three man-day amount of effort for even minor issues is a conservative estimate, putting all of the above together. So we’ll have at least 3 * 8 * USD $54 in costs to deal with, which need to be spread across almost 130 million paying users to be as little as a millicent.

Comparison

Services such as GitHub are industry darlings reaching some 10 million users or so, only a fraction of which will actually pay for the service. It’s hard finding numbers on paying customers from such services, even if you try to gauge them based on company revenue and standard subscription fees. So let’s as a ballpark assume that to have a million paying customers, you’ll be in a really good shape.

That gives us a way to compare the results from the above calculation with reality a little: if you need 130 million users paying a millicent for a small adjustment to an app, then a million users need to pay 130 millicent each.

That’s still not a lot, but it’s starting to make a difference. Every 7-8 chunks of work this small size result in an extra cent you pay on your subscription.

So let’s scale this up to some more serious development.

Nowadays, for smaller web apps, it’s quite common for an app development team of at least three people to work in sprints of two weeks. In that time, just as a rough estimate, you can get a noticeable set of changes to software implemented and tested, and to a standard that is releasable. An adjustment to work around a badly implemented API may just fall into that category — it really starts depending on just how central to your app the incompatibility is, and how complex the workaround needs to be.

Such a sprint will cost 40 hours per week for two weeks for three people, so 40 * 2 * 3 * USD $54, or USD $12,960. Spread across the one million paying users mentioned above, it amounts to about 1.3 cents each.

It should be quite clear by now that “millicents” may have been a good starting point for defining a truly negligible amount, but that even the smallest changes to software result in whole cent costs to the development company.

Operations

The calculation above completely ignores operational costs, such as cost for hardware (either physical or virtual), rent and administrative overhead. Those costs are much harder to to express in a generalized view, so we’ll guesstimate them. But it means that the whole cent threshold we’re now discussing is highly optimistic.

Scaling up to a medium sized business, you’ll typically find that somewhere between a third and half of the staff are directly involved with software production; the rest fulfil other functions. Your staff cost for any change will therefore double to triple. Adding another 50% per staffer in rent, utilities and similar costs to the business is not entirely out of this world, either.

So that’s approaching a cent for the most trivial change, and any meaningful change must be measured at upwards of five cents each.

Hardware

Mapping hardware costs into this is critical, but difficult. A decent server or virtual machine can be rented for around USD $100 a month, but how much of this does each customer use?

Most systems for the kind of small app that we’re using for our example here will use at least a handful of machines, but will support thousands of users with that handful. When we’re looking for a system that will support a million users, though — just to keep to the same scale as before — the overall costs aren’t so easy to estimate. A whole lot depends on where you host your app, how well your software distributes load, etc., but mostly on how much load it generates. spriteCloud Calliope generates a lot of load.

Just to get into a reasonable enough ballpark, let’s look at Facebook for example. Apparently in 2010, they used 60,000 servers for round about 400 million active users.

You can scale that down quite easily to the one million subscribers scale we’ve been using above: 60,000 divided by 400 comes to about 150 servers.

Once you operate on the scale of 150 servers, renting them for $100 a pop isn’t the optimal choice. You’ll end up saving on operating costs building your own data center, but you’ll have to make a significant up-front investment, too. So for simplicity’s sake, let’s just run with the USD $100 for 150 servers, and say that supporting a million subscribers will cost you round about USD $15,000 a month.

It’s a very sketchy calculation. But it does compare very easily to the USD $12,960 development cost for a two week sprint: in order to just keep everything up and running while your team cranks away on an improvement, you need to add another 50% or so to the cost. Your trivial change exceeds a cent now, and your minimally meaningful change costs in the range of 7.5 cents then.

Compatibility

The last thing we need to look at to put the cost of compatibility into perspective is that for every sprint spent on compatibility issues, you’re not getting a new feature developed (again in our small app example). For that, we’ll need to estimate how much time a development team will spend on compatibility issues. If before we’ve made a few big assumptions to keep everything generalized and simple, here we’re going to make a few more.

A useful way to approach this is to look at it from the testing perspective rather than the development perspective: the cost of manually testing a piece of functionality per browser is relatively constant, which means that if you support the four major browsers out there, each browser contributes to 25% of the testing cost.

Reality is more complex, of course. In particular, development costs don’t scale quite the same to supporting multiple browsers. In fact, as the article referenced in the beginning should make clear, supporting two out of the four browsers comes at fairly negligible cost; it’s the other two that soak up extra attention.

But in lieu of a better metric, let’s take 25% of effort for each browser. That translates to every fourth sprint being spent on a compatibility issue, and that jives reasonably well for GUI-rich cloud apps.

So of every 25 cents you spend on subscription, about 7.5 cents go to compatibility issues. Or for a consumer-grade subscription fee of USD $25.00/month, that’s USD $7.50/month.

Caveat

The example’s small app development team of three people will not be able to support a million paying subscribers. A million paying subscribers would be a very excellent result for companies numbering their employees in the hundreds. That means that the economies of scale that such a company benefits from just don’t apply to the small app developer; their per-subscriber costs will be significantly higher. That makes every number we crunched above highly optimistic at best.

But that should help illustrate the real cost of development even better.

Conclusion

We took a long detour through costing meaningful changes to arrive at a fairly hand-wavy estimate for the cost of compatibility that completely forgoes the results of this detour.

You shouldn’t feel cheated, though.

While the article was motivated by trying to estimate the cost of compatibility, it’s equally worthwhile keeping in mind the cost of development — for any kind of reason. Compatibility with a given browser is just one feature out of an entire bag full, after all!

The picture easily gets muddled by free services numbering their users in the tens of millions or more, and by offshore developers working for a fraction of your local employee’s rate. Either of these, and plenty of comparable factors will make the cost of any change seem absolutely trivial.

However, if those factors apply at all, they only apply to the exceptions of the rule.


Comments are closed.

Reputation. Meet spriteCloud

Find out today why startups, SMBs, enterprises, brands, digital agencies, e-commerce, and mobile clients turn to spriteCloud to help improve their customer experiences. And their reputation. With complete range of QA services, we provide a full service that includes test planning, functional testing, test automation, performance testing, consultancy, mobile testing, and security testing. We even have a test lab — open to all our clients to use — with a full range of devices and platforms.

Discover how our process can boost your reputation.