We have all come across performance issues, whether it be a painfully slow eCommerce site during the holiday shopping or simply a page taking so long to load that it times out. What would you do in that case? That’s easy to answer, as a visitor you would probably quickly move on to another platform offering the same service. However, when it’s your business that loses potential customers because of performance issues, the situation is more complicated.
The numbers around eCommerce revenue and performance issues make a good case for performance testing. Taking an eCommerce platform earning $100,000 a day as an example, a one-second delay could potentially amount to $2.5 million every year in lost sales. Statistics are one thing, but we all know how frustrating it is to use a slow website or application. It’s bad for your user experience and for your business’s revenue. Performance testing can help avoid these kinds of issues by uncovering the root causes behind a lagging performance and suggesting how to fix these issues.
In this guide, we will walk you through everything you need to know about performance testing, from the performance testing types to the tools you need to help you improve your performance.
If you landed on this page because you need help conducting performance tests on your applications check how spriteCloud can support you.
What is Performance Testing?
It’s always good to first understand the terminology, so we’ll explain what performance testing is. It’s the process of determining how a system behaves and performs under a certain workload.
In this context, the speed, scalability, responsiveness and stability of the system are meticulously examined. The performance of the system’s components is checked by passing different parameters in different load scenarios. Doing so allows us to determine the benchmarks and standards for the application under normal conditions.
Further tests can help to find how an application behaves when it’s put under extreme stress and when it reaches its breaking point. This can be helpful to plan for busy periods. Tests can also be focused on various areas of the back-end to determine where performance issues reside.
For example, in a recent project our performance engineer discovered that the application was running slow because the CPU was spending its efforts on garbage collection rather than running the application. Read the case study or watch the video to see how we supported this client and how we could help you.
What happens when a website is subjected to a flood of visitors on Black Friday for example. Will it remain stable and fast? When will it reach the breaking point?
Performance tests aim to proactively find out if an application meets the speed, stability and scalability requirements needed to keep your business goals attainable.
Why is it important to conduct performance tests?
We all had the experience of trying to use a website with a slow internet connection and the frustration that nearly made you smash your computer or phone. Performance testing helps ensure that your applications never make your visitors feel this frustration and leave your application looking for a faster competitor.
You might think your website is fast enough. And, worst-case scenario, what are a few seconds of delay? Your visitors certainly disagree.
As a matter of fact, 40% of people abandon a website that takes more than three seconds to load. And this number reaches a whooping 90% user abandonment rate after 5 seconds. When it comes to user experience (UX), speed is a crucial factor. Irrespective of the number of concurrent users accessing your application, the device or browser they use, or their connection speed, your application should aim to deliver consistent performance. Conducting performance tests allows you to consistently offer an excellent experience to your customers.
From a marketing perspective, since Google has now placed an importance on page speed with its Web Core Vitals update to its search algorithm, speed also impacts organic search rankings (SEO). You may not depend on organic traffic for revenue but it is typically the best converting marketing channel. After all, the best place to keep a secret hidden is on page two of Google search results.
Many companies are also moving away from hosting physical servers on their premises to using cloud services. These usually have a pay-as-you-go pricing structure, in which an application with performance issues requires more space and compute time on the servers, needlessly costing you money. Optimising your performance can therefore have a direct impact on your operating costs.
All in all, may it be because you are losing customers, have less traffic coming to your website or have high operating costs, it’s clear, performance issues have a negative impact on your bottom line.
To help you make sure you avoid these consequences, continue reading this guide to performance testing for more tips and best practices, and don’t hesitate to get in touch with spriteCloud if you require the assistance of our expert Performance Engineers.
What performance metrics should you monitor?
This won’t come as a surprise, before you start testing it’s important to decide what measurements will be used to assess your system’s performance and allow you to set performance goals. There are many possible metrics, but the four most important ones are response time, the number of concurrent users, throughput and resource utilization.
Response time
The response time of an application may well be the most important one as it determines how responsive an application is to a user’s request, i.e. how fast does the webpage feel. And as we know, slow response times contribute to a poor user experience, and a potential loss of revenue.
How is this measured? It’s pretty straightforward, the time it takes the application to process a user’s request and send the response back to them is the response time.
Concurrent users vs. simultaneous users
Concurrent users are another crucial term for performance testing. They are used to determine how much load an application can take before it becomes unresponsive and its performance plummets. A common misconception is to see these as simultaneous users. Concurrent users are not all using an application at the exact same time and requesting the same thing. They access an application and request different things within a certain time frame.
Let’s look at an example, on an eCommerce website one customer might be browsing through the books of an online shop. Three other customers forgot their password and are requesting a new one. Another one could be creating an account. All these user requests use different resources of your application during a certain time interval. However, the only ones who could be considered simultaneous users are the ones who are requesting a new password and only if they do so at the same time. So concurrent users are a more realistic test of real world conditions.
Throughput
When conducting a test, one will also look into the throughput, this is the number of requests an application can handle in a particular time period. In the performance testing jargon, this is also referred to as transactions per second (TPS).
This metric is useful to consider for occasions like sale periods, like Cyber Monday or the launch of new marketing campaigns where suddenly there may be many more users on a particular webpage.
Resource utilisation
Finally, let’s look at resource utilisation. This indicator is rather broad and encompasses the different ways in which a system’s resources are used during a performance test. Simply put, it tells us how and which resources are busy when a performance test is conducted. There are many different metrics that can be used to monitor utilisation, the most common ones are CPU (central processing unit), memory, disk and network utilisation.
The CPU utilisation can be visualised as a system’s brain: if it has to deal with too many requests or there’s a single task that requires all its attention it can get overwhelmed. The percentage of time the CPU needs to handle a process allows you to know if your test is effective and to flag performance issues. High percentages, usually around 80%, signal possible performance problems. It’s also possible to use the CPU as a general indication of how a system is performing after a change has been made.
Memory usage specifies the amount of memory needed to process a request. If this is too high the system may be slowed down. Unusually high numbers suggest that there is a memory leak, which is a red flag for any performance tester. Disk usage corresponds to the time a disk needs to execute a request.
Finally, monitoring network utilisation, the ratio between the current traffic and maximum traffic, will allow you to find out how busy a network is. If network utilisation exceeds what it should be under normal conditions, the consequences will be low transmission speed, intermittence or request delays for example. Tracking if the network utilisation is idle, normal or substantial allows to establish benchmarks and troubleshoot network failures should they happen.
Depending on the project and business needs other metrics will be included and certain metrics will have a higher priority compared to others.
What are the different types of performance tests?
Now that we explained what performance testing is, why it’s important, and what key metrics to consider, let’s dive into the different types of performance tests and what their goal is.
Load Testing
Here the idea is to understand how the system behaves under increasing user load, to see how the performance is impacted by this increase, and finally to determine the maximum capacity of the system.
Can the system cope with user requests when there are 100, 1,000 or 100,000 concurrent users? Is the performance consistently reliable, even with a high load? When does the system reach its limits? All questions that load testing can answer.
In this process, we want to simulate real-life conditions. Let’s quickly come back to what we discussed in the section above about concurrent users. They are not necessarily simultaneous users.
In a load test, we want to simulate a realistic situation where many concurrent users are doing different things on a website during a certain time interval. This test should emulate real-world conditions. This is different from having all users doing the same action on a website at the same time (aka simultaneous users).
Stress Testing
A stress test looks into the system’s performance when it’s subjected to extreme load. It checks if the system remains stable even after it reaches its tipping point. To do so the load is gradually increased until the application becomes unresponsive. Beyond that, a stress test also examines the mechanisms in place when a system fails, such as error management and recovery procedure.
Does this sound too abstract? Have a look at our case study, we recently performed a stress test for a renowned eCommerce platform.
Endurance Testing
As the name indicates, endurance tests are conducted over a longer period. A continuous load is put on the system and different responses are monitored to detect if there is a loss of performance. For example, memory utilisation is usually tracked to see if there are any leaks. An increase in response time or decrease in throughput can also signal a performance deterioration, ultimately signalling that the system isn’t as robust as expected.
Volume Testing (or Flood Testing)
Here the goal is to test the efficiency of a system regarding data processing. Can it cope with voluminous quantities of data? During a volume test, the amount of data in a database is dramatically increased and the general system behaviour is observed to detect possible bottlenecks. Relevant metrics to track could be the response time of the database, its memory utilisation, as well as proper data storage, potential data loss and data integrity.
Scalability testing
This test is undertaken to find out if the application’s number of concurrent users and user requests can be scaled up or down without seeing a deterioration in effectiveness. In a nutshell, this type of test looks into how adaptable a system is to varying needs considering the maximum number of users it can sustain without compromising user experience.
That is particularly useful before an expected increase in traffic, like in the case of a new marketing campaign being launche