Test automation in continuous integration/continuous delivery (CI/CD)

Why continuous delivery?

A key thing to be able to compete in the market is to be able to build new products and services faster than other companies. That’s why the usage of Agile and DevOps is increasing. Software products need to be released on the shortest cycles possible. This need for fast development cycles leads to another problem. The potential for more bugs.

CD pipeline Dilbert comic
Source: https://www.qmetry.com/

Why Automation Testing is important for CI/CD?

The key to integrating quality into our software is ensuring that we can get quick feedback on the impact of new changes. Continuous Testing helps to realize the speed and flexibility of agile and ensures faster recovery in the case that any defects are found in the product. Test automation makes it possible to test fast and test early, and gives stakeholders more accurate information on risk, helping them make better decisions at the business level. In order to achieve a high quality of software, we need to run different types of tests continually within the delivery process.

continuous integration in agile methodology
Source: http://tryqa.com

Automation testing is important for a successful Continuous Integration/Continuous Delivery (CI/CD) pipeline because:

  • Easy and efficient assessment of minor changes – considering that in a continuous integration process most of the changes are small and test automation is able to cover them, the team can continuously deliver changes which have already been tested enough. Besides automated tests, some infrequent manual testing activities remain crucial after a major update of software, like usability testing.
  • Faster regression tests – the continuous delivery process requires quick feedback. Automation testing can be executed much more quickly than manual testing and provide results quickly. To help shorten the overall execution time, automated tests can run in parallel.
  • More consistent results – an efficient continuous integration pipeline should not contain too many variations or anomalies. To meet expectations all the way to the release, process consistency should be maintained.
  • Better agility – new technologies appear and requirements change. That’s why a good pipeline should be flexible to update frameworks, adjust configurations and introduce new tools. Having too much manual testing makes a CI/CD pipeline less agile. Pipelines utilizing test automation can be reconfigured automatically. Read more about the spriteCloud agile testing approach here.

Test automation is a complex topic but incredibly important in today’s software development projects. For more information on test automation head to our guide titled “Test Automation: nearly everything you need to know.”


Deployment pipeline

Deployment Pipeline
Source: https://blogs.msdn.microsoft.com

A key pattern of CD is a deployment pipeline which includes continuous integration and test automation. As shown in the diagrams, code changes should pass a set of tests:

  • Unit tests
  • Automated regression tests (functional tests)
  • Exploratory and usability tests (functional tests)
A diagram of the Continuous Delivery Process
Source: https://leanpub.com

In the continuous deployment pipeline, every code change could be a release candidate. One of the tasks of a deployment pipeline is to catch regression issues. If no regression problems can be detected, packages could be released successfully. If any defects are discovered later, the pipeline should be improved by adding or updating tests.

Problems should be found as soon as possible and the time from check-in to release should be shorter. Parallelization of activities in the deployment pipeline instead of many consecutive stages execution could help. The feedback process is important as well:

  • If bugs are discovered in exploratory testing, automated tests should be improved.
  • If defects are discovered in the acceptance tests, unit tests should be improved (unit testing should find most of the defects).

A reliable approach to automation testing

Stages of development and testing

The ability to identify tests for automation is key to creating the best technical approach. Frequently repeated tasks are consuming the most resources and are, therefore, the best candidates for the automation process. Once automation testing is completed then manual testing is still required. In the context of automation, there are various types of tests.

The test pyramind
Source: http://james-willett.com

Unit testing

This type of test checks that individual units of the code are functioning and could be integrated into the repository. Usually, these tests are focused on locations where bugs might be hiding and on individual functions. Unit tests should be very fast and cover around 80% of the codebase. This can give enough confidence that the application is functioning properly at this point. If unit tests are not automated, there will not be much chance to achieve success.

Component integration testing

Integration tests are executed to find defects in the interactions between integrated components or systems.

User Interface (UI) Testing

UI testing checks that the frontend interfaces of an application are shown and working as it was described in requirements, in all supported browsers and platforms. Virtual machines can be and are often used to test multiple combinations at high speed in parallel.

API (middleware) testing

API tests are checking the interoperability between software systems and components. These tests simulate calls to API endpoints and validate the responses from the server.

Functional Software Testing

During functional testing, different functions of the app are compared against the requirements or specifications. A tester represents the user or customer, who is using the application. Functional testing focuses on the user functionality rather than internal interaction of the application’s components.

Regression Testing

The goal of regression testing is to discover bugs that might occur because of enhancements and bug fixes. In order to validate that existing functionality is still working, existing test cases should be executed against new code changes prior to the release. To save time during frequent releases, teams sometimes spend less time on regression testing. Doing this could mean that bugs could reach the end-users. This is a type of testing that is a good candidate for automation.

World's most interesting programmer.

The challenges in Test Automation

A lot of effort and expertise is required to create and configure test environments and an effective automation framework. To set up the right automation framework more time is required and costs could be higher. Another challenge is to find test automation experts with the right skill sets who can set up and maintain the test automation framework, build tools, and fix technical issues. spriteCloud and other organisations offer their test automation expertise to help teams achieve this. spriteCloud can provide automation testing for you completely or consult you in the setup and maintenance of a test automation suite.

Common questions about automation testing in a CI/CD pipeline.

Just choosing a tool and then immediately start using it if it works, and then rolling back to manual processes if it doesn’t, is a bad practice. Don’t do this. Automation should be integrated into the continuous testing process early across application lifecycles and architecture layers, from the API to the web, mobile, business intelligence (BI) and data warehouses to deliver comprehensive automated testing capabilities.

How to start introducing test automation into the development process?

The most important functionality in the system should be automated first. Running those on every commit is the next step. Then add new tests to cover changed functionality and newly added functionality. It will eventually grow to an extensive automated test suite. It’s better to have a couple of tests that run quickly and are stable, than a large number of tests that are weak, random, and frequently failing. In this case, the team will likely ignore the tests, jeopardizing the integrity of the project.

Who should create and maintain automated tests?

Everyone in the team. Developers should especially work on the creation and maintenance of test automation suites. Fixing them whenever there is a failure should have priority. It motivates developers to write testable software. Developers don’t care about software’s testability when automated tests are created and maintained by a separate team. Retrofitting automated tests onto such systems is very expensive. Poorly designed untestable software is a major factor contributing to automated test suites that are expensive to maintain. This is one reason why DevOps is growing in importance.
If you feel your team doesn’t have the in-house knowledge to set up test automation, spriteCloud has a lot of experience in setting up and training development teams to handle automation testing.

Should all manual testers be fired in continuous delivery process?

No. Testers know how users interact with the application. Developers should work together with testers to understand their perspective, and testers can learn more about test automation. It’s important that testers perform exploratory testing continuously. As with anyone in the IT industry, testers have to learn new skills.

Should all tests be automated?

No. Manual activities like exploratory and usability testing can’t be replaced by test automation. In order to make sure that the product is working properly, a usable continuous development process should include all types of test activities.

Calliope Pro

The advantages of continuous testing by running test automation in a continuous integration/continuous delivery pipeline should be pretty clear at this point. Our tool Calliope Pro is an excellent solution for dashboarding your test results in a way that all project stakeholders can benefit from. Calliope Pro is tools agnostic, meaning it will work with any testing tools or frameworks you use. It is also free so you can evaluate its usefulness to your team, without consequence.

Written by: spriteCloud

This is the spriteCloud company author. We use this profile when posting general company-related things and as a catch-all for posts that were written by former colleagues.

Subscribe to our mailing list!

Stay up-to-date on all things quality assurance,
test automation, and cybersecurity.

We’re spriteCloud, a leader in software and cybersecurity testing.

Aside from interesting articles, we also have a team of software testers that can help your organisation.

Have a look at our testing solutions.