Test Automation

Our client was rolling out a second generation version of their multimedia site, developing with agile methodologies. The requirement was to test the javascript components and user experience on multiple platform and browser combinations each time new code was committed to the project.

Automation scripts were built using Ruby and Watir, a framework for driving browsers such as Internet Explorer and Firefox. The many browser and operating system configurations was achieved by utilizing a virtual server farm, to expand out to 15 combinations of Internet Explorer, Firefox, Safari, Chrome, Windows, Linux and Mac.

Automation test runs took less than 30 minutes, and performed the tests up to 12 times a day. It was estimated that a manual test run would take 2 people 8 hours to perform the same tasks and it would only be performed once a week. Bugs were reported within 30 minutes of creating them, resulting in a faster turn around time for the developers, whose code fixes were verified in the very next round of automated test runs.

Automation tests covered 95% of the user experience and resulted in a drastic and measurable decrease in the number of defects reported by manual testing as well as the percentage of each sprint dedicated to fixing defects.

Performance Testing

Our client had specific operation on their website, which they needed to get tested before a tight deadline. They contacted us and in a very quick pace, we created test automation, setup it up to run from Amazon EC2 instances and got the results they needed.

Websites that do more than show static pages can have surprising problems when a high number of users hits the site. Another typical characteristic of websites is that the number of visitors can change quickly, even as a result of a link to your website being posted on a high profile website. That will lead to lot of visitors and help in your site becoming more popular due to viral marketing.

This is exactly the time when you don't want your site to fail as viral marketing also delivers a negative message very quickly. A high load time for the pages, let alone the site not working at all, will quickly lead to people leaving and going to a competing site.

User Acceptance/Compatibility Testing

While many sites still concentrate on supporting only Microsoft browsers, there are a number of alternatives with all the time increasing popularity. Firefox is by far the most popular "alternative" browser, which is getting close to 50% market share in some countries. Google's Chrome has gained 5% popularity in a very short time according to some statistics. This makes testing on multiple browsers a must on all but the most basic websites.

Our client wanted to make sure their website works well with a number of different browsers on 3 different operating systems, which cover 95% of their user base. We setup a test plan where we went through the basic functionality on the following browser/operating system combinations:

We run through the basic functionality of the site to make sure an average users usage patterns were covered.

Note: We can run full test plans and even create test automation for the complete functionality of the site, but that takes more time and needs to be discussed and planned separately.