How to Automate Web App Installation Testing

Software testing at a company I used to work at included a suite of over 3500 unit tests.  Beyond those tests, there were a number of additional areas of development they engaged in to ensure that their customers had a successful experience with their released software, including:

• Automated installation testing of base configurations on supported operating systems
• Automated functional tests of end to end scenario-based testing
• Performance & stress testing

There are many challenges to each of the above areas of automated test development and I approached them in a manner that parallels the agile development process that ther product development group followed.  After all, test development is still, in fact, software development, but instead of requirements coming from external customers, the requirements are internal and are aimed at providing insight into the quality level of the application under test.

In this blog article, I will focus on the first bullet point above, automated installation testing, and I will post additional articles on the remaining topics in the future.

Automated Installation Testing

A key milestone for the QA team, prior to the company's first release, was to make sure our software installed and passed tests on all of our supported platforms (both Windows and Linux operating systems and both 32 bit and 64 bit versions).  The manual tests previously executed would then need to be automated to catch any regressions in the installers or tests going forward.

Running the installers automatically was a rather simple task since our installation executables are built with install4j.  After you run an installation manually the first time, it captures your input and stores it in a file.  The next time you want to do the exact same installation, you can provide that file as a command line argument, and it will complete without any additional interaction.  The key here is to start with a base state where the application is not previously installed.  For this we rely on “snapshots” of virtual machines in our deployment of VMWare’s Virtual Infrastructure.

Using VMWare Infrastructure

All the supported operating systems have a pair of virtual machines in our VMWare Infrastructure deployment: one for doing the nightly build; and one for doing an installation of the nightly build.  Those two processes are chained together using Bamboo by setting up two “plans” where the nightly scheduled build is one bamboo plan, and the installation is a “child plan” of the build plan.  When the installation plan fires off, it uses Ant to invoke a target that then executes a JUnit test (since we’re a pure Java shop).

It's always a good idea in QA to start with a known "base state" for system testing. VMWare gives you this ability with "snapshots". It's easy enough to fire up a VMWare Infrastructure Client desktop application to see your virtual machines and manage them, but how do you do that from inside a Java program?

The answer: VMWare's Virtual Infrastructure SDK. It includes the jars (java files), plenty of samples, and a development guide. The representation of virtual machines is communicated between a VirtualCenter server and the VMWare Infrastructure Client desktop application in the same way you would want to do it from a custom automation component that integrates with the rest of the framework you're developing.

One problem I had using the SDK is that it is so extensive that I found it providing a lot of functionality that I didn't necessarily need. I also found it quite challenging to wrap my mind around all the "Managed Objects" the SDK uses. The data representation and API routines take a while to get used to, but eventually I was able to setup a utility class in Java that contained just the methods I needed, taking just the parameters I wanted to provide:

• StopVM(“myvmname”)
• RevertVmToSnapshot(“myvmname”,”mysnapshotname”)
• StartVM(“myvmname”)

There were many iterations of tweaking the code to make it repeatable and there were some challenges that I worked around. For example, I inserted code to sleep for a couple minutes after invoking a Stop or Start command, and then had it call the VM to verify its state prior to executing the next command.  This enabled the tests to continue where they would previously fail because the VMs were not yet online or responsive.

Installing on a Remote Machine

Once this utility class was ready to use, I wanted to be able to roll out a distributed testing system where the communication between the build server, where Ant was running, and the target VM for the installation, was taken care of by some tool.  I looked at various tools, including SmartFrog, Distributed Junit, Ant-Server, and TestNG.

I decided against all of those for one or more reasons, and went with a much simpler method, but one in which I’d have to write more code to manage the details. The method I went with was to use SSH (Secure Shell). This required a SSH daemon running on a remote machine, and an SSH client to talk to it. The remote machines could be either linux or Windows. Running an SSH daemon on a linux box is very simple. But it proved a little bit more difficult on Windows Server 2003. I used OpenSSH and had to perform several manual steps at first to get it to work properly before automating the final solution.

The test clients could also be either linux or Windows, but since the JUnit test was part of a source code tree that is checked into a source control system, I decided on integrating a Java implementation of SSH2 from JCraft, called JSch, or Java Secure Channel. Here's a brief description of JSch from it's home page: “JSch allows you to connect to an sshd server and use port forwarding, X11 forwarding, file transfer, etc., and you can integrate its functionality into your own Java programs.”

Through trial and error, this all came together to enable a totally hands free installation of the web app installers that were compiled as part of the nightly build. The next step is to start the web app server, perform some setup steps, and run some tests on it through the GUI in a browser.

Smoke Testing the Installation

The functional tests use the example application that our SDK release ships with. From the build box, I perform a remote execution to start the web app server and run the setup steps. When that process completes successfully, I can run tests.

The web app also comes with a sample user interface for doing  via a web browser. Automating these tests was accomplished with Selenium RC. The automation invokes a series of tests via the Selenium server that resides on another VM.  The tests are run against each installed instance and the test results are captured as part of the overall test run.

Each morning I analyze the build reports. If the nightly build was successful, I know that all the installation, JUnit and GUI tests passed and that the build is ready for further testing.  If the build fails due to a test failure, the failure is noted in the build logs and the tests can be analyzed and defects reported to developers.