testing software product interoperability

I have been trying to write on product interoperability for quite sometime. So here it goes. What do we mean by product interoperability? In my opinion, interoperability of a product is its ability to function while different variables of its environment – software as well as hardware – are changed. Testing software interoperability assumes more importance for software products as compared to enterprise applications or services offered under SaaS model. Enterprise and SaaS applications are hosted in an organization or in the cloud, in both cases the deployment and network configurations and other such variables remain constant for a considerably long time – discounting exigencies, of course. Any change made to an organization’s IT infrastructure impacts all applications and hence we do not worry about it when it comes to a particular application’s interoperability.

Designing a test plan for interoperability poses many challenges due to the sheer number of possibilities when it comes to environment configurations. Let me try and find a method to this madness.

1. Identify the external variables:  External variables are what I was talking about in earlier parts of this post. What could they be?

• Platform/Operating System(including version)
• SDK version – if it is a Java application then JDK version, if .Net based then .Net framework version. For other software development frameworks, there may other criteria.
• Browser (including version)
• Proxy/Firewall/Security – how does the application handle them
• Application Server (including version)
• Database(including version)

I am sure there can be more variables that can be added here.

2. Identify the internal variables: Internal variables are configuration/customization parameters provided by the application. Such configurations tend to be specific to the application. Even common configuration parameters like authentication and authorization vary in their implementation and interpretation. Other configuration options may include file size limits or support for encoding formats. These features are very specific to the type of product being built. Technically intensive products and/or those that target the technically proficient user will have more handles for the user to use.

3. Understand the most used set(s) of configurations. This is an exercise in perpetuity because the data that we will rely on to arrive at our configuration set “may” change with every new client. This job can get very mind boggling because the tester inside us will pinch us to cover even remote possibilities whereas the pragmatic side of ours will ask us to keep track of time and effort required. It is important to strike a balance in this regard. This balance can be achieved by using scientific methods like Orthogonal Arrays – not to forget experience and market research. Once we have a probably list in place we are good to go.

4. Setup the configuration (at least most of it): Depending on resources at disposal this exercise in itself can be a point of innovation. Limited resources will push you towards more planning for resource sharing.

5. Create an interoperability test suite: Arriving at this test suite requires us to keep in mind the number of software testers in the team, the degree to which test cases have been automated, the number of configurations arrived at in step 3 and obviously the time available for this exercise. In my opinion, this test suite lies somewhere between the “eagle eye view” of smoke testing and the “devil lies in the details” style functional testing.

6. Automate test cases: Please do not start thinking of QTP, LoadRunner and WinRunner. Automation is much more than that. Identify what exactly do you intend to test with a particular set of test cases. Now, explore the possibility of repeating this set of test cases across multiple installations of your product. This could involve using a spreadsheet or flat file to store data for testing, writing batch/shell scripts with adequate comments so that we can take a look at the output and see which test cases failed and which passed. I have found Ruby+WATIR to be really simple for web-based applications (Thanks Imtiyaz).

Hope this helps you get started with interoperability testing.