ISTQB ATM Advanced Test Manager – Test Tools and Automation

1. Introduction

Welcome to the new chapter that is Taste, Tool and Automation. For this particular chapter.  Our discussion points would be tool Selection, tool Lifecycle and Tool Metrics. So let’s try to get into detail one by one.

2. Tool Selection

So, how the test tools should be selected and what are the criteria to be kept in mind while selecting the test tools. What are the options available? What is the pros and cons of different types of types of test tools? Let’s discuss all the details in this sub topic. So options are to look selection. There are multiple options available like purchase from commercial vendor. Some are open source to and some can be custom to cost benefit analysis of the tools ownership throughout the lifecycle helps in selection process. Careful analysis of the actual needs of testing group is required before selection of tools. The target should always have positive ROI that is written on investment on affords understanding the licensing scheme of selected tool is equally important because for some of the tools no matter whether you buy or whether they are open source or whether they are custom tool, they have limited period of license available. So keeping that license duration in time is equally important. So let’s talk about open source tool. What are the pros and cons of open source tool?

As we all know, open source tools are available for almost any facet of testing process from test case management to defect tracking to test case automation. There is no initial high purchase cost for this kind of tools. It can be modified or extended by users because ultimately it’s open source and code is available to public with core competencies, multiple open source tool can be combined to solve the problem vendor tool cannot address that is so right and these all are considered as pros of open source tool. But what are the cons? No formal support available for the tool. Of course nowadays there are many open source tool where they have discussion forums, support forums available but ultimately there is no typical customer care. Support is available originally created to solve a specific problem, the tool may not perform all functions of similar vendor tool and finally, accuracy of any open source tool is likely not certified.

Now let’s talk about custom tools to fulfill specific needs due to proprietary hardware problem customized environment process modified in unusual way test manager might wish to develop custom tool with the help of core competencies available in house custom tools must be designed and tested just like another software to make sure that it works as per expectations. Pros of custom tools that meets needs of team in precise manner because ultimately it has been developed in house by a team of experts, it can operate efficiently the way team wants tool can be developed to interface other tools in use data can be generated in exact form it’s needed. It may be useful in other projects in organization but what are the cons? Often dependent of persons creating tool because the team only knows all the details about the tools, if proper documentation not maintained, it may be orphan or fall into disuse. If the creator leaves the project over the time.

If the scope has been extended beyond initial intent, it might cause quality problems in the tool. So that was all about open source tool and custom tools. Now let’s try to understand what is ROI and how it should be considered when it comes to tool selection. So, as we all know, ROI is written on investment in cost benefit analysis. ROI should consider both recurring and non recurring costs, some of which are monetary and some of which are resource and time costs. Overall risk that may reduce the value of the tool the opportunity costs inherent in any tool. The time spent on acquiring, administering, training and using the tool could have been spent on actual testing tasks. Therefore, more testing resources may be needed upfront until the tool goes online.

What is non recurring cost and what does it include? These non recurring costs include defining tool requirements to meet the objectives and goals evaluating and selecting the correct tool and tool vendor purchasing, adapting or developing the tool performing the initial training for the tool integrating the tool with other tools procuring the hardware software needed to support the tool and the recurring costs include owning the tool, porting the tool to different environments, adapting the tool to future needs, improving the quality and processes to ensure optimal use.

Of selected tools when it comes to owning the tool. It’s like licensing and support fees. Maintenance cost for the tool itself. Maintenance of artifacts created by the tool. Ongoing training and mentoring costs. Risks to consider while deciding ROI. The immaturity of the organization like it is not ready to use the tool. The risk of artifacts created by the tool may be difficult to maintain, requiring multiple revisions when the software under test is changed. Reduction of test analysis involvement in testing tasks may reduce the value of testing like detection effectiveness may be reduced when only automated scripts are run. Reduction in repetitive work these all are benefits of tool introduction in process as I said, reduction in repetitive work.

Reduction in test cycle time because we are using automated tools for regression testing reduction in test execution costs increase in certain type of testing like as the tests are automated, you can do more regression testing. Reduction in human error in different phases of testing like test data may be more valid using data generation tools test result comparisons may be more accurate using comparison tools. Test data entry may be more correct by entering with a scripting tool. Reduction in afford required to access information about tests like tool generated reports and metrics reuse of test assets such as test cases, test scripts and test data increasing testing that was not possible without tools like performance testing, load testing, et cetera. Improvement in the status of testers who create automation and the testing organization as a whole by demonstrating the understanding and use of sophisticated tools. Now, let’s talk about tool selection process. How does it happen? Test tools are a long term investment and extends over many iterations of single project or multiple projects.

Different viewpoints should be considered while selecting the test tools. And which are those different viewpoints? Let’s talk about them one by one. So number one is from the business point of view, a positive ROI is required. Test tools and non test tools should work together in short collaboration with another tools being used. The viewpoints from the project point of view are the tool must be effective. The tool may require an appropriate amount of time in order to start earning positive ROI. Total life cycle should be considered well tool selection because it might happen that ROI may occur during maintenance rather than initial phase and from the person point of view, the usage of tool, the tool must support the project members in doing their task in a more efficient and effective way.

The learning curve must be considered to ensure the users will be able to learn the tool quickly with minimal stress and when first introduced, test tools require training and mentoring for the users. Tool selection process as per ISTQB foundation level includes assess organizational maturity, identify requirements for tools, evaluate tools, evaluate the vendor or service support, identify internal requirements for coaching and mentoring in the use of tools, evaluate training needs considering the current test teams, test automation skills and estimate cost benefits. Considering capabilities of Tool Test managers should consider capabilities as below for any tool, regardless of it in which phase it’s going to be used. The first one is analysis.

Will this tool be able to understand the input it is given? Is the tool suited for the purpose then it comes to design? Will this tool help design testware based on existing information? Can the design be generated automatically? Can the actual testware code? Has code be generated or partly generated in a maintainable and usable format? Can the necessary test data be generated automatically for data and test selection? How does the tool select the data it needs? Can the tool accept selection criteria entered either manually or automatically? Can the tool determine how to scrub production data based on selected inputs?

Can the tool determine which tests are needed based on coverage criteria? Like given a set of requirements, can the tool traverse the traceability to determine which test cases are needed for execution? Execution? Will the tool run automatically or will manual intervention be required? How does the tool stop and restart? Should the tool be capable of listening for pertinent events? Then it comes to evaluation. How will the tool determine if it has received a proper result? Like the tool will use a test oracle to determine the correctness of a response? What type of error recovery capabilities does the tool possess? Does the tool provide adequate logging reporting? So that was all about taste to selection process.

3. Tool Lifecycle

After discussing tool selection process, let’s talk about tool lifecycle. Stages of tool Lifecycle the first stages acquire that includes acquisition appoint administrator to make decisions about how and when the tool to be used and training to users. Then come support and maintenance. This is a continuous process. This process is mostly assigned to administrator or a tool group. Data interchange and cooperation process should be defined while working with another tools. Backup and restore of tool and its output decisions. Evaluation which time, environment, business needs and vendor issues might demand an update to the tool.

The tool vendor may require an update to the tool which causes issues with cooperating tools. A necessary change to the environment for business reasons may cause problems with the tools. Depending upon role tool play, test managers should ensure the process continuity while evolution and finally retirement. Graceful retirement when the tool has outstanded its useful lifetime. Functionality supplied by tools should be replaced. Data should be preserved and archived. This happens because the tool tool has reached to the point where benefits and opportunities of conversion to a new tool exceeds cost and risk. So this was all about tool lifecycle.

4. Tool Metrics

Let’s talk about tool metrics. Different tools collect different types of data. For example, if test management tools are there, traceability from requirements to test cases and automated scripts allows coverage metrics to be obtained. A snapshot of currently available test plant and current execution status whether it’s past failed, keep blocked in queue, et cetera. Can be taken at any time. When it comes to defect management tools, it can supply a wealth of information about defects including current status, severity and priority distribution throughout the system, et cetera. And the phase in which defects are introduced and found, escape rates, et cetera. For process improvement. When it comes to static analysis tools maintainability issues can be found out with performance tool scalability of the system can be identified. Coverage tools can identify how much of the system has actually been exercised through testing. So this is all about tool metrics and with this we are concluding this chapter, that is Test Tools and Automation.