ISTQB CTFL-2018 – 2018: Tool Support For Testing

1. Tools Support for Testing: Introduction

As seen in the previous sections, there are many tasks and activities that need to be performed during the testing process. In order to assist in making the testing process easier to perform and manage, many different types of tools or software applications have been developed and used for a wide variety of testing tasks. Some of them have been developed inhouse by an organization’s own software development team and testing department. Others have been developed by software houses to sell to organizations that perform testing.

This section discusses the potential benefits of using tools. We will see that success with tools is not guaranteed even if the most expensive tool is acquired. There are also risks in using tools even though we will not mention any specific tool by name. But we will describe the most commonly used types of test tools and concludes with a process for introducing a tool into a test organization. We can see various test tools used for one or more activities that support testing. These include tools that are directly used in testing, such as test execution tools and test data preparation tools. Tools that help to manage requirements, test cases, test procedures, automated test scripts, test results, test data and defects, and for reporting and monitoring test execution tools that are used for investigation and evaluation. For example, tools that monitor file activity for an application. Any tool that assists in testing a spreadsheet or email application are also a test tool.

In this meaning, test tools can have one or more of the following purposes, depending on the context improve the efficiency of test activities by automating repetitive tasks or tasks that require significant resources when done manually. For example, test execution, migration testing improve the efficiency of test activities by supporting manual test activities throughout the testing process. Improve the quality of test activities by allowing for more consistent testing and a higher level of defect vbility. Automate activities that cannot be executed manually, for example, largescale performance testing increase reliability of testing, for example, by automating large data comparisons or simulating behavior.

2. Test Tools Classification: Introduction

Tools can be classified based on several criteria, such as purpose, pricing, license model, for example, commercial or open source, and technology. Used tools are classified in this syllabus according to the test activities that they support. Some tools clearly support one or mainly one activity. Others may support more than one activity but are classified under the activity with which they are most closely associated. Tools from a single provider, especially those that have been designed to work together, may be provided as an integrated suite and bundled into one package.

Isttb classified the testing tool into the following categories tool support for management of testing and tests tool support for static testing tool support for test design and implementation tool support for testing, execution and logging tool support for performance and monitoring tool support for specific testing needs. First, notice that in the syllabus that all the previous points are of type K one, which means you only need to remember them. You only need to know that they exist. Second, I want you to think carefully about each title. It says tool support for whatever tool support, not tool that will do whatever. This means that those tools are helper tools, not necessarily tools that will do the actual job. So for example, tool support for management of testing are tools that will be used during the management of testing activity and not necessarily our tools that will do the actual management of testing for us. So an email application can be considered as tool support for management of testing and tests and so on. Some tools offer support that is typically more appropriate for developers tools that are

used during component and integration testing. For example, such tools are marked with D next to its name or title. There are something extra that the Iced UKB mentioned in its foundation curriculum, so I will share it here with you. A tool that measures some aspect of software may have unexpected side effects on that software.

For example, a tool that measures timing for non functional performance testing needs to interact very closely with the software in order to measure it. A performance tool will set a start time and a stop time for a given transaction in order to measure the response time, for example. But the act of taking that measurement, for example, storing the time at those two points, could actually make the whole transaction take slightly longer than it would do if the tool wasn’t measuring the response time. Of course, the extra time is very small, but it is still there, it still exists. This effect is called the probe effect, and tools suffer from this effect are called intrusive tools.

We will learn now about many types of tools. We will not mention any tool by name. Even if I needed to mention any tool by name, then it’s only for clarification purposes, you don’t need to memorize it. Therefore, we only care about the kind of tools questions in the exam related to this part are asking about what does the tool do? Main feature, which category does it belong to? Which step in the testing process does it use in? And finally, who uses it? So I will try to highlight those points when I talk about each tool.

3. Test Tools Classification: Part 1

This broad category contains any tool that provides support for managing the tested process over the entire software development lifecycle. Think of most of those tools as databases where everyone is using those tools either by adding content to the database or by getting reports from those databases to help taking decisions to direct the testing activity tools are under this category are Test management tools. Everyone is using the test management tools, but those tools are mainly used by test managers and test leaders, mainly to create reports that would help them manage the testing activities and make decisions. Such tools are also called ALM, which stands for Application Lifecycle Management Tools. We also have under this category requirements Management Tools.

Tools that help managing the requirements are also tools that can be very helpful in managing the testing process. Such tools help to trace tests to requirements and requirements tests. Everyone is using the requirement management tools, but particularly system analysts. Defect Management Tools We have talked about defect management tools in a previous section. Defects are used heavily during the implementation and the execution process and used by Everyone, but mainly are used by testers and developers. Configuration management Tools and also we have talked about configuration management tools before, and configuration tools are used by Everyone and Last. Continuous Integration Tools continuous integration is built on other software development best practices, including automated testing, version control, build automation, and automated deployments. Each one of these practices of continuous integration has its own collection of tools, and as you see, it’s mainly used by developers. If we can talk more about test management tools, we can say that test management tools often need to interface with other tools or spreadsheets for various reasons, including to reduce useful information in a format that fits the needs.

Of the organization to maintain consistent flexibility to requirements in a requirement Management Tool to link with test object version information in the Configuration Management Tool. This is particularly important to consider when using an integrated tool, for example Application Lifecycle Management, or ALM as we have said before, which includes a test management module and possibly a defect management system as well as other modules, for example, project, schedule and budget information that are used by different groups within an organization. Another category of tools are tool support for static testing.

Static testing tools are associated with the activities and benefits described in the static testing section we have talked about before. Examples of such tools include tools that support reviews. We have discussed the review process and different review types in a previous section. Review process support tools are any tools that can be used to help or support the review process. We also have static analysis tools. This is mainly or normally used by developers as part of the development and component testing process prior to dynamic testing.

The input to those tools is usually the source code, and the tool will list all the possible defects it finds in the code without running it. Another category for tool support for test design and implementation. Test design tools aid in the creation of maintainable worker products in test design and implementation, including test cases, test procedures and test data. Examples of such tools include test design tools, modelbased testing tools, test data preparation tools, acceptance testdriven development at DDD and behavior driven Development BDD Tools and last test driven Development TDD tools, which are used by developers. Let’s talk a little bit about each one of them. Test Design Tools Test design tools are used to generate automatically test cases with inputs and expected results, or at least test inputs only. There is a common term in the testing industry called test Oracle. Oracle here has nothing to do with the famous database provider Oracle.

Oracle is an English word that means computerized, robotic or automatic. Therefore, a test oracle means any computerized means to help generate test cases or test inputs. We expect to provide such tools with some kind of input. For example, the requirements document, the design document, the graphical user interface or the database model and the output from those tools would be some sort of ready made test cases or test inputs. Imagine if we have a document about the graphical user interface of a system and the fields in each screen. I guess we can create a tool which can take such document as an input and the output will be distributed for the number of fields. We would have a test case for the above range, a second district case for within range, and a third district case for the below range using equivalence partitioning test design technique.

If ever messages are stored in this document as well, then the expected result for each test case can be specified as well. Just as like creating test cases automatically, we can have tools that automatically create that test data and all the test results for us. Needless to mention, we may end up with the problem of having too many tests and then we would need to find a way of identifying the most important tests to run. Second, model Based Testing Tools as we have mentioned before, model based testing is a technique which is used by the team of testers to check the runtime behavior of a software under test against the predictions made by a formal specification of model. Model based testing of MBT tools enable a functional specification to be captured in the form of a model such as an activity diagram. This task is generally performed by a system designer. The MBT tool interprets the model in order to create test cases specifications which can then be saved in a test management tool and or executed by a test execution tool. If you need to learn more about model based testing, you can refer to IST KB MBT Foundation Level Model Based Testing syllabus. Third category under the tool support for design and implementation is Test Data Preparation tools.

Now, imagine if you want to have a test case where you need to test adding a million rows of data to a table in a database which already holds millions of records of data as a stress testing test case, would you be able to create the data needed manually? No way. Tested data preparation tools enable data to be selected from an existing database or created, generated, manipulated, and edited for views in tests. The most sophisticated tools can deal with a range of files and database formats.

In some cases, tools that support test design and implementation may also support test execution and logging, or provide their outputs directly to other tools that support test execution and logging. Next, we will talk about Acceptance Test Driven Development ETDD and Behavior Driven Development BDD tools behavior Driven Development and Acceptance test Driven Development involve generating test conditions and test cases from user stories and acceptance criteria period to coding.

They also verify, validate and detect defects in the user stories and acceptance criteria. You can learn more about those techniques in the Isqb Foundation Level Agile Testing extension. Last Test Driven Development TDD Tools test Driven Development, or TDD, is an approach with the exact reverse of traditional development. In this approach, testing is done first and then the code is written. As you can see, tools that support TDD are mainly used by.

4. Test Tools Classification: Part 2

Tools support for Test Execution and Logging Many tools exist to support and enhance test execution and logging activities. All the types of tools mentioned under this category are used in the execution part of the testing process. Examples of these tools include test execution tools for example, to run with vision tests cover edge tools, for example requirements coverage or code coverage. Test Harnesses and Unit test Frameworks let’s talk about each of them in a little bit of detail. Test Execution Tools When people mention testing tool, Test Automation Tool is the one that comes to mind. Test Automation Tool or Test Execution tool is a tool that can run tests for you automatically. So instead of executing a test procedure manually by hand, we would write the test procedure in a way that the computer will understand and the computer can execute it automatically.

To write the test procedure in a way that the computer will understand would mean that we might need a specific language to write the testable procedure, which in this case will be called Test script. So a test script is a testable procedure written using a scripting language such as scripts would need to store the inputs that will be used by the script and maybe store the expected outcome as well. If the tool can compare the expected result versus the actual result, then the tool should usually provide a test log for each test run with the result of the test execution and test comparison. If the tool doesn’t store the expected outcome of the script and therefore cannot compare the actual outcome of the test versus the expected result, then we could call the tool semi automatically. In this case, we would need to store the actual outcome of the test execution for further investigation.

They can also be used to record tests and usually support a scripting language or Guid based configuration for parameterization of data and other customizations in the test. Although they are commonly referred to as testing tools, they are actually best used for regression testing, so they could be referred to as regulation testing tools rather than testing tools. A test execution tool most often runs tests that have already been run before.

One of the most significant benefits of using this type of tool is that whenever an existing system is a change, for example, for a defect fix or an enhancement, all of the tests that were run before could potentially be run again to make sure that the changes have not disturbed the existing system by introducing or revealing a defect. We will talk more about those types of tools in a separate lecture. Coverage Measurement Tools We have learned how to measure test coverage in a previous section, but do we actually do it manually? Thankfully not. Coverage tools are here for the risk queue and they are mainly used by developers. Last, we have test Harnesses unit test framework tools also used by developers. Remember, the steps and drivers we mentioned before when we talked about unit testing and integration testing. Well, the stops and drivers are types of test harnesses. Test harness is any code written by the developer to help testing the code.

There are also unit test frameworks that would help the developer to write such test code tools. Support for Performance Measurement and Dynamic Analysis Performance measurement and dynamic analysis tools are essential in supporting performance and load testing activities as these activities cannot effectively be done manually. Examples of these tools include performance testing tools which are concerned with measuring the system performance to see whether or not the system will stand up at different circumstances. Under the umbrella of performance testing, there are different kind of testing, for example load testing, volume testing and stress testing. Also, we have monitoring tools. I want you to think of monitoring tools as task manager under the Windows operating system. It can show you measures and statistics about your software.

How does your software use the CPU, memory networks and other system resources? I always advise or recommend to my sisters to keep the task manager open and visible all the time while testing dynamic analysis tools which are used by developers. We have mentioned before static testing tools and they are tools that could find defects in the software while it’s not running. And now we are talking about dynamic testing tools. So from its name, what do you think it do? Correct dynamic analysis tools are tools that can find effects in the software while it is running. Last, we have tools of both for specialized testing needs. In addition to tools that support the general testing process, there are many other tools that support more specific testing issues.

Examples of these tools include data quality assessment, data conversion and migration usability testing accessibility testing localization testing which is converting the software to a specific language to be local in some country security testing portability testing for example, testing software across multiple supported platforms.