ISTQB CTFL-2018 – Test Management Part 2

4. Test Strategy and Test Approach

A testing strategy provides a generalized description of the test process, usually at the product or organizational level. The testing strategy describes at a high level, independent of any specific project, the how of testing for an organization. The choice of the testing strategy is one of the powerful factors, if not the most powerful factor in the success of the test effort and the action you see of the test plans and estimates. Let’s look at the major types of test strategies that are commonly found analytical. In analytical test approaches, our testing will be based on the analysis of some factor, usually during the requirements and design stages of the project, to decide where to test first and where to test more and when to stop testing. For example, the risk based strategy involves performing a risk analysis using project documents and stakeholder input.

Riskbased testing gives higher attention to areas of highest risk. Another analytical test strategy is the requirements based strategy, where an analysis of the requirement specifications forms the basis for planning, estimating and designing tests. Model Based In model based approaches, we create, design or benchmark some formal or informal model that our system must follow. The model will be based on some required aspects of the product.

For example function, a business process, an internal structure, or a non functional characteristic like Viability for example. To give you an example, our software response time should be faster than that of the competitors software. We will keep on testing our software until the behavior of the system under test confirms to that predicted by the model. Examples of such models include business bosses models, estate models, and reliability growth models. Methodical this type of test strategy relies on making systematic use of some predefined set of tests or test conditions such as a taxonomy of common or likely types of failures, a list of important quality characteristics or companywide look and feel standards for mobile applications or web pages.

Examples of methodical approaches are failure based, including error guessing and fault attacks. A checklist based and quality characteristic based bosses are standard compliant. In this approach, you might adopt an industry standard or unknown process to test the system. For example, you might adopt the IEEE 29 one one nine standard for your testing.

Alternatively, you might adopt one of the agile methodologies such as extreme programming process or standard compliant strategies have in common reliance upon external rules or standards. Reactive in this type of test strategy, testing is reactive to the component of a system being tested and the events occurring during test execution rather than being replanned as the preceding strategies are, tests are designed and implemented and may be immediately be executed in response to knowledge gained from period test results. Exploratory testing is a common technique employed in reactive strategies, consultative or directed.

This type of strategy is driven primarily by the advice, guidance or instructions of stakeholders, business domain experts or technology experts who may be outside the test team who may be outside the test team or outside the organization itself. For example, you might ask the users or developers about the system to tell you what to test or even rely on them to do the testing themselves. Regression Averse this type of test strategy is motivated by a desire to avoid regression of existing capabilities. Regression adverse test strategy includes reuse of existing testware, especially test cases and test data, extensive automation of regression tests and standard test suites so that whenever anything changes, you can rerun every test to ensure nothing has been broken.

So which approach is the best? Again, there is no one best answer to this question. An appropriate test strategy is often created by combining several of these types of test strategies according to your own situation. For example, risk based testing and analytical strategy can be combined with exploratory testing and active strategy. They complement each other and may achieve more effective testing when used together. While the test strategy provides a generalized description of the tested process, the test approach is the implementation of the test strategy for a specific project or release. So when you assess the risk of your project, as we mentioned in a previous lecture, and refine your project objectives and your project test objectives, then you can decide on the test approach you will take to test your project based on your organization.

Test Strategy the test approach will be reflected in your decisions in planning the test effort. It’s the starting point for planning the test process, for selecting the test techniques, test levels and test types to be applied, and for defining the entry and exit criteria or the definition of done. The tailoring of the test strategy is based on decisions made in relation to the complexity and goals of the project, the type of the product being developed and product. Risk analysis questions in this topic in the historical exam are more about defining a situation and ask you which best approach or strategy to use. For example, if you are using Agile or Waterfall then what the approach you are using?

Correct process or standard compliant? If you are asking the user which areas to test, then what’s the approach you are using Directed? If you are using test cases from an old version of the software, then what’s the approach you are using? Aggressive, adverse and so on. So how do you know which strategies to pick or blend for the best chance of success of your project? There are many factors to consider, but let’s highlight a few of the most important ones. Risks testing is about risk management, so consider the risks and the level of risk for a well established application that is evolving slowly. Regression is an important risk, so regression adverse strategies make sense for a new application.

A risk analysis may reveal different risks if you pick a risk based analytical strategy, available resources and skills so you have to consider which skills your testers possess and lack. A standard compliance strategy is a smarter choice when you lack the time and skill in your team to create your own approach. Objectives testing must satisfy the needs of stakeholders to be successful. If the objective is to find as many defects as possible with a minimal amount of upfront time and effort invested, then a reactive strategy makes sense.

Regulations sometimes you must satisfy not only stakeholders but also some regulations. In this case, a methodical test strategy may satisfy those regulations. Product the nature of the product and the business. For example, a different approach is required for testing mobile phone coverage than for testing an online banking operation. Safety of course, safety considerations promote more of formal strategies. And last technology.

5. Test Planning

A test plan is a project plan for the testing work to be done in development or mentors projects. Planning is influenced by many factors. For example, planning is influenced by the test policy or strategy of the organization, the development life cycle, the scope of testing, objectives, risks, constraints, criticality, and the availability of resources. Having a test plan guides our thinking, use as a vertical for communication with other members of the project team, testers peers, managers and other stakeholders and also help us manage a change.

A test plan ensures that there is initially a list of tasks and milestones in a baseline plan to track progress against, as well as defining the shape and size of the test effort. Any plan in the world should contain at least what to do, when, how, by whom, how to manage a change in the plan and how to track the progress of the plan and how much it will cost. If your plan doesn’t contain all these elements within the plan, then there’s something missing. In small projects, your test plan could be a single paper. In larger projects, you might find it useful to write multiple test plans for different test levels.

So you would have a test plan for unit testing, another plan for integration testing, another for system testing and another for acceptance testing and a master plan to according to the work between the different plans. In the unit testing plan, for example, you would have what to be done, when, how, by whom, how to manage the changes and how to track the progress of the plan according to whoever is doing the unit testing. In the system testing plan, you would have again what to be done, when, how, by whom, how much, how to make the changes, and how to track the progress of the plan and so on. You can even have separate test plans for test types such as usability testing and performance testing. We find that using a template when writing test plans help us remember the important challenges. You can use a test plan template from either standard 29 1119 three or you can use someone else’s template or create your own template.

Over time as a project and test planningprogress, more information becomes available and more detail can be included in the test plan. That’s why tested planning is a continuous activity, so it’s very normal to replen if things didn’t work out. Feedback from test activities should be used to recognize the changing risks so that planning can be adjusted. As risks and changes occur. The plan should be adjusted to reflect the current position or current situation. Test Planning Activity Test managers put a lot of effort to create the test plan. They have to go through various activities and communicate with different stakeholders to book the plan.

Test planning activities may include the following and some of these may be documented in the test plan identifying and agreeing on the objectives of testing. This will enable us to know if we have reached our objective or not yet. Determine the scope and the risks that need to be tested putting together the overall approach of testing, ensuring that the test levels and entry and exit criteria are defined, coordinating with the project manager and making sure that the testing activities have been included within the software lifecycle activities in the correct sequence and dependency decide what needs to be tested, what roles are involved and who will perform the test activities, deciding how the test results will be evaluated and how test activities will be carried out, and so on.

Scheduling of test analysis, design, implementation, execution and evaluation activities either on birticular dates, for example, in sequential development or in the context of each iteration, for example, in Iterative development. Selecting metrics for measuring and controlling the project budgeting for the test activities determining the level of detail and structure for test documentation, for example, by providing templates or example documents.

The content of test plans vary and can extend beyond the topics identified before. Sample test plans can be found in either Standard I E 29 1193. Other important element of the test plan is defining the entry and exit criteria for testing, which we will talk about in the next video.

6. Entry and Exit Criteria

Anticriteria, which is more typically called Definition of Ready in Agile development, defines the reconditions for undertaking a given test activity. This could include the beginning of a level of testing when test design or when test execution is ready to start. If anti criteria are not met, it’s likely that the activity will move more difficult, more time consuming, more costly and more risky. You can define different entic criteria for different test activities.

Typical entry criteria include availability of testable requirements, user stories and or models. For example, when following a model based testing strategy availability of test items that have met the exit criteria for any barrier test levels, availability of test environment and readiness, availability of necessary test tools, availability of test data and other necessary resources. On the other hand, we have heard the term exit criteria before in the test process.

In the first section, exit criteria are used to determine when a given test activity has been completed or when it should stop. Exit criteria, which is more typically called Definition of Done in Agile Development define what conditions must be achieved in order to declare a test level or a set of tests completed. Typical exit criteria include blend tests have been executed, a defined level of coverage, for example, of requirements, user stories, acceptance criteria, rests, code has been achieved.

The number of unresolved defects is within an agreed limit. The number of estimated remaining defects is sufficiently low. The evaluated levels of reliability, performance, efficiency, usability, security and other relevant quality characteristics are sufficient even without exit criteria being satisfied. It’s also common for test activities to be cut due to the budget being expanded, the scheduled time being completed and or pressure to bring the product to market. It can be acceptable to end the testing under such circumstances if the project stakeholders and business owners have reviewed and accepted the risk to go live without further testing. Entry and exit criteria should be defined for each test level and each test type and will differ based on the test objectives.

Remember that according to the tester bosses that when we reach the exit criteria, the tester should provide a report of his findings to the test manager or the stakeholders to decide if the testing should continue or stub. Let me give you an example to make the overall picture more clear. Imagine you are testing the performance of a specific web page. You want this web page response time to be 6 seconds. Therefore, when a user loads the page, it should be loaded within maximum 6 seconds. To measure the response time accurately, you need to rent a performance tool and you only have a budget of $500. But you must approve the budget for this first.

So the entry criteria to start the testing would be the budget to get approved. The past criteria of the test would be to pass the 6 seconds. The exit criteria would be to spend the whole $500. So as you see from this example that we can reach the exact criteria which is penned the $500. But the item might fail the test because its response time is say 8 seconds, which is greater than the expected 6 seconds. In this case, the tester should write a summary review board to the stakeholders indicating his findings and that the bid response time is only 8 seconds.

The decision makers then should decide if they should continue testing and maybe ask for more money for the testing activity, or if they are happy with the 8 seconds result and will stop the testing at this point.

7. Test Execution Schedule

Once the various test cases and test procedures are produced. With some test procedures potentially automated and assembled into a test suites. The test suites can be arranged in a test execution schedule that defines the order in which they are to be run. The test execution schedule should take into account such factors as paratrization, dependencies, confirmation tests, regression tests and the most efficient sequence for executing the tests. Generally, on your day to day work you should be effective and efficient. For example, if I told you that we have three test cases add record, delete record, modify record, and Brent record, if you decided to try Add, Delete, Add, modify them Brent then this is not efficient. You tested the Add record twice for no reason. Why? If you decided to try Add, Record, Modify, Brent then Delete, then that’s the best solution. Why is the best solution? Because when you’re done running your test suite the database for the records is left as it is. Before you start testing. The records that you needed to add is deleted so the database is clean.

 If you haven’t deleted the record for any reason then your database will get bigger and bigger each time you try to run such test suite. Ideally, test cases would be ordered to run based on their priority levels, usually by executing the test cases with the highest priority first. However, this practice may not work if the test cases have dependencies or if the features being tested have dependencies.

If a test case with a higher priority is dependent on a test case with a lower priority, the lower priority test case must be executed first. As simple as that. In our previous example, we would say that the brand, Delete or Modify record are dependent on the Add record. Expect questions in the exam asking you to order or sequence distances, boating into consideration, biologyty, dependency and maybe other factors.

So always start with the test cases with the highest priority and then put the dependency into consideration. Similarly, if there are dependencies across test cases they must be ordered appropriately regardless of their relative priorities. Confirmation and regression tests must be prioritized I’d as well based on the importance of rabbit feedback on changes. But here again, dependencies may apply. In some cases, various sequences of tests are possible with differing levels of efficiency associated with those sequences. In such cases, trade offs between efficiency of test execution versus adherence to paratrization must be made. Let’s look an example to get a better idea.

The following table shows six test procedures p to you that must now be entered into a test execution schedule. Business severity is regarded as the most important element in determining the sequence of the test procedures, but other dependencies must also be taken into consideration. Regulation testing can be run once all other tests have completed. Which of the following represents the most effective sequence for the test execution schedule? Where the first entry in the sequence is the first procedure to run. The second entry is the second to be run, and so on. And we have four options for answers. If we look at the table, we have three factors here to consider priority dependency and regulation. Regression test cases should be considered lost.

The test cases with the highest priority are Q and R. Q is a regulation only test case, so ignore it for now. R is good, but dependent on S. S is not dependent on anything. So we should consider S first. Only after S, we can consider R. So either answer option B or D is correct. The following priority test case is to consider S and T. We already considered S and T. We already considered S. And T is regression only, so we have nothing to do here.

The following priority to consider p and U. B will be delivered late. Then we should consider U first. So we have SR, u and B so far. So the correct answer is D. The remaining list of cases are Q and T, which admission only tests and should be taken according to their priority. So q then t will be considered. I feel that such questions are more IQ questions, not testing questions. So it’s all up to you here and up to your IQ. Not me.