 |
 |
|
 |
 |
5
are stored somewhere and accessible, it will be easy to decide which approaches should be used
during each testing activity based on previous experiences. Features to be tested are also
determined during test plan generation. Using history of previously tested applications of similar
type can also reduce effort needed for this activity. Schedule specifies the amount of time and
effort to be spent on different activities of testing. For novice testers guessing it correctly is
difficult. By using experience of experienced testers they can guess it somewhat correctly.
Personnel allocation activity identifies the persons responsible for performing the different
activities. By reusing previous test plan for similar projects, the experts of different testing
activities can be determined easily and reduce the risks of not performing testing well. The idea
is, if test plan generation information of similar type of software, i.e. for applications belonging to
same domain, is stored in library then during testing of any software belonging to that domain,
testers can make use of this information and much of the time can be saved. Further, it would
lead to better testing. As design patterns are developed and stored for design reuse, test plan
generation patterns can be generated and stored that would have several decisions related to
test plan generation and thus can be used millions of time without doing it again and again.
Further, test plan templates should be developed for different domains that would help in
developing test plan during testing of applications belonging to that domain. Templates for test
plan generation may guide the novice testers so that they would feel confident that they are doing
correct things and not leaving something necessary, by mistake.
2.2 REUSE IN TEST CASE GENERATION
Test case generation activity starts with deciding the testing criterion. According to that test
criterion test cases are developed. It is difficult to decide whether a set of test cases satisfy a
criterion or not. For structural testing, several tools are used for this purpose that provides
feedback regarding what needs to be tested to fully satisfy the criterion. Test case generation
cannot be fully automated. It is an NP hard problem. Thus to reuse test cases the only and best
method is to use patterns for test case generation. These test case patterns describe for what
type of functionality this would help to generate test cases. In this way, time involved in test case
generation (the most difficult activity in testing process) can be reduced to some extent. Generally
testers select test cases in an iterative manner starting with an initial test case set and selecting
more test cases based on experience during execution of those test cases [2]. This matured set
of test cases should be stored in the library for public access so that testers testing similar
applications can get idea from this set and can design test cases easily, in less time. Regression
testing is one of the examples of testing where teVWFDVHVDUHUHXVHG³DVLV´,WLVWKHVHOHFWLYH
retesting of a software system that has been modified to ensure that any bugs have been fixed
and that no other previously working functions have failed as a result of the reparations and that
newly added features have not created problems with previous versions of the software. We can
say, it a reuse in project level. For applying reuse at domain level test cases should be developed
for a domain so that one can reuse those test cases in the testing of the similar type of
applications belonging to that domain. Although, test FDVHV³DVLV´UHXVHLVSRVVLEOHRQO\HLWKHULQ
regression testing or when any unit is being used in some system development. For the later
case, all the test cases would be used to see whether the integration of this unit with others make
a negative effect on this unit. That is, with other units this unit is not performing as for what it was
designed.
Test cases can be described in UML. UML has been standardized by OMG (Object Management
Group) that is an industry-standard analysis and design notation. One should use it in creating
test cases so that one can easily reuse them due to standardization. Further, generation of test
cases for software design using UML would be easy. It may be easier to test some features,
without executing the software, by seeing UML design of software if UML is used for this purpose.
Further, its automation may also be possible.
2.3 REUSE IN TEST CASE EXECUTION AND ANALYSIS
With the specification of test cases, the next step in testing process is to execute them. Executing
the test cases may require construction of deriver modules or stubs. A stub is a dummy routine
that simulates a module. It is used in top down testing strategy where testing is started by testing
6
the top of the hierarchy, and modules are added incrementally that it calls and then test the new
combined system. Drivers are needed to perform bottom up testing that starts from the bottom of
the hierarchy. Thus derivers are needed to set up the appropriate environment and invoke the
module, which has no subordinate. These stubs and derivers can also be developed already by
identifying them by analyzing the domain and can be reused. For analysis of the execution results
data is to be collected. Sometimes, test procedure specification document is used to execute the
test cases [2]. This document specifies any special requirements that exist for setting the test
environment, the methods and formats for reporting the results of testing and measurements, if
needed, along with how to obtain them. Here is again the possibility of reuse of this specification
document among applications belonging to the same domain. Data collection forms and software
can be developed in a particular domain and reused. The process of test case execution involves
instrumentation of probes in the source code, executing the source code and analyzing the probe
values. Probes are instrumented in the source code to compute the coverage of testing,
execution results that are produced in intermediate stages etc. The most common method of
instrumentation is to insert some statements, along with probe variables, called probes in the
program. Probe insertion can be done automatically by a preprocessor. The analysis of the probe
values can also be done automatically by a postprocessor. These preprocessor and
postprocessor can be reused if they are implemented as generic components and can be
customized during use.
Different types of preprocessors/postprocessors can be constructed based on different kinds of
probes: probes that are inserted where a path splits into multiple paths and where multiple paths
merge or a different probe for each path. For a domain different preprocessors and
postprocessors should be generated in advance to reduce testing cost. According to need, users
may select the one among all. Test oracles are needed to test any program. A test oracle is a
mechanism that can be used to check the correctness of the output of the program for the test
cases. The output of the two is compared to determine if the program behaved correctly for the
test cases. Test oracles should be automated to give always a correct answer. But it is very
difficult to automate test oracles. Although, for some systems there exists automated oracles but
since automation of oracles are done based on requirements specification it is not always
possible to obtain an error-free oracle and hence automation of this activity cannot be taken up.
Since there is a possibility of errors in specifications thus oracle would too produce wrong output.
6\VWHPVIRUZKLFKRUDFOHVFDQQRWEH³DXWRPDWHG´, oracles created during testing of previous
systems may be used for generating oracles for systems of similar types. Test oracles should be
developed as components and put in repositories. ThHVHWHVWRUDFOHVFDQEHUHXVHGHLWKHU³DVLV´
or after customization, if needed.
5(86(,1³(5525&$86(6´'(7(&7,21$1'7+(,55(029$/
During execution errors are detected. The next step is to locate the cause of errors and correct
them. Here also, we have a possibility of reuse of the test summary report generated for a
domain during testing of the application belonging to that domain. Test summary report stores the
summary of the entire test case execution. The summary gives the total number of test cases
executed, the number and nature of errors found, and a summary of any metrics data (e.g. effort)
collected. This report can be reused either at project level or within domain. At the project level it
is used to track the status of defects found during testing while, across the projects within a
domain, it can be used to concentrate on errors specified in the report already.
2.5 REUSE IN EVALUATION/REVIEW OF TESTING PROCESS
The last step in testing process is to evaluate/review the testing carried out to determine its
quality. The evaluation of testing process is necessary to see whether it has been carried out
satisfactorily i.e. carried out thoroughly or not. As with many other verification methods,
evaluation of quality of testing is done through ³WHVWUHYLHZ´$QGIRUDQ\UHYLHZDIRUPDO
document or work product is needed [2]. All the test deliverables are reviewed, using a formal
review process, to make sure that the testing has been carried out with the policy specified in the
test plan, test cases specification is generated for regression testing etc. In this activity, the formal
review process is being reused. If the review will be done in an ad hoc way, that is, if there will not
7
be a formal (repeatable) process, testing evaluation may not be done successfully because
testers would always try to show that testing is carried out successfully. The
quantitative(s)/quantitative(s) that can be used to measure the quality of the scripts that are
developed by Engineers can also be reused in a particular domain. The traditional
qualitative(s)/quantitative(s) would be:
Religious practice of coding guidelines,
Duration taken for a test run,
Quality of documentation (Test script comments, Test script description document),
Classifying the Review Comments in to different levels (e.g. minor, major, trivial etc.) based on
the impact of the same on the script execution,
And number of times the script is run on the test harness before the same is run successfully etc.
Thus we have seen that there is much possibility of reuse in testing process. For each activity
carried out during testing, test design patterns can be created to store the decisions that should
be taken during different situations. Test environment can be developed to aid testers. The test
design patterns schema is described as follows [8]:
The Test Design Pattern Schema
Name
A word, or phrase, that identifies the pattern and suggests its general approach.
Intent
What kind of test suite does this pattern produce? This is a succinct statement, no more than two
sentences.
Context
What test design problem does this pattern solve? When does this pattern apply?
Fault Model
What kind of faults does this pattern target and why will it hit them? This section must explain how
the test suite will reach the targeted faults, how it can cause a failure to be triggered, and how a
failure will be propagated to become observable.
Strategy
How the test suite is designed and implemented? A test strategy must address four issues.
Test Model section defines a representation for the responsibilities and/or implementation
that are the focus of test design.
Test Procedure section defines an algorithm, technique, or heuristic by which test cases
are produced from the model.
Oracle section defines the algorithm, technique, or heuristic by which actual results to a
test case are to be evaluated for pass/no pass.
Automationsection discusses automated approaches to test suite generation, test run
execution, and test run evaluation. The strategy is typically presented by example.
Entry Criteria
What are the preconditions for design and/or running a test using this pattern?
Exit Criteria
:KDWFRQGLWLRQVPXVWDWHVWUXQDFKLHYHGWRPHWWKLVSDWWHUQ¶VWHVWJRDOV"
Consequences
What are the disadvantages and advantages of using this pattern? The costs, benefits, risks, and
general considerations for using this pattern are discussed.
Known Uses
What are the known uses of this test design pattern? What are the efficiency and effectiveness of
this pattern or similar strategies, as established by empirical studies?
Related Patterns
What other test design patterns are similar or complementary?
3 TEST ARTIFACTS AND TEST REUSE REPOSITORY
We have seen there are different artifacts that can be reused during testing. Two basic types of
test knowledge are recognized: generic test knowledge, which applies to all projects, and project-
8
specific test knowledge. The arrows in the figure 2 represent the possible kinds of test knowledge
that may be placed in the test repository and have been already explained above.
These artifacts must be stored in some reuse library in such a way that they can be accessible to
users easily. Artifacts in test reuse library should be categorized in such a way that they can be
accessed easily. Categorization of different testing artifacts is explained in figure 3. Some
artifacts can be categorized based on domain while others may not. It is shown in column one.
Other criteria for classifications are shown in column second. Second column for some artifacts is
empty because they cannot be categorized except based on domain.
Figure 3: Kind of test artifacts
Testing artifacts Categorization Examples
Preprocessors/ According to the type 1. A different probe is inserted for each path.
postprocessors of probes
(According to domain)
2. Inset a probe where a path splits into multiple
paths and where multiple paths merge.
3. Inset a probe after each line of code etc.
Test cases
(According to domain) testing
1. According to level of 1. Unit testing, integration testing, system testing
$FFRUGLQJWR³DVLV´design patterns.
reusable versus
customizable:
2. Test cases (project level reuse), templates, test
Testing tools 1. According to 1. Test case evaluation tool, probes
activities
2. According to 2. Tools for data flow-based testing, mutation
particular testing
instrumentation tool etc.
testing etc.
Test environments Environment for unit testing, environment for test
(According to domain)
plan generation etc.
Test oracles
(According to domain) reusable versus test oracle templates
$FFRUGLQJWR³DVLV´Automated test oracles (test oracle components),
customizable:
Testing metrics According to activities Defect removal efficiency, testability etc.
Test reositor
Test plan generation
temlates
Project specific
information
Test design
atterns
Test
environments
Test cases templates
for different testin
Components for
Test oracles
Testing reviews/
Evaluation
experience
Preprocessors/
Postprocessors
component
Figure 2: The test repository
Test metrics
Testing tools
Testing
deliverables
 |
|
 |
|