| 
|
Topic: statistical testing based on a usage profile
topics >
computer science >
programming >
Cleanroom software development
Topic:
expert users
Topic:
quality assurance
Topic:
statistics
Topic:
test data selection
Topic:
testing by voting or N-version
Topic:
testing testing
Topic:
user profile
Summary
Instead of random test cases or coverage testing, one can concentrate on test cases that are most likely to occur in practice. This should catch errors that are likely to occur. It is a cornerstone of Cleanroom. Studies have reported excellent improvements.A usage or operational profile identifies operations that are likely to occur for different types of users. The corresponding functions must be identified. White-box testing can also identify important test cases.
Statistical testing does not work if the overall quality of the system is low. If so, infrequent errors will occur frequently just because there are many of them. In any case, all of the code should be exercised by test cases. Testing infrequent combinations and conditions is unlikely to increase the overall reliability of the software. For high-reliability software, every instruction must be tested. (cbb 4/98)
Subtopic: usage based testing
Quote: statistical quality control is useful for software development [»choCK_1987] Quote: for statistical tests of real-time systems need to limit trajectory length by partitioning the state; periodically reinitialize the state [»parnDL6_1990] Quote: Cleanroom uses successive times between execution failure to estimate reliability Quote: Cleanroom's certification team adds an increment to previous increments and measures its MTTF with random test cases [»cobbRH11_1990] Quote: Cleanroom's certification team estimates MTTF for the next version; force a redesign if failures are too frequent or serious
Subtopic: white box testing
Quote: traditional coverage testing finds errors in random order, usage testing finds them in failure-rate order, 20x better for MTTF [»lingRC5_1993] Quote: Cleanroom estimates the user input distribution to generate test cases Quote: need an operational profile to estimate the reliability of software under actual use [»musaJD3_1993] Quote: generate test cases by selecting operations randomly according to the operational profile and input states randomly with its domain [»musaJD3_1993] Quote: statistical testing by selecting tests according to a usage profile [»cobbRH11_1990] Quote: testing via operational profiles identifies failures, on average, according to their frequency; efficient [»musaJD3_1993]
Subtopic: operational profile
Quote: by examining program structure can select statistically significant test cases; not so if treat program as a black box [»huanJC_1977]
Subtopic: functional profile
Quote: develop an expected usage profile with a Markov model on usage states and estimated transition probabilities Quote: an operational profile built from other profiles; i.e., disjoint alternatives with probability of occurrence; use logarithms [»musaJD3_1993] Quote: operational profile by breaking system down into customer groups, user groups, system modes, functions, and operations [»musaJD3_1993] Quote: must limit operation profiles to several hundred elements; several thousand at most [»musaJD3_1993] Quote: use novice and expert system modes to record changes to an operational profile as users become experienced [»musaJD3_1993]
Subtopic: sampled execution
Quote: the final function list of an operation profile is all possible combinations of functions and values of environmental variables Quote: determine the functional profile of a system from its work-flow; gives context for software [»musaJD3_1993] Quote: use environmental variables for conditions that effect the way a program runs; e.g., traffic load; use brainstorming [»musaJD3_1993] Quote: an operation of an operational profile implements a function for a range of input values
Subtopic: partition
Quote: use sampling to identify predicates that are always true for a bug; called deterministic bugs [»liblB6_2003] Quote: sample code execution as a Bernoulli process using pre-generated random countdowns; fast path is a decrement, compare, and branch [»liblB6_2003]
Subtopic: experimental results
Quote: do not use partitioning for test-case selection; use partitioning when narrowly defined with a high probability of error [»hamlD12_1990]
Subtopic: problems with statistical testing
Quote: statistical usage testing is 21 times more effective than coverage testing because most errors are seldom observed [»cobbRH11_1990] Quote: the operational profile and quality-improvement techniques reduced problem reports and maintenance costs ten-fold; test interval doubled [»musaJD3_1993]
Quote: Cleanroom rejects testing's fundamental purpose: to find bugs
[»beizB3_1997]
Topic: Cleanroom software development (38 items)
Topic: expert users (25 items)
Topic: quality assurance (22 items)
Topic: statistics (12 items)
Topic: test data selection (39 items)
Topic: testing by voting or N-version (10 items)
Topic: testing testing (13 items)
Topic: user profile
(16 items)