Topic: statistical testing based on a usage profile

topics > computer science > programming > Group: testing

Cleanroom software development
expert users
quality assurance
test data selection
testing by voting or N-version
testing testing
user profile


Instead of random test cases or coverage testing, one can concentrate on test cases that are most likely to occur in practice. This should catch errors that are likely to occur. It is a cornerstone of Cleanroom. Studies have reported excellent improvements.

A usage or operational profile identifies operations that are likely to occur for different types of users. The corresponding functions must be identified. White-box testing can also identify important test cases.

Statistical testing does not work if the overall quality of the system is low. If so, infrequent errors will occur frequently just because there are many of them. In any case, all of the code should be exercised by test cases. Testing infrequent combinations and conditions is unlikely to increase the overall reliability of the software. For high-reliability software, every instruction must be tested. (cbb 4/98)

Subtopic: statistical quality control up

Quote: statistical quality control is useful for software development [»choCK_1987]
Quote: for statistical tests of real-time systems need to limit trajectory length by partitioning the state; periodically reinitialize the state [»parnDL6_1990]
Quote: Cleanroom uses successive times between execution failure to estimate reliability
Quote: Cleanroom's certification team adds an increment to previous increments and measures its MTTF with random test cases [»cobbRH11_1990]
Quote: Cleanroom's certification team estimates MTTF for the next version; force a redesign if failures are too frequent or serious

Subtopic: usage based testing up

Quote: traditional coverage testing finds errors in random order, usage testing finds them in failure-rate order, 20x better for MTTF [»lingRC5_1993]
Quote: Cleanroom estimates the user input distribution to generate test cases
Quote: need an operational profile to estimate the reliability of software under actual use [»musaJD3_1993]
Quote: generate test cases by selecting operations randomly according to the operational profile and input states randomly with its domain [»musaJD3_1993]
Quote: statistical testing by selecting tests according to a usage profile [»cobbRH11_1990]
Quote: testing via operational profiles identifies failures, on average, according to their frequency; efficient [»musaJD3_1993]

Subtopic: white box testing up

Quote: by examining program structure can select statistically significant test cases; not so if treat program as a black box [»huanJC_1977]

Subtopic: operational profile up

Quote: develop an expected usage profile with a Markov model on usage states and estimated transition probabilities
Quote: an operational profile built from other profiles; i.e., disjoint alternatives with probability of occurrence; use logarithms [»musaJD3_1993]
Quote: operational profile by breaking system down into customer groups, user groups, system modes, functions, and operations [»musaJD3_1993]
Quote: must limit operation profiles to several hundred elements; several thousand at most [»musaJD3_1993]
Quote: use novice and expert system modes to record changes to an operational profile as users become experienced [»musaJD3_1993]

Subtopic: functional profile up

Quote: the final function list of an operation profile is all possible combinations of functions and values of environmental variables
Quote: determine the functional profile of a system from its work-flow; gives context for software [»musaJD3_1993]
Quote: use environmental variables for conditions that effect the way a program runs; e.g., traffic load; use brainstorming [»musaJD3_1993]
Quote: an operation of an operational profile implements a function for a range of input values

Subtopic: sampled execution up

Quote: use sampling to identify predicates that are always true for a bug; called deterministic bugs [»liblB6_2003]
Quote: sample code execution as a Bernoulli process using pre-generated random countdowns; fast path is a decrement, compare, and branch [»liblB6_2003]

Subtopic: partition up

Quote: do not use partitioning for test-case selection; use partitioning when narrowly defined with a high probability of error [»hamlD12_1990]

Subtopic: experimental results up

Quote: statistical usage testing is 21 times more effective than coverage testing because most errors are seldom observed [»cobbRH11_1990]
Quote: the operational profile and quality-improvement techniques reduced problem reports and maintenance costs ten-fold; test interval doubled [»musaJD3_1993]

Subtopic: problems with statistical testing up

Quote: Cleanroom rejects testing's fundamental purpose: to find bugs

Related Topics up

Topic: Cleanroom software development (38 items)
Topic: expert users (25 items)
Topic: quality assurance (22 items)
Topic: statistics (12 items)
Topic: test data selection (39 items)
Topic: testing by voting or N-version (10 items)
Topic: testing testing (13 items)
Topic: user profile
(16 items)

Updated barberCB 2/05
Copyright © 2002-2008 by C. Bradford Barber. All rights reserved.
Thesa is a trademark of C. Bradford Barber.