Topic: automated testing

topics > computer science > programming > Group: testing


automated tests of specifications and designs
data-driven design
debugging by usage rules
debugging techniques
model checker
test hardware
test data selection
test scripts
testing by program mutation
testing by voting or N-version

Subtopic: regression test up

Quote: PL.8 optimizer bugs produce catastrophic results since optimizer applies to all code; regression test of 150 self-checking programs [»auslM6_1982]
Quote: need systematic regression testing on both distributing and receiving sites; can automate acceptance test; more reliable under change [»nowiDA8_1978]

Subtopic: crash diagnosis up

Quote: automatic diagnosis of Windows XP failures; 83% third-party failures, 12% hardware, and 5% Microsoft bugs [»murpB11_2004]

Subtopic: automated test generation up

Quote: generate SQL by random parse trees and delayed decisions; propagate state information downwards and decisions upwards [»slutD8_1998, OK]
Quote: must automate the generation of SQL tests; manually written tests cover a tiny fraction of the SQL input domain [»slutD8_1998, OK]
Quote: Cleanroom estimates the user input distribution to generate test cases
Quote: generate test cases by selecting operations randomly according to the operational profile and input states randomly with its domain [»musaJD3_1993]
Quote: TOPD's checker generates test cases from the model and determines if the procedure's result matches the model's result [»hendP9_1975]

Subtopic: usage rules up

Quote: find bugs by automatic extraction of usage rules; as templates (beliefs) with few contradictions [»englD10_2001]
Quote: use finite state model to check proper usage of uid-setting system calls; build a finite state model of the program; check for privileged regions [»chenH8_2002]

Subtopic: test bus up

Quote: over a century ago, telephone companies added a test bus to its phone switches; the built-in test access allowed automatic, nightly testing of every phone line; problems found and fixed before subscribers noticed [»martRC7_2005]
Quote: a software test bus is an API for unit and acceptance tests, using the same APIs as the user interface [»martRC7_2005]

Subtopic: test case reduction up

Quote: automated testing should include automated test case simplification [»zellA2_2002]
Quote: use Delta Debugging to help identify bugs tripped by a test; e.g., reduced an HTML page to a single line [»zellA11_2001]
Quote: automatic simplification of erroneous SQL; helped debug problems [»slutD8_1998, OK]

Subtopic: database testing up

Quote: use small, canned, in-memory databases for testing; create database just before test execution; faster, known content, repeatable, self-contained [»martRC7_2005]

Subtopic: random input up

Quote: wynot minimized the random input that caused a crashed program; crash identification by stack trace [»zellA2_2002]

Subtopic: exceptions up

Quote: automated test of exception-safety; ThisCanThrow() throws an exception when a global counter becomes zero; rerun with increasing values of the counter; operations must be exception-neutral [»abraD4_1998]

Subtopic: compare multiple implementations up

Quote: automated, stochastic testing of SQL across multiple database managers; randomly generate SQL statements [»slutD8_1998, OK]
Quote: compare SQL queries across multiple vendors; avoid sorting by comparing row counts and column checksums [»slutD8_1998, OK]
Quote: example of comparing random SQL across multiple database systems; 0.07% had possible errors [»slutD8_1998, OK]
Quote: SQL comparison across database systems not effective for testing NULL, strings, and numeric type coercion; needed for portability [»slutD8_1998, OK]

Subtopic: sampling up

Quote: use sampling to identify predicates that are always true for a bug; called deterministic bugs [»liblB6_2003]
Quote: use adaptive profiling to identify memory leaks in long running programs; sample code segments inversely to execution frequency; a leak is a stale object that is not accessed; SWAT has a low false positive rate [»chilTM10_2004]

Subtopic: problems with automated UI testing up

Quote: automatic testing of software through its UI is slow, opaque, and dangerous; even tiny changes can cause may tests to fail or become inoperable

Related Topics up

Group: debugging   (10 topics, 333 quotes)

Topic: automated tests of specifications and designs (12 items)
Topic: data-driven design (41 items)
Topic: debugging by usage rules (41 items)
Topic: debugging techniques (23 items)
Topic: model checker (49 items)
Topic: test hardware (5 items)
Topic: test data selection (39 items)
Topic: test scripts (13 items)
Topic: testing by program mutation (18 items)
Topic: testing by voting or N-version
(10 items)

Updated barberCB 7/05
Copyright © 2002-2008 by C. Bradford Barber. All rights reserved.
Thesa is a trademark of C. Bradford Barber.