Topic: experimental results on programming

topics > computer science > Group: programming

testing and evaluating user interfaces

collecting statistics on a UserInterface
examples of usability studies
experimental results on structured programming
learning a programming language
programming language design
programmer productivity
software metrics


A number of experiments have been run on the effect of programming language on programmer productivity. The tendency is to develop a mini-language containing only ore or two features. Experimental evidence is usually on Algol-like languages. Simes et al have set up a very clean experimental format where the subjects "program" the actions of a card eating "hare". Studies seldom compare different language systems due to the overriding effect of problem selection on language performance. (cbb 5/80)
Subtopic: individual differences up

Quote: normalize for developer differences by comparing developers with similar number of changes, whether or not they use a software tool or process; e.g., select from thousands of modification request records
Quote: in all studies of programming, individual differences in performance overshadow computer system differences [»weinGM1_1974]
Quote: software tools and processes often have 10x less influence than project size, developer capability and experience [»gravTL5_2001]
Quote: while a quick reading of Sackman's paper gives 28:1 programmer variability the true figure is more 5:1 [»dickTE7_1981]
Quote: there are order-of-magnitude differences between programmers on programming tasks [»sackH1_1968]
Quote: found a high correlation between student's GPA and the ability to read short (73 line) programs
Quote: investigated documentation formats for programs; individual differences had largest effect, natural language was least effective [»curtB2_1989]
Quote: choice of tester was a larger factor in test effectiveness than was the choice of test technique [»lautL6_1989]
Quote: wide individual differences in designing self-checks from requirements and code [»leveNG4_1990]
Quote: test programmer productivity with the Coding War Games; work in pairs from same specification, exchange programs for testing after desk-check and compilation [»demaT5_1989]

Subtopic: programmer studies up

Quote: with proper precautions, people are willing to be observed and measured; e.g., they forgot about observation sessions [»perrDE7_1994]
Quote: estimate effect of using a tool by recording tool usage, comparing it to change history, analyzing similar developers and modifications with a change effort estimation algorithm [»atkiDL7_2002]

Subtopic: statistical studies up

Quote: test the effectiveness of software engineering tools and practices by modeling an unobserved, independent variable from a sparse nonnegative two-way table [»gravTL5_2001]

Subtopic: code decay up

Quote: statistically significant evidence for code decay; a change requires 20% more effort than a similar change a year before [»gravTL5_2001]

Subtopic: paper solutions up

Quote: use paper solutions to capture mental plans; as transformed into a programming language [»paneJF2_2001]
Quote: studies of paper solutions to programming tasks by non-programming children and adults; used PacMan video game and a database of names and values [»paneJF2_2001]
Quote: studies of paper solutions to database access tasks by non-programmers, both adults and children [»paneJF2_2001]
Quote: 1/2 of children used event-based rules in their paper solution to a programming task; no programming experience [»paneJF2_2001]
Quote: in paper solutions to programming problems, 18% used constraints, 16% used declarations; 12% used imperative programming; children without programming experience
Quote: in paper solutions to programming problems, AND mostly used for boolean conjunction, then sequencing [»paneJF2_2001]
Quote: in paper solutions, 75% used AND incorrectly; e.g., "starts with G and L" instead of "G through L" [»paneJF2_2001]
Quote: in paper solutions, OR used much less frequently than AND; primarily for boolean disjuction, then clarification [»paneJF2_2001]
Quote: in paper solutions, sets or subsets used for 95% of statements on multiple objects; 5% loops or iteration [»paneJF2_2001]
Quote: in paper solutions with looping constructs, 75% had only a terminating condition [»paneJF2_2001]
Quote: in paper solutions with multiple options, 33% with mutually exclusive rules, 25% with general condition modified by exception [»paneJF2_2001]
Quote: in paper solutions that referred to past events, 50% used the present tense to mention a past event, 20% used the word 'after', 10% used a state variable [»paneJF2_2001]
Quote: in paper solutions, 85% treated progress as all or nothing instead of counting [»paneJF2_2001]
Quote: in paper solutions, 66% of participants used pictures or diagrams [»paneJF2_2001]
Quote: if remove queries, aggregates, and visibility from HANDS, only 1 of 9 solved a task vs. 7 of 9 with HANDS [»paneJF9_2002]

Subtopic: time management up

Quote: studied why the elapsed time for software development is more than the actual time [»perrDE7_1994]
Quote: studied time management with time diaries which were validated by direct, detailed observation
Quote: even when coding, half of the time is occupied by noncoding tasks such as support, test, design, and planning [»perrDE7_1994]
Quote: developers worked on a particular project only 40% of the time; the remainder was spent on other projects, waiting for resources, or redoing work [»perrDE7_1994]
Quote: developers spent 75 minutes per day of unplanned interpersonal interaction, usually through email and in-person visits; most interactions were less than five minutes [»perrDE7_1994]
Quote: former colleagues made a surprising percentage of a developer's daily interactions

Subtopic: software reviews up

Quote: the process has little effect on software inspections; most variation due to the reviewers, the code size, and its functionality [»portA11_1997]
Quote: random selection of software review treatments: 1,2, or 4 reviewers with 1 or 2 review sessions or 2 reviews with defect repair; all reviewers prepared ahead of time

Subtopic: micro-language studies up

Quote: to evaluate programming languages, it is better to devise micro-languages which stress the features of interest [»simeME1_1973]
Quote: use micro-languages to determine the effect of language features [»simeME1_1973]
Quote: the programming task consisted of feeding a 'hare' with 'food'; either accepted and 'cooked' or rejected [»simeME1_1973]

Subtopic: conditionals up

QuoteRef: simeME1_1973 ;;Title: Psychological evaluation of two conditional constructions used
Quote: in studies found 'if p..not p' far easier to understand than 'if..then..else'; later never used in natural languages [»greeTR_1980]
Quote: it is better to redundantly restate the else clause; i.e., case statement better than if..then..else [»atkiLV9_1979]

Subtopic: program understanding up

Quote: a measure of program understanding is the number of times spent viewing the program in order to answer a question [»scanDA9_1989]
Quote: a cloze test replaces words in a text with blank lines; subjects comprehend text if they can fill in the blanks [»hallWE5_1986]
Quote: some question about the usefulness of cloze testing for program comprehension [»hallWE5_1986]
Quote: programmers appear to slice a program when debugging; i.e., ignoring statements that do not influence a particular variable [»weisM7_1982]
Quote: programmers remember program by their semantics independently of the program's syntax [»shneB_1985]
Quote: Pascal programmers can easily remember a Pascal program; other programmers can recreate the program in their own language

Subtopic: decision tables up

Quote: users did better with decision trees than with a linear syntax; languages should allow a graphical syntax [»vessI1_1986]
Quote: procedure tables work best for simple problems [»millLA10_1978]
Quote: found procedure tables (transposed decision table) significantly better than IF, BRANCH, and FLOW control [»millLA10_1978]

Subtopic: flow charts up

QuoteRef: ramsHR6_1983 ;;Flowcharts versus program design languages: An experimental
QuoteRef: shneB1_1982 ;;Control flow and data structure documentation: Two
QuoteRef: scanDA9_1989 ;;Structured flowcharts outperform pseudocode: An experimental

Subtopic: design & specification up

Quote: more reliable elevator scheduler from formal methods than informal analysis; style also better; study using 20 teams [»sobeAE3_2002]
Quote: plans for programming are difficult to merge effectively; novices almost always make mistakes [»soloE9_1986]
Quote: use examples to test API design; test usability, documentation, usefulness; API validated by its users [»mcleSG5_1998]
Quote: compared prototyping vs. specifying on a 3000 line program [»boehBW5_1984]

Subtopic: multiple versions up

Quote: compared 7 programming languages with 80 implementations of a phonecode conversion from telephone numbers to words; interpreted, object-oriented, and procedural languages [»precL10_2000]
Quote: built an exact duplicate of the A-7E flight software; hard real-time program to process and display flight data; tight memory and time constraints [»parnDL3_1985]

Subtopic: type checking up

Quote: type checking of function calls reduces duration of interface errors and frequency of interface defects; good experimental design [»precL4_1998]

Subtopic: module size up

Quote: programmers who did not use large modules had 38% fewer defects and were nearly twice as likely to pass the acid test [»demaT5_1989]

Subtopic: change history up

QuoteRef: weisDM2_1985 ;;Evaluating software development by analysis of changes: Some data from the software engineering laboratory.
Quote: could collect and validate data on all changes concurrently with software development [»weisDM2_1985]
Quote: the developer and type of change effected effort, but size of change did not; small changes such as MRs are more uniform in size than releases or features [»atkiDL7_2002]

Subtopic: fault analysis up

Quote: fault analysis of the first 13 releases of an inventory tracking system; 500,000 lines of code, mostly Java [»ostrTJ7_2002]

Subtopic: file system up

Quote: large-scale study of 10,000 Windows file systems at Microsoft [»doucJR5_1999]

Subtopic: Cleanroom up

Quote: Cleanroom teams met requirements better and passed more test cases; all intermediate deliveries were on schedule [»selbRW9_1987]

Subtopic: formating and documentation up

Quote: book format significantly better than normal formatting of program listing for modifying a program [»omanPW5_1990]
Quote: programming improved by indentation, comments, mnemonic names; but not by structured programming [»pariG_1980]
Quote: an English editor did much better than an equivalent, notational editor; demonstrates the importance of an interface's surface syntax [»ledgHF11_1980]
Quote: investigated documentation formats for programs; individual differences had largest effect, natural language was least effective [»curtB2_1989]
Quote: a brief data structure description or half-page diagram was more helpful than detailed pseudocode or detailed flowchart [»shneB1_1982]
Quote: as individuals, made notes to understand and catch errors in Turing's description, then compared and evaluated everyone's notes

Related Topics up

Group: testing and evaluating user interfaces   (9 topics, 262 quotes)

Topic: collecting statistics on a UserInterface (12 items)
Topic: examples of usability studies (31 items)
Topic: experimental results on structured programming (11 items)
Topic: learning a programming language (15 items)
Topic: programming language design (53 items)
Topic: programmer productivity (57 items)
Topic: programmers (15 items)
Topic: software metrics
(32 items)

Updated barberCB 6/05
Copyright © 2002-2008 by C. Bradford Barber. All rights reserved.
Thesa is a trademark of C. Bradford Barber.