| 
|
Topic: token
topics >
computer science >
programming >
lexical analysis
Topic:
spaces in text
Topic:
strings
Topic:
syntax analysis
Summary
Lexical analysis breaks a continuous string of characters into tokens. The lexer is a state machine operating on character classes. Tokens are usually treated as atomic units for syntactic analysis. (cbb 5/80)
Subtopic: components of tokens
QuoteRef: bennRK6_1968 ;;2-2 "For the purpose of word delimitation...; the characters are assigned classes" -- along with operations to manipulate classes. QuoteRef: bennRK6_1968 ;;3-2 lexic rules based on character classes e.g., same class combine, different classes combine if in alternative class of lead class -- this with exception. QuoteRef: bennRK7_1976 ;;34 primitives are word, phrases, infix and prefix operators. QuoteRef: conwME7_1963 ;;398 break text into tokens by character-pair analysis. QuoteRef: cbb_1973 ;;2/3/75 tokens can be uniquely determined within a minimum neighborhood of the token.
Subtopic: boundaries between tokens
QuoteRef: sammJE_1969 ;;424 strings of concatenated constituents e.g., t+h+i+s. or this+is+data+. QuoteRef: cbb_1973 ;;2/3/75 see a string constant as a string of characters not tokens.
QuoteRef: cbb_1973 ;;(lost) continuous to objects is seeing boundaries
Topic: lexical analysis (14 items)
Topic: spaces in text (6 items)
Topic: strings (13 items)
Topic: syntax analysis
(29 items)