Module 06.1: From Source Text to Structure

So far, most of our little languages have started from abstract syntax.

For example, we might write an arithmetic expression directly as Haskell data:

Add (Num 1) (Mul (Num 2) (Num 3))

and then write an evaluator or compiler that operates on that data.

Real programmers do not usually want to write programs this way.

They want to write:

1 + 2 * 3

or, in piq:

square size
move right size + gap

So there is a missing step in the language pipeline:

source text  --->  abstract syntax  --->  meaning
                 parser              evaluator/compiler

This module is about that first arrow.

Parsing turns concrete syntax into abstract syntax.

The goal here is not to study every major parsing algorithm. Parsing is a large subject, and there are many ways to implement parsers.

For CS 131, we are going to focus on the concepts and vocabulary you need to understand what a parser is doing, then use a provided parser-combinator library in Module 06.2 to build one.

By the end of this module, you should be able to:

• distinguish concrete syntax from abstract syntax,
• explain what a parser does,
• distinguish a concrete syntax tree from an abstract syntax tree (AST),
• read a small context-free grammar,
• identify terminals, nonterminals, alternatives, and repetition in a grammar,
• explain the difference between identifiers and reserved words,
• recognize common kinds of tokens,
• explain why parsing is about recovering structure rather than evaluating a program, and
• read a grammar rule as a specification for the structure a parser should recognize.

How to use this module

The questions labeled Gradescope question are the completion questions for this redesigned version of Module 06.1.

Other boxes labeled Pause and think, Try it, or Practice are optional checks for understanding.

Background: CS81's notes on grammars and parsing

If you've taken CS81, this module and the next one revisit ideas from its computability unit, with a practical, Haskell-flavored spin. Two pages there are especially good background/refresher material:

• 6.11: Context-Free Grammars — the formal definition of a CFG (terminals, nonterminals, a start symbol, replacement rules), derivations, and parse trees.
• 6.12: Parsing — ambiguous grammars, and how to encode precedence and associativity directly into a grammar's structure.

Nothing here requires having taken CS81 — this module is self-contained — but if any of this feels familiar, that's why, and those pages are a good place to go deeper.

---

The Missing Arrow

Recall the programming-language pipeline:

concrete syntax       abstract syntax          semantics
---------------       ---------------          ---------

  1 + 2 * 3     --->       Add           --->       7
                           /   \
                        Num 1   Mul
                               /   \
                            Num 2   Num 3

We already know a lot about the right-hand side.

In earlier modules, we:

• designed Haskell datatypes to represent abstract syntax,
• wrote evaluators that gave expressions meaning,
• represented functions as data,
• and built environments and closures to evaluate expressions involving variables and functions.

Now the focus moves one step earlier:

concrete syntax  --->  abstract syntax
                parser

---

Concrete Syntax

Concrete syntax is the actual text a programmer writes.

For arithmetic:

1 + 2 * 3

For piq:

move right size + gap

Concrete syntax contains details that make programs readable and writable as text:

• keywords such as move,
• names such as size,
• punctuation such as (, ), {, }, and ,,
• operators such as + and *,
• numeric literals,
• whitespace.

Ultimately, the computer receives a sequence of characters.

For example:

move right 12

is, at the lowest level, just characters:

'm' 'o' 'v' 'e' ' ' 'r' 'i' 'g' 'h' 't' ' ' '1' '2'

But that is not how a programmer thinks about it.

We immediately see larger meaningful pieces:

move    right    12

and, more importantly, we see a structure:

move [direction] [distance]

Parsing is about recovering that structure.

---

Abstract Syntax

Abstract syntax keeps the parts of the program that matter structurally while discarding many details of how the programmer wrote them.

For example, suppose our arithmetic AST is:

data Expr
  = Num Double
  | Add Expr Expr
  | Mul Expr Expr

Then:

1 + 2 * 3

might become:

Add
  (Num 1)
  (Mul (Num 2) (Num 3))

The AST makes the grouping explicit.

It no longer needs to remember that:

• the programmer used spaces,
• + was literally the character +,
• * was literally the character *,
• multiplication was written after the 2,
• the original program was a string.

Instead, the AST stores the structure we care about:

        Add
       /   \
    Num 1   Mul
           /   \
        Num 2  Num 3

That is exactly the kind of data our evaluator or compiler already knows how to consume.

Gradescope question: What information should disappear?

Submit this response on Gradescope.

Suppose these two pieces of concrete syntax mean the same thing:

1 + 2

( 1 + 2 )

If both are parsed into the same AST, what information from the concrete syntax has been discarded? Why might that be a good thing?

---

Concrete Syntax Trees and Abstract Syntax Trees

There are two related kinds of trees worth distinguishing.

A concrete syntax tree

A concrete syntax tree (CST), sometimes called a parse tree, follows the grammar very closely.

Suppose we have a grammar like:

Expr   ::= Number "+" Number
Number ::= Digit+
Digit  ::= "0" | "1" | "2" | ... | "9"

Then a concrete syntax tree for:

12 + 3

might look conceptually like:

Expr
├── Number
│   ├── Digit "1"
│   └── Digit "2"
├── "+"
└── Number
    └── Digit "3"

The tree exposes the grammatical structure used to recognize the text.

An abstract syntax tree

The AST throws away grammar scaffolding that the rest of the program does not need:

    Add
   /   \
Num 12 Num 3

or, as Haskell data:

Add (Num 12) (Num 3)

The concrete syntax tree answers:

How does this text fit the grammar?

The abstract syntax tree answers:

What structured program did the programmer write?

For this course, the final product we care about is the AST.

We will usually skip building an explicit CST

A parser does not necessarily need to construct a concrete syntax tree as an intermediate data structure.

Our parser combinators will typically recognize the concrete structure and construct the AST directly.

The CST is still a useful conceptual picture because it helps us see the relationship between the grammar and the source text.

---

Parsing Is Not Evaluation

Consider:

2 + 3 * 4

Parsing should produce something like:

Add
  (Num 2)
  (Mul (Num 3) (Num 4))

It should not produce:

Num 14

Why?

Because parsing answers a structural question:

What program did the text describe?

Evaluation answers a semantic question:

What does that program compute?

Keeping those stages separate is powerful.

The same AST can later be:

• evaluated,
• compiled,
• pretty-printed,
• optimized,
• analyzed,
• transformed.

For piq, the parser will recover the AST, and the compiler you already have will turn that AST into Python drawing commands.

The parser does not need to know how to draw anything.

---

What Does a Parser Do?

For our purposes, a parser has two jobs:

1. Reject invalid concrete syntax.
2. For valid concrete syntax, construct the corresponding abstract syntax.

For example:

square 20

might parse successfully as:

Square (Num 20)

while:

square +

should fail because the text does not have the required structure.

A useful mental model is:

source text
   |
   v
 parser
   |
   +---- syntax error
   |
   `---- AST

---

Grammars Describe Valid Structure

To build a parser, we need a precise description of what counts as valid syntax.

A common tool is a context-free grammar.

Here is a small grammar for arithmetic expressions:

expr   ::= term   (("+" | "-") term)*
term   ::= factor (("*" | "/") factor)*
factor ::= number | name | "(" expr ")"

You do not need to learn formal-language theory in depth here.

You do need to be able to read rules like these.

More on context-free grammars

For the formal definition of a CFG and more worked examples of derivations and parse trees, see CS81's 6.11: Context-Free Grammars.

Nonterminals

Names such as:

expr
term
factor
number
name

are nonterminals.

They name categories of syntactic structure.

Terminals

Concrete pieces of syntax such as:

"+"
"-"
"*"
"/"
"("
")"

are terminals.

These are pieces of actual source text that must appear in the input.

::=

Read:

factor ::= ...

as:

a factor may have the following form.

Alternatives

The vertical bar means or:

factor ::= number | name | "(" expr ")"

means that a factor can be:

• a number,
• or a name,
• or an expression surrounded by parentheses.

Repetition

The * after a parenthesized grammar fragment means zero or more repetitions.

So:

term ::= factor (("*" | "/") factor)*

means:

1. first parse one factor,
2. then parse zero or more occurrences of either * or / followed by another factor.

This can describe:

4

or:

4 * x

or:

4 * x / 2 * y

---

Why Does the Grammar Have expr, term, and factor?

Consider:

x + 5 * 2

We want this to mean:

x + (5 * 2)

not:

(x + 5) * 2

The grammar encodes that distinction by using levels:

expr
  made from terms joined by + or -

term
  made from factors joined by * or /

factor
  number, name, or parenthesized expression

So:

5 * 2

forms one term before it participates in the surrounding expr.

The corresponding AST is:

Add
  (Var "x")
  (Mul (Num 5) (Num 2))

That is operator precedence.

Similarly:

10 - 3 - 2

normally means:

(10 - 3) - 2

rather than:

10 - (3 - 2)

That grouping choice is associativity.

For the parsing assignment, you need to understand what precedence and associativity mean and what AST should result.

You do not need a survey of parsing algorithms for implementing them.

More on precedence, associativity, and ambiguity

This expr/term/factor layering is exactly the technique CS81 calls precedence encoding — see 6.12: Parsing for more on why an ambiguous grammar is a problem, and how stratifying a grammar by precedence level (and choosing left- vs. right-associative rules) fixes it.

Gradescope question: Grammar to structure

Submit this response on Gradescope.

Consider:

expr   ::= term   (("+" | "-") term)*
term   ::= factor (("*" | "/") factor)*
factor ::= number | name | "(" expr ")"

Explain why:

x + 5 * 2

should produce:

Add (Var "x") (Mul (Num 5) (Num 2))

rather than:

Mul (Add (Var "x") (Num 5)) (Num 2)

---

From Characters to Useful Pieces

A source file is a sequence of characters, but parsers often reason in terms of larger pieces such as:

square
size2
42
2.5
+
(
)
,

These pieces are often called tokens.

A token is an atomic piece of concrete syntax that has meaning to the language.

Common categories include:

• identifiers, such as size or box2,
• keywords or reserved words, such as for, from, to, or define,
• numeric literals, such as 17 or 2.5,
• operators, such as + or *,
• punctuation, such as (, ), {, }, and ,.

Whitespace usually helps separate pieces but is not itself part of the AST.

Our supplied parser library will handle most of this low-level work for us.

We are not going to build a separate tokenizer from scratch.

---

Small Examples of Recognition

Before thinking about whole programs, consider a few tiny parsing tasks.

One digit

Recognize exactly one digit:

7

but reject:

x

A number

Recognize:

42

or:

-3

or:

2.5

and produce a Haskell numeric value.

An identifier

For piq, a name has the form:

letter followed by zero or more letters or digits

Examples:

x
size
size2
box17

but not:

2box

Exact text

Sometimes we need a specific literal sequence of characters:

move

or perhaps, in another language:

pen-up

That is different from "parse an identifier."

An exact-text parser cares about those exact characters.

An identifier parser cares about a lexical category.

This difference becomes especially important for keywords.

---

Keywords Are Not Just Prefixes

Suppose dot is a reserved word.

The name:

dotty

should still be a perfectly ordinary identifier.

A naive parser for the exact characters:

dot

could consume the first three characters of:

dotty

and leave:

ty

behind.

That is not what we mean when we say "dot is a keyword."

We want to recognize dot as a whole word.

A useful strategy is:

1. parse a complete identifier,
2. check whether the result is exactly "dot".

Then:

dot

succeeds as the keyword, while:

dotty

does not.

The reverse issue matters too.

If circle is reserved, then:

circle

should not also be usable as an ordinary variable name.

So the language needs a distinction between:

reserved word

and:

user-defined name

Gradescope question: Keyword versus identifier

Submit this response on Gradescope.

Suppose dot is a reserved word.

Why is matching only the exact character sequence "dot" from the front of the input not enough to implement a correct keyword parser?

Explain what should happen for:

dot
dotty

---

The piq Example

Here is a small piq program:

define box(size, gap) {
    square size
    move right size + gap
}

for i from 1 to 5 {
    box(i * 8, 10)
}

The parser's job is to recover a structured representation such as:

Program
  [ ProcDef "box" ["size", "gap"]
      [ Square (Var "size")
      , Move Rt (Add (Var "size") (Var "gap"))
      ]
  ]
  [ For "i" (Num 1) (Num 5)
      [ Call "box"
          [ Mul (Var "i") (Num 8)
          , Num 10
          ]
      ]
  ]

Look at some of the structure that the source text implies:

define box(size, gap) { ... }

contains:

• the keyword define,
• a procedure name,
• a parenthesized comma-separated list of parameter names,
• a block of statements.

And:

for i from 1 to 5 { ... }

contains:

• the keyword for,
• a loop-variable name,
• the keyword from,
• an expression,
• the keyword to,
• another expression,
• a block.

The parser recognizes all of those concrete pieces, but the final AST keeps only the meaningful structure.

For example, the AST does not need a constructor saying:

OpenParenthesis

because the parentheses have already done their job of communicating structure to the parser.

---

Reading a Grammar as a Parser Specification

A useful habit for the assignment is to read each grammar rule and ask:

What smaller pieces would I need to recognize, and what AST value should I build from them?

For example:

factor ::= number | name | "(" expr ")"

suggests:

factor parser:
    try a number
    OR
    try a name
    OR
    try an expression inside parentheses

Similarly:

call ::= name "(" arguments ")"

suggests:

call parser:
    parse a name
    THEN
    parse a parenthesized argument list
    THEN
    build a Call AST node

And:

block ::= "{" statement* "}"

suggests:

block parser:
    parse {
    THEN
    parse zero or more statements
    THEN
    parse }

Module 06.2 gives us Haskell tools that let us write parser definitions in almost exactly this style.

That is the real payoff of parser combinators.

---

What We Are Deliberately Not Doing

Parsing is a large subject.

There are many important techniques and terms that we are intentionally not studying here, including detailed treatments of:

• recursive-descent parsing,
• LL and LR parser families,
• lookahead calculations,
• shift-reduce parsing,
• parse tables,
• FIRST and FOLLOW sets,
• parser generators.

Those topics are valuable in a compilers course.

Our goal is narrower:

understand the structure of the problem well enough to build a useful parser with a parser-combinator library.

---

Put the Ideas Together

The important pipeline is:

source code
    |
    v
concrete syntax
    |
    |  grammar describes valid structure
    |
    v
parser
    |
    v
abstract syntax tree
    |
    v
evaluator / compiler / analysis

The vocabulary to carry forward is:

concrete syntax
abstract syntax
concrete syntax tree / parse tree
abstract syntax tree (AST)
parser
grammar
production / rule
terminal
nonterminal
identifier
keyword / reserved word
token
precedence
associativity

The next module turns these ideas into Haskell code.

Rather than writing a giant parser from scratch, we will start with tiny parsers and combine them.

That gives us parser definitions that can closely follow the grammar itself.

---

Finish Module 06.1

Make sure you have answered the required Gradescope questions embedded above:

1. What information should disappear when concrete syntax becomes abstract syntax?
2. Why does the expression grammar make * bind more tightly than +?
3. Why must a keyword parser distinguish dot from dotty?

Gradescope question: Time spent

Submit this response on Gradescope.

Approximately how much time did you spend on this module?

• less than 30 minutes
• 30 - 45 minutes
• 45 - 60 minutes
• 60 - 75 minutes
• 75 - 90 minutes
• more than 90 minutes

Gradescope question: Lingering questions

Submit this response on Gradescope.

What lingering questions or thoughts do you have about this module?