| # The Token stream |
| |
| The token stream is a doubly linked list with special 'EOF' tokens at the |
| beginning and end. The EOF token at the end points to itself as the `next` |
| token, and the one at the beginning points to itself as the `previous` token. |
| So, for a correctly formed token stream (which we routinely assume will always |
| be the case) neither `previous` nor `next` can ever return `null`. We can't |
| express that in the type system because those fields will be `null` temporarily |
| while the list is being built. As a result, the convention throughout the code |
| base, is to suffix those two getters with a bang (`!`) everywhere they're |
| invoked. |
| |
| Tokens are typically accessed through the [AST](ast.md). |
