TYCHO Search

Search documentation

    ↑ ↓ to move · Enter to open · Esc to close

    3. Lexical structure

    This chapter defines how a Tycho source file is decomposed into a stream of tokens. The token stream — including the synthetic layout tokens NEWLINE, INDENT, and DEDENT — is the input to the phrase grammar (§4).

    3.1 Source text

    A Tycho source file is a sequence of bytes. The lexer is byte-oriented: it recognizes ASCII letters (A–Z, a–z), ASCII digits (0–9), and the punctuation below; all other bytes are meaningful only inside string, f-string, and character literals and in comments, where they are carried as raw bytes.

    There is no byte-order-mark handling and no source-level Unicode escape processing. Identifiers and keywords are ASCII (§3.6). string and bytes *values* are fully byte-safe at run time (interior NUL bytes are preserved), but a string *literal* cannot contain a NUL (§3.9.4).

    An implementation MUST accept files terminated by a final newline and files whose final line has no trailing newline; the lexer flushes any pending DEDENTs and appends EOF at end of input either way.

    A line ends at \n, and a \r immediately before that \n (or before end of input) is part of the ending: a CRLF file lexes exactly as its LF twin does, blank lines included. A \r anywhere else on a line is whitespace. A source file containing a NUL byte MUST be rejected — silently truncating at it would compile a program the file does not contain.

    3.2 Logical lines and NEWLINE

    The lexer is line-oriented. Each non-blank, non-comment-only line produces its tokens followed by exactly one NEWLINE terminal. A blank line (only whitespace) and a comment-only line (only whitespace then a comment) produce no tokens and do not affect indentation (§3.4).

    There is no line-continuation character (no trailing backslash). A logical line may still span several physical lines, by two implicit joins:

    1. Bracket depth. While any ( or [ is open, a physical line boundary produces no NEWLINE and the next line's leading whitespace is not indentation. This has always been the case.
    2. A trailing operator. A physical line whose last token cannot *end* an expression — an arithmetic, comparison, bitwise, or logical operator, ,, =, or := — joins the line that follows, provided that line is indented strictly deeper than the first physical line of the statement. The continuation's leading whitespace is not indentation.

    Only tokens that cannot end an expression qualify, so no program that was accepted before either join can change meaning. : is excluded: it opens a block and must end its line.

    The deeper-indent condition is normative, and it is what preserves the diagnostic for a truncated line. Given

    x := 1 +
print(str(x))

    at equal indent, the + does *not* join, and the program is rejected where the operand is missing rather than being parsed as x := 1 + print(str(x)) and rejected later, on the wrong line, for an unrelated reason.

    Multi-line *constructs* are still expressed through indented blocks (§3.4); these joins are for a single over-long expression.

    3.3 Comments

    Comment ::= "#" (any byte except newline)*

    A comment begins with # and runs to the end of the line. Comments do not nest, and there is no block-comment form. A # inside a string or character literal is an ordinary byte, not a comment.

    3.4 Indentation (INDENT / DEDENT)

    Tycho is indentation-sensitive. Block structure is expressed by leading whitespace, and the lexer emits INDENT and DEDENT tokens to mark it.

    The algorithm, applied per logical line after skipping blank and comment-only lines:

    1. Measure indent depth. The leading whitespace of a line is a run of space (0x20) *or* tab (0x09) bytes. Its depth is the *count* of those bytes — not a display width. A file indented consistently with all spaces and one indented with all tabs nest identically.
    2. Mixing is an error. If a single line's leading whitespace contains *both* a space and a tab, the program is rejected (mixed tabs and spaces in indentation). Indentation MUST use one whitespace character consistently within each line.
    3. Compare to the stack. An indent stack holds the depths of the currently open blocks (initially [0]).
    1. Depth bound. The indent stack has a fixed capacity; nesting deeper than 255 levels is rejected (indentation too deep). This bounds statement nesting and, with the expression-nesting bound (§3.9), guarantees the parser cannot be driven to stack overflow by crafted input — a *fail-closed* requirement (§1.3).
    2. End of input. At end of file the lexer emits a DEDENT for every still- open level, then EOF.

    A block in the phrase grammar is therefore INDENT Stmt+ DEDENT (§4).

    3.5 Tokens

    The token kinds are: EOF, NEWLINE, INDENT, DEDENT; the literal and name tokens IDENT, INT, FLOAT, STR, CHAR; the keyword tokens (§3.6); and the operator and punctuation tokens (§3.8). Whitespace other than the leading indentation and newlines separates tokens but is otherwise insignificant.

    Longest-match applies: the lexer forms the longest valid token at each position (e.g. >> is one shift token, not two >; := is one token, not : then =).

    3.5.1 Case and the two name spaces

    An IDENT may hold any letter case in any position; the token shape does not depend on it. What the case decides is which space the name may declare in:

    The rule is exclusion, not preference: because no run-time binding can be spelled the way a variant is spelled, a binding can never shadow a constructor, and a bare Name in a payload slot is unambiguously a pattern rather than a binding. _ and a leading-underscore name stay legal in every binding position (_ is the discard, §22.4). Procedure names are unaffected — they are neither run-time bindings nor uppercase by convention — except for the four constructor names §3.7 already excludes. The identifier prefix Ztycho_ is reserved in all name spaces for compiler-generated temporaries and is rejected by the lexer.

    3.6 Keywords

    The following words are reserved. A reserved word is never an identifier; using one where a name is expected is a syntax error.

    and     bool    break   bytes   continue elif    else    enum
f32     false   float   fn      for      handle  i8      i16
i32     i64     if      in      inout    int     is      match
not     null    or      or_else  or_return parallel ptr  return
select  spawn   string  struct  true     type    u8      u16
u32     u64

    Those 42 are the whole of keyword(); the same list is tabulated in Appendix B. The words int, bool, string, float, ptr, bytes, f32, and the fixed-width integers u8, u16, u32, u64, i8, i16, i32, i64 are the primitive type keywords; the rest are declaration, control-flow, operator, or literal keywords. or_return and or_else are each matched as a single word (neither is or followed by _return or _else).

    soa is reserved too, but by the parser rather than the lexer: it lexes as an ordinary identifier and every expression position that sees it demands a [, so soa := 5 is accepted and the next mention of the name dies with expected '[' after soa. A binding of that name is declarable and permanently unusable, which is why Appendix B lists it with the reserved words and not with the contextual ones.

    3.7 Contextual identifiers

    Several words are significant only in a specific position and are ordinary identifiers everywhere else. A variable, parameter, or field of the same name is unaffected — subject to §3.5.1, which bars an uppercase spelling from every run-time binding position. They are not reserved:

    A builtin name may not be shadowed. In an unqualified call f(...), if f names a builtin then the builtin is selected, always — whatever else the file declares, and including a call written inside a same-named procedure's own body, which therefore does not recurse.

    That governs the call, and it reaches a declaration only through reachability — so the two declaration positions differ:

    Some implementations currently warn rather than reject some of these declarations in the main program; the rejection above is the normative rule. A warning on a shadowing declaration is quality of implementation, not a conformance requirement; selection of the builtin in an unqualified call is the normative part.

    3.8 Operators and punctuation

    The operator and punctuation tokens, longest-match first:

    SpellingRole
    ... ..<variadic parameter (...T) and spread (x...); half-open counting range (0..<N, §22)
    :=declare-and-infer
    == != <= >=comparison
    ->return-type / function-type arrow
    << >>left / (logical) right shift
    ::reserved token, currently unused by the grammar
    :block colon, typed-declaration colon, map-type / slice colon
    =assignment
    < >comparison
    + - * / %arithmetic
    &bitwise-AND (binary) and address-of / inout argument (unary)
    `\ ^ ~`bitwise OR, XOR, NOT
    ( ) [ ]grouping, calls, tuples; arrays, indexing, slices, maps
    .field / tuple-index access
    , ;separator: argument / element lists; three-clause for-header clauses (§14.4)
    $generic type-parameter sigil ($T) and explicit type arguments (f$(T))

    The byte ! occurs only as part of !=; a bare ! is a lexical error. Boolean negation is the keyword not, not !. The braces { and } are not tokens of the language; they are significant only inside an f-string literal (§3.9.5). The only range operator is ..<; .. alone is not a token, and the range(…) form it replaced is gone (§14.4). Operator precedence and associativity are defined with the expression grammar in §4.5.

    3.9 Literals

    To bound parser recursion, expression nesting (parentheses and unary operator chains) is limited to a fixed depth; a more deeply nested expression is rejected (expression nesting too deep) — a fail-closed guard, the expression-level counterpart to the indentation-depth bound (§3.4).

    3.9.1 Integer literals

    IntLit    ::= DecimalLit | HexLit | BinLit
DecimalLit ::= [0-9]+
HexLit    ::= "0" ("x" | "X") HexDigit+
BinLit    ::= "0" ("b" | "B") BinDigit+
HexDigit  ::= [0-9a-fA-F]
BinDigit  ::= [01]

    An integer literal is decimal, hexadecimal (0x/0X), or binary (0b/0B). The prefixed forms are recognized only when the prefix directly follows the digit 0, so 10x is the integer 10 followed by an identifier — the same tie-break as C and Go — and a prefixed literal is always an integer: there is no hexadecimal-float form (§3.9.2). A 0x or 0b prefix with no following digit is a lexical error, as is a binary literal containing a digit other than 0 or 1. There is no octal form, no digit-group separator (_), and no type suffix (such as u32 or L). An integer literal denotes a value in the range 0 through 9223372036854775807 (2^63 − 1); a literal larger than that is rejected (integer literal out of range).

    Because a literal is non-negative and negation is a separate unary operator (§4), the most negative int value, −9223372036854775808, has no literal spelling; it is obtained by computation (for example negating a smaller value, relying on defined wraparound — §30). An integer literal adapts to a float, u32, u64, or f32 context by the literal-adaptation rules of the type system (§8); it does not change the literal's syntax.

    3.9.2 Float literals

    FloatLit ::= [0-9]+ "." [0-9]+ Exp?
           | [0-9]+ Exp
           | "." [0-9]+ Exp?          /* leading-dot form, position-restricted */
Exp      ::= ("e" | "E") ("+" | "-")? [0-9]+

    A float literal has a fractional part (3.14), an exponent (1e10, 2E8), or both (1.5e-3), or begins with a dot (.5). The value is that of the C strtod parse of the same text and denotes an IEEE-754 binary64 (float); a float literal adapts to f32 by literal adaptation (§8). There is no hexadecimal-float form.

    Two disambiguation rules are normative:

    3.9.3 Character literals

    CharLit ::= "'" ( CharEscape | (any byte except "'", "\", newline) ) "'"
CharEscape ::= "\" ( "n" | "t" | "r" | "0" | "\" | "'" ) | "\x" HexDigit HexDigit
HexDigit ::= "0".."9" | "a".."f" | "A".."F"

    A character literal is delimited by single quotes and denotes exactly one byte (a value 0–255) of type char. The supported escapes are \n, \t, \r, \0, \\, \', and \xNN. An empty literal (''), an unterminated literal, or one holding more than one byte is a lexical error.

    \xNN takes exactly two hex digits, in either case: '\x41' is 'A' and '\x0a' is '\n'. The fixed width is normative and is not C's greedy \x; '\x4' is a lexical error rather than a short read. Together with \0 it makes every one of the 256 byte values writable as a literal.

    \xNN is a character-literal escape only — a string literal still rejects it (§3.9.4), and for a reason that is about representation rather than about hex. A character literal is decoded to its byte at lex time and never reaches a C string literal, whereas a string literal's text is carried as *escaped source* into codegen, where the adjacent-piece join is sound only while every escape is exactly two characters. \0 has the same asymmetry, and for the same reason.

    3.9.4 String literals

    StrLit ::= StrPiece StrPiece*
StrPiece ::= QuotedPiece | RawPiece
QuotedPiece ::= '"' StrElem* '"'
StrElem ::= StrEscape | (any byte except '"', "\", newline, and raw control bytes below 0x20 other than tab)
StrEscape ::= "\" ( "n" | "t" | "r" | "\" | '"' )
RawPiece ::= "`" RawElem* "`"
RawElem ::= (any byte except "`" and raw control bytes below 0x20 other than tab and newline)

    A string literal is delimited by double quotes and denotes a string value. One piece is single-line: an embedded newline is an error (unterminated string literal). The supported escapes are exactly \n, \t, \r, \\, and \" — a smaller set than C; any other \-escape is rejected. A raw control byte below 0x20 (except tab) is rejected and MUST be written with an escape. A single string-literal *piece* is limited to a fixed maximum length (string too long beyond it). Consequently a string *literal* cannot contain a NUL byte; a byte-safe string/bytes *value* containing interior NULs is produced at run time (e.g. via bytes), not written as a literal.

    Adjacent pieces join. Two or more string literals written next to one another are one literal, concatenated left to right at parse time: "ab" "cd" is "abcd". Because the implicit line-join inside (…) and […] (§3.2) suppresses the newline, this is how a multi-line string is written:

    _page := ("<!doctype html>\n"
         "<title>hello</title>\n"
         "<p>body</p>\n")

    Apart from a raw piece (below) there is no other multi-line string form, and there is no backslash line-continuation. An f-string never joins (§3.9.5): it is already sugar for a + chain, so f"a" "b" and "a" f"b" are each a syntax error. Joining is defined on the literals' *escaped source text*, which is sound only because every escape is exactly two characters — the reason \0 and \xNN are not in this escape set (a greedy C-style \x would absorb a hex digit across a join, so "\x4" "1" would mean one byte where the author wrote two, and a \0 would truncate the interned literal, whose length comes from strlen). Both are legal in a *character* literal (§3.9.3), which is decoded to a byte at lex time and so has no such exposure; the asymmetry is deliberate and normative.

    Raw pieces. A piece delimited by backticks — … — is a *raw* string piece. No escape is interpreted inside it: a backslash is a backslash, so a\nb is the four characters a, \, n, b and equals "a\\nb". A " needs no escape. An embedded newline is a literal newline byte, so a raw piece is the one literal form that genuinely spans source lines:

    _page := `<!doctype html>
<title>hello</title>`

    Because no escape exists, there is no way to write a backtick inside a raw piece — the first backtick after the opener closes it. A backtick is an ordinary byte inside a *quoted* piece, so a literal that needs one is written by joining: s := a "" b is the three characters ab . A raw piece that is never closed before end of file is rejected (unterminated raw string literal, reported at the *opening* line). Raw control bytes below 0x20 are rejected exactly as in a quoted piece, with tab and newline excepted. The same fixed per-piece length bound applies (string too long).

    A raw piece is a StrPiece, so it joins with adjacent pieces of either kind: raw "and normal" and "normal and " raw are each one literal. This is sound under the escaped-source-text rule above because the scanner re-escapes a raw piece's bytes into the ordinary two-character escapes as it reads them; the token a raw piece produces is an ordinary string token carrying escaped text, and nothing downstream can tell which spelling produced it. A raw piece is never interpolated: there is no f… form, and { inside a raw piece is an ordinary byte.

    Concatenating two string literals with + also folds to one literal in a const (§12.2), so const TERM = "\r\n" + "\r\n" is a single four-byte literal and not a run-time concatenation.

    3.9.5 f-string (interpolated) literals

    An f-string is a string literal prefixed with f. Inside it, a { expr } hole interpolates the value of an expression, and {{ and }} denote literal { and }. Everything outside a hole obeys the string-literal rules of §3.9.4; text inside a hole is Tycho source, re-lexed and parsed as an expression, and nested string literals inside a hole are permitted.

    An f-string is syntactic sugar: at parse time f"a{e}b" becomes the concatenation "a" + str(e) + "b". There is no distinct f-string AST node. Because the desugaring wraps each hole in str(…), a hole expression MUST be of a type accepted by str (the numeric and string scalars); other hole types are rejected with the same diagnostic str gives (§29).

    3.9.6 Boolean and pointer literals

    true and false are the two bool literals (§3.6). null is the literal of the opaque FFI pointer type ptr; it denotes a null pointer and participates only in FFI passing, null-comparison, and is_null (§24). There is no bytes literal — a bytes value is produced by to_bytes from a string or from an [int] of byte values (§8).

    3.10 Non-normative tree-sitter grammar

    The tree-sitter grammar in editors/zed/grammars/tycho/ exists for editor syntax highlighting and is not part of this specification. It is a flat token stream that models no indentation, and it diverges from the language as defined here in at least the following ways (all resolved in favor of this specification):

    1. it lists while as a keyword — there is no while in the language;
    2. it lists char and void as type keywords — neither is spellable as a type;
    3. it treats import, package, and extern as reserved keywords — they are contextual identifiers (§3.7) — and omits the reserved word handle. Its fourth entry, soa, is right: soa is reserved (§3.6);
    4. its number pattern does not model exponents or the leading-dot form (§3.9.2);
    5. it lists {/} as punctuation and omits ... — braces are not tokens and ... is a real operator (§3.8);
    6. its builtin list is partial and includes the removed names map_get/ map_set (which the language rejects at parse time);
    7. it models no INDENT/DEDENT/NEWLINE (§3.4), so it cannot represent block structure.

    A conforming implementation MUST follow §3.1–§3.9; the tree-sitter grammar has no normative force.