Skip to content

Add a lookahead buffer to TokenList::Stream #739

Closed

Description

@danmar

TokenList::Stream turns the input bytes into characters: it decodes UTF-16, and it turns CR and CR LF into LF. Putting a character
back (ungetChar()) is passed down to the stream classes as raw byte unget() calls:

Class unget()
StdIStream istream::unget()
StdCharBufStream --pos
FileStream ungetc(), or fseek() for UTF-16

So only one character can safely be put back, and ungetChar() must undo the UTF-16 decoding byte by byte (C only guarantees one
character of ungetc() pushback).

Step 2 of #708 moves line splicing into the stream. Then readChar() must look past \, any spaces, and an optional CR to see if a
newline follows. If it doesn't, all of those characters must be put back.

Add a small buffer of already decoded characters to Stream:

  • readChar() and peekChar() take characters from the buffer first.
  • ungetChar() puts the last character back into the buffer, instead of calling unget() on the stream classes.
  • More than one character can be put back.

files with CR-only line endings are misread with file input, and a CR at the end of the file is not treated as a newline.

Activity

  1. changed the title [-]Refactoring: add a lookahead buffer to TokenList::Stream[/-] [+]Add a lookahead buffer to TokenList::Stream[/+] on Nov 7, 2026
  2. added a commit that references this issue on Nov 9, 2026
    bccf566
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions