Skip to content

Handle line splicing (backslash-newline) in TokenList::Stream #741

Open

Description

@danmar

simplecpp removes a backslash-newline only when it is between tokens. A splice inside a token splits the token instead of joining it.
The standard does line splicing in translation phase 2, before tokenization, so the token must be joined.

Input gcc simplecpp
ab\⏎cd abcd ab cd
12\⏎34 1234 12 34
x +\⏎= 1; x += 1; x + = 1 ;
/\⏎/ comment (comment) / / comment
#def\⏎ine A 1 / A 1 A

Proposed change

Remove backslash-newline in TokenList::Stream::readChar(), using the lookahead buffer from #739, and count the removed lines.
readfile adds the count to the line number at the next newline, as it does now for splices between tokens. The splice code in
readfile, readLineComment() and readErrorText() can then be removed or simplified.

Must stay unchanged

  • Raw token locations. Tokens after a splice keep the line of the start of the logical line (cppcheck uses these locations, for
    example for inline suppressions).
  • Splices between tokens, in string and character literals, in comments, and in #define bodies.
  • Raw string literals: the splice is kept, so R"(a\⏎b)" still contains the backslash and the newline (C++ [lex.pptoken]).
  • \ followed by spaces or tabs and a newline is still a splice, and still gives the "Combination 'backslash space newline' is not
    portable." warning.

Testing

  • Tests for the cases in the table above, with all input types (including FileStream through readfileFromFile()).
  • Compare -l output and raw tokens before and after for real code (/usr/include, cppcheck sources, simplecpp testsuite) and CRLF,
    CR, BOM and UTF-16 copies of them, with all 4 -input types.

Part of #708.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions