Skip to content

Restructure the preprocessor to follow the standard's translation phases #708

Open
11 of 14 issues completed
Open

Description

@danmar

simplecpp's output is correct for most code, and it passes nearly all the macro examples
from C11 §6.10.3.5 and the C23/C++20 __VA_OPT__ examples. But the code doesn't follow
the translation phases in C §5.1.1.2 and C++ [lex.phases], and that causes bugs.
Comparing with gcc -E shows these differences:

  • Line joining (phase 2) happens after tokenizing, so ab\<newline>cd becomes two tokens.
  • Tokenizing (phase 3) splits the input into single characters and joins them again in
    combineOperators(). Numbers like 0x1e+e and 1.2.3 are split, and <=>, ->*, .*
    and digraphs aren't supported.
  • #line is handled while the file is read. It takes effect inside #if 0 and does not
    expand macros.
  • #elif is evaluated even after an earlier group was taken.
  • #if arithmetic has no unsigned math (-1 > 0u is false) and no short-circuit
    (0 && (1/0) is an error). true is 0 in C++.
  • ## is checked against a hand-written list: valid <<=, -> and && are rejected,
    and invalid .. is accepted.
  • # escapes every backslash (str(\n) gives "\\n") and drops the space a comment
    stands for.
  • Macro expansion approximates the standard rescanning rules with an expanded-macro set
    and special cases, instead of hide sets.

Proposed steps, each one a separate PR:

  1. Small fixes: #line in the directive loop, skip #elif after a taken group, a new
    #if evaluator, fix # escaping.
  2. Do line joining in the character stream, and replace combineOperators() with a
    longest-match tokenizer.
  3. Move cppcheck-specific changes (raw-string conversion, &= split, #/## in plain
    text) to options applied to the output.
  4. Rewrite macro expansion using Prosser's hide-set algorithm.

The full report with test cases is attached (preprocessing-report.md).

You may want to split it into one main issue for the restructure and separate small issues for the step 1 bugs (#line, #elif, #if arithmetic). Each of those can be fixed on its own.

preprocessing-report.md

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions