Repository navigation
Conversation
Skip sequence of ASCII at once rather than one char per loop reusing faster code from unicodeobject. Can't use the checking from stringlib or unicodeobject fully because those assume copying/writing to an output buffer which slows down this case. Line numbers are only needed for the error message, so count them there. The function is exported because test_peg_generator builds the tokenizer sources into an extension module. The check drops from about 0.9% of compile() time to about 0.1% (perf record). CPU time per compile(), PGO+LTO, 20 paired runs on one pinned core: | Workload | main | this | runs faster | |---|---:|---:|---:| | typing.py | 6.72 ms | 1.2% faster | 17/20 | | all top-level Lib/*.py | 241 ms | 0.7% faster | 17/20 | | synthetic CJK literals | 4.32 ms | 1.3% faster | 16/20 | _PyTokenizer_ensure_utf8 instructions during compile(), callgrind: | File | main | this | |---|---:|---:| | typing.py | 1,811,154 | 132,909 | | _pydatetime.py | 1,246,691 | 91,566 | | test_email/test_contentmanager.py | 487,525 | 43,273 | | synthetic Cyrillic literals | 5,825,590 | 3,832,223 | | synthetic CJK literals | 3,450,590 | 2,479,019 | Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bedevere-app
Bot
added
awaiting review
type-feature
Member
|
Mmh. Benchmarks don't seem convincing. 1% faster is... quite small IMO. |
Contributor
Author
|
The diff is small and this is currently fully duplicated code. This to me is more a refactor than perf change, and sets up for "we can make decode 1.5x faster for ASCII" (the next commit). |
Contributor
Author
|
Related to: #153568 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Skip sequence of ASCII at once rather than one char per loop reusing faster code from unicodeobject. Can't use the checking from stringlib or unicodeobject fully because those assume copying/writing to an output buffer which slows down this case. Line numbers are only needed for the error message, so count them there. The function is exported because test_peg_generator builds the tokenizer sources into an extension module.
The check drops from about 0.9% of compile() time to about 0.1% (perf record). CPU time per compile(), PGO+LTO, 20 paired runs on one pinned core:
_PyTokenizer_ensure_utf8 instructions during compile(), callgrind: