Repository navigation
Speed up ASCII str.lower() and str.upper() (2 to 4x at 1 KB) #161155
Description
Feature or enhancement
str.lower() and str.upper() on ASCII strings, and lower() and upper() on bytes and bytearray, all go through _Py_bytes_lower() and _Py_bytes_upper() in Objects/bytes_methods.c. These convert one byte at a time through the _Py_ctype_tolower table, which compilers can't vectorize.
Converting a machine word at a time (8 bytes on 64-bit platforms, 4 on 32-bit) with plain integer arithmetic, and keeping the table loop for the last few bytes, gives these pyperf results with gcc 13 -O3 on an x86-64 Linux VM:
| before | after | |
|---|---|---|
s.lower(), 1,024 ASCII chars |
430 ns | 131 ns (3.3x) |
s.upper(), 1,024 ASCII chars |
420 ns | 135 ns (3.1x) |
b.lower(), 1,024 bytes |
402 ns | 134 ns (3.0x) |
s.lower(), 64K ASCII chars |
22.0 µs | 4.7 µs (4.7x) |
s.lower(), 32 ASCII chars |
40.3 ns | 35.7 ns (1.13x) |
s.lower(), 8 ASCII chars |
33.4 ns | no measurable change |
clang 18 gives 2.8 to 3.2x at 1,024 characters. MSVC on Windows x64 doesn't vectorize the new loop and gets about 2.2x, with or without PGO, and a gcc PGO+LTO build gets about 1.9x. 32-bit builds gain 1.2 to 1.3x. Short strings gain little because the method call and the allocation dominate: at 8 characters the results range from about 9% faster to 7% slower, depending on the build. The PR will have the full tables.
The change is about 35 lines in that one file: a small helper with a comment explaining the arithmetic, and two loops, with no intrinsics or #ifdefs. Output is identical for every byte value. I checked it exhaustively at word level on x86-64, 32-bit x86, arm64 and big-endian s390x, with gcc, clang and MSVC.
These loops were last optimized in gh-57947 (2012). I have a PR ready if this sounds acceptable.
Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
No response
Activity
sergey-miryanov commentedon Nov 2, 2026
cc @eendebakpt
eendebakpt commentedon Nov 2, 2026
A PR seems reasonable (but I am not convinced yet). As always we have to balance performance gain to added complexity and possible regressions for important cases (like small strings here). 35 lines extra/diff might be ok. If possible aim for a small diff and easy to review patch.
StanFromIreland commentedon Nov 2, 2026
Please also benchmark without vectorisation.
ptweezy commentedon Nov 2, 2026
Release builds with auto-vectorization disabled in both trees (gcc
-fno-tree-vectorize, clang-fno-vectorize -fno-slp-vectorize). pyperf on GitHub-hosted runners; time onmain/ time with the patch:In the default builds, only gcc
-O3vectorizes the loop. MSVC, Apple clang 17 and gcc PGO+LTO keep it scalar. Windows release builds: 2.10x (x86-64) and 2.19x (arm64) at 1,024.These numbers are for a revised patch. Strings shorter than a word skip the word loop after one length check, and the last word overlaps the previous one instead of falling back to the byte loop. Patch, harness and raw data: https://github.com/ptweezy/cpython-ascii-case-bench