Repository navigation
gh-155460: Pre-allocate the output buffer in compression.zstd decompression - #155464
Open
ThomasWaldmann wants to merge 3 commits into
Open
ThomasWaldmann wants to merge 3 commits into
ThomasWaldmann wants to merge 3 commits into
Conversation
Contributor
|
Tested on Windows x64 (MSVC 14.41, Release): cherry-picks cleanly onto current main |
…ecompression Frames produced by the one-shot compression APIs record the decompressed size in the frame header. When the input starts at such a frame header, read that size with ZSTD_getFrameContentSize() and allocate the output buffer at its exact size, instead of growing it progressively and shrinking it on finish. An exactly filled buffer is then returned without a copy. This is only a sizing hint: decompression does not rely on it, so a hand-crafted header recording a wrong size produces exactly the same results and exceptions as before. Two guards bound the allocation: the recorded size is trusted only up to 1 GiB, and recorded sizes claiming more than a 32768x expansion of the available input -- more than the format can produce -- are ignored. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
On top of the pre-allocation in the previous commit: - Pre-size only when the first input of a frame holds the whole zstd1 frame, and check the recorded size against that frame's compressed size instead of all of the input, so that neither a frame fed in pieces nor the data following a frame leads to a pre-allocation. The 1 GiB limit stays. - Check the frame's window size against window_log_max before pre-sizing: libzstd's single-pass mode, which a pre-sized output enables, doesn't check it. - zstd.decompress() feeds each frame whole, sized with get_frame_size(), so every frame of multi-frame input is pre-sized, not only the first. Co-authored-by: Maik Schulze <maik.schulze.cv@googlemail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… feeding test_output_allocated_once_each_frame passed even without the whole-frame feeding in zstd.decompress(): its 2 MiB frame compresses to 459 bytes, so it fits whole into the 1 KiB that _decompress_all() feeds a later frame first. Use 2 MiB of zeros followed by 4 KiB of random bytes instead: the frame compresses to more than 1 KiB, and most of its output is decoded from its first KiB, which grows the output buffer unless the frame is fed whole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ThomasWaldmann
force-pushed
the
zstd-decompress-prealloc
branch
from
November 6, 2026 23:27
e051e36 to
d81eed6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
compression.zstddecompression grows its output buffer step by step, even though most frames record their decompressed size in the frame header (zstd.compress()writes it by default).With this PR, when the first input of a frame holds the whole frame and the frame records its content size, the output buffer is allocated once, at that size. libzstd then decodes straight into it in a single pass, instead of through its internal window buffer.
The recorded size is untrusted, so it is used only if:
OUTPUT_PREALLOC_MAX);OUTPUT_MAX_EXPANSION): a block decodes to at most 128 KiB and takes at least 4 bytes, so no valid frame expands more;window_log_max, which libzstd's single-pass mode doesn't check.Otherwise the buffer grows as before. libzstd fails a frame that decodes to another size than recorded, whether the output was pre-allocated or not, and
max_lengthstill limits the allocation.zstd.decompress()feeds each frame of multi-frame input whole (sized withget_frame_size()), so each frame is pre-allocated, not only the first.Throughput at level 3, measured with pyperf (table): on Linux/x86-64 about 2x for 256 KiB to 16 MiB frames and unchanged from 32 MiB up; on macOS/arm64 7-29% higher at all sizes. For 1000 frames of 512 KiB to 8 MiB it is 14% higher on Linux and 18% higher on macOS.
Co-authored-by: Maik Schulze maik.schulze.cv@googlemail.com
🤖 Generated with Claude Code