Skip to content

Tuple Sketches: Deserialization improvement + bug fix - #76

Open
cboumalh wants to merge 1 commit into
apache:mainfrom
cboumalh:cboumalh-tuple-deserialization-fix
Open

cboumalh wants to merge 1 commit into
apache:mainfrom
cboumalh:cboumalh-tuple-deserialization-fix

Conversation

@cboumalh

@cboumalh cboumalh commented Nov 10, 2026 •

Copy link
Copy Markdown

Switches the per-entry deserialization buffer from an allocated nb::bytes (which copied the full remaining serialized blob on every retained entry, producing O(N²) copy work on large sketches) to a zero-copy memoryview over the C++ buffer. Updates the PyObjectSerDe.from_bytes contract from bytes to a buffer-protocol object and adjusts PyStringsSerDe accordingly. Also fixing a 3-byte length-prefix read that corrupted string summaries.

default performance:

Building compact tuple sketch: lg_k=19 (k=524288), n=524288
  build time: 0.27 s, estimate=524288, retained=524288
           serialize: mean=  176.46 ms  stdev=   0.00 ms  min=  176.46 ms  n=1
  serialized size: 6,291,472 bytes
         deserialize: mean=73239.24 ms  stdev=   0.00 ms  min=73239.24 ms  n=1

performance with memory view:

Building compact tuple sketch: lg_k=19 (k=524288), n=524288
  build time: 0.21 s, estimate=524288, retained=524288
           serialize: mean=  147.77 ms  stdev=   0.00 ms  min=  147.77 ms  n=1
  serialized size: 6,291,472 bytes
         deserialize: mean=  150.76 ms  stdev=   0.00 ms  min=  150.76 ms  n=1

bench_tuple_serde.py

@cboumalh

cboumalh commented Nov 10, 2026 •

Copy link
Copy Markdown
Author

Hi @leerho @tisonkun @proost, hope all is well. Please let me know if this is worth pursuing!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant