Skip to content

KLL: allocate heap item storage on first use; add ensureCapacity() - #800

Open
leerho wants to merge 1 commit into
mainfrom
kll-lazy-allocation
Open

leerho wants to merge 1 commit into
mainfrom
kll-lazy-allocation

Conversation

@leerho

@leerho leerho commented Nov 12, 2026

Copy link
Copy Markdown
Member

This applies the lazy allocation convention agreed on dev@ (vote, proposal) to the heap KLL sketches. It is the Java counterpart of apache/datasketches-cpp#558. Separate PRs will follow for t-digest and REQ.

Behavior

  • When storage is allocated: heap KllDoublesSketch, KllFloatsSketch, KllLongsSketch and KllItemsSketch allocate their item storage on first use. That's the first update, or the first merge of a non-empty sketch.
    • Creating a sketch doesn't allocate it.
    • Heapifying an empty image doesn't allocate it.
    • reset() releases it.
    • A NaN or null update is still ignored, so it doesn't count as a first use.
  • Order: the first-use allocation happens before anything else changes, min and max included. If it fails, the sketch is left as it was.
  • KllSketch.ensureCapacity(): a new public method that allocates the initial storage of k items up front. It does nothing once the storage exists, and nothing for sketches backed by a MemorySegment, which are unchanged.
  • Unchanged: results and the serialized form, including the updatable form of an empty sketch.
  • Docs: the KllSketch class javadoc says that an OutOfMemoryError from the first allocation now comes from the first update or merge, not from sketch creation. The newHeapInstance factories point to ensureCapacity().

Tests

The new KllLazyAllocationTest covers the doubles, floats, longs and items sketches:

  • An empty sketch holds no item storage, and none of these allocate any:
    • an ignored (NaN or null) update;
    • iteration and toString;
    • compact or updatable serialization;
    • heapifying its image;
    • merging an empty sketch into it.
  • A sketch with ensureCapacity() and a sketch without it produce byte-identical compact and updatable images. These comparisons stay below k items, because KLL compaction is random.
  • After ensureCapacity(), the first k updates allocate nothing more. A second call changes nothing.
  • Each first use allocates and gives correct results: a single update, a weighted update (both paths), a vector update containing a NaN, merging a large sketch, and merging a single-item image.
  • reset() releases the storage, and the sketch then matches a fresh one.
  • Direct sketches are unaffected.

The full suite passes (2,392 tests). javac -Xlint:all -Xdoclint:all,-missing reports nothing in the changed files.

🤖 Generated with Claude Code

Heap KLL sketches (doubles, floats, longs and items) no longer allocate
item storage when they are created, when an empty image is heapified,
or on reset(). The storage is allocated on the first update, or on the
first merge of a non-empty sketch, before anything else changes.
ensureCapacity() allocates it up front. It has no effect once the
storage exists, nor for sketches backed by a MemorySegment.

This follows the lazy allocation convention agreed on dev@. Results and
the serialized form are unchanged. An OutOfMemoryError from the first
allocation now comes from the first update or merge instead of the
constructor; the class javadoc says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant