Skip to content

Write sparse files without allocating their gaps, so multi-GB file tests pass - #54

Merged
Project516 merged 4 commits into
masterfrom
p516/big-file-memory
Nov 5, 2026
Merged

Project516 merged 4 commits into
masterfrom
p516/big-file-memory

Conversation

@Project516

Copy link
Copy Markdown
Owner

Eleven OpenJDK tests that write multi-GB files now pass instead of running out of memory: FileChannel/Transfer2GPlus, Transfer4GBFile, TransferTo6GBFile, LoopingTruncate, Channels/TransferTo_2GB_transferTo, TransferTo_2GB_transferFrom, Files/SBC, FileInputStream/LargeFileAvailable, ZipFile/CenSizeTooLarge, ZipFile/EndOfCenValidation and zipfs/EndOfCenValidation.

All of them write after a large seek, the way sparse files are made. #47 moved a file to the host once it grew past 4 MiB, but only after the write: a write at 2 GB first allocated the whole 2 GB gap in memory. Fixes:

  • A write that would take a file past 4 MiB switches it to the host first, so a gap is never allocated.
  • Append mode: every write on a descriptor opened for append goes to the end of the file, as O_APPEND does. Positional writes and mapped-buffer write-back keep their position. The transfer tests found this once the allocation was fixed.
  • A directory can be opened read-only as a channel, as on Linux (SBC does this); reading it fails with Is a directory, and a snapshot is refused while one is open.
  • Large files on read-only Node mounts also use host handles.

Tests:

  • test/java/StreamingFile.java adds a write far past the end and appends after a seek, matching HotSpot. test/nio.test.ts covers the append and directory cases.
  • The eleven tests passed on GitHub Actions, peak RSS about 258 to 497 MB, the slowest 105 s, and go into passing.txt with their durations in times.tsv.
  • pnpm typecheck and pnpm test pass.

Noticed, not fixed:

  • An explicit file mapping still takes its whole length from the VM's linear memory, which tops out at 512 MB and does not shrink after unmapping.

@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 5 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 15b8dcd4-80f6-4b71-b8e0-a4b633e24ef5
Commits

Reviewing files that changed from the base of the PR and between 9b4490b and 6a1d88b.

Files ignored due to path filters: 1

  • test/jtreg/times.tsv is excluded by !**/*.tsv
Files selected for processing: 6

  • src/fs.ts
  • src/natives/io.ts
  • src/natives/nio.ts
  • test/java/StreamingFile.java
  • test/jtreg/passing.txt
  • test/nio.test.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@project516-review-bot project516-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR fixes sparse file handling by switching to host-backed storage before allocating large gaps in memory. It adds append-mode semantics (writes always go to end), allows read-only directory channels, and ensures large files on read-only mounts use host handles. Eleven previously failing OpenJDK tests now pass with peak RSS under 500 MB.


review-bot, model nvidia/nemotron-3-ultra-550b-a55b:free, verdict approve

@project516-review-bot project516-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR fixes sparse file handling by switching to host handles before allocating gap memory, adds append-mode semantics matching O_APPEND, and supports read-only directory channels. Eleven previously failing OpenJDK tests now pass. The changes span VFS, native I/O, NIO, and tests.


review-bot, model nvidia/nemotron-3-ultra-550b-a55b:free, verdict approve

Comment thread src/natives/io.ts Resolved

@project516-review-bot project516-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR fixes sparse file handling by switching to host handles before allocating gap memory (preventing OOM on multi-GB writes), adds O_APPEND semantics for append-mode descriptors, allows read-only directory channel opens (failing reads with 'Is a directory'), and makes large files on read-only mounts use host handles. Eleven previously failing OpenJDK tests now pass with peak RSS 258-497 MB.

Not approved: 1 earlier point is still open in the threads.


review-bot, model nvidia/nemotron-3-ultra-550b-a55b:free, verdict comment

@project516-review-bot project516-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR fixes sparse file handling so that writes past 4 MiB switch to host-based storage, preventing memory allocation for gaps. This enables eleven OpenJDK multi-GB file tests to pass. Changes include modifying the write method to stream to host before allocating gaps, adding DirectoryFile class for read-only directory support, updating canOpenFile and openFile for proper writability checks, and adding append mode and mapped buffer write-back support.

Not approved: 1 earlier point is still open in the threads.


review-bot, model nvidia/nemotron-3.5-lightning:free, verdict comment

@project516-review-bot project516-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All points from the last review are settled in the threads.


review-bot, model nvidia/nemotron-3.5-lightning:free, verdict approve

@Project516
Project516 merged commit 2016d13 into master Nov 5, 2026
16 checks passed
@Project516
Project516 deleted the p516/big-file-memory branch November 5, 2026 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant