Repository navigation
Conversation
Corpus 1619 pins WeakMap and WeakSet behavior with thousands of keys; the scaling test bounds how the time of the same work grows from 25,000 to 200,000 keys (vercel-labs#1619). The scaling test fails on main: each lookup scans every entry, and the 200,000-key run is stopped at 60 seconds.
WeakMap and WeakSet found a key by walking every entry of the collection, so filling one with n keys took O(n^2) time: 200,000 keys ran for about a minute where Node takes a tenth of a second (vercel-labs#1619). - Each collection keeps its entries in a hashed index that doubles when its entries outnumber its buckets; get, set, has and delete look up and unlink through it. - The thread-wide index of all entries, which key disposal searches, grows the same way from its static 1,024 buckets, and returns to them when the last entry goes. - Key disposal returns at once when no weak entry exists. - Corpus 1619 covers growth, overwrites, deletes, keys shared by several collections, a dropped collection, dropped keys and fresh keys after them; a harness test bounds how the time grows from 25,000 to 200,000 keys.
Contributor
|
@ejc3 is attempting to deploy a commit to the Vercel Labs Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
Author
|
Main has moved since the Testing section's 1ee5c84. The two commits apply cleanly to 7c235ce, and there the scaling test, |
jlucaso1
pushed a commit
to jlucaso1/scriptc
that referenced
this pull request
Oct 24, 2026
WeakMap and WeakSet kept each collection's entries in one list and found a key by walking it, and every collection's entries for a key in 1,024 shared buckets. Filling a collection and reading it back took quadratic time: 40,000 keys took 6.7 s against Node's 0.2 s, and 200,000 keys did not finish. Each thread now indexes its entries twice, in open-addressing tables with linear probing and backward-shift deletion: by collection, key and kind, for get, set, has and delete, and by key pointer, for a key's disposal, which ends the key's entries in every collection. A collection's entries and a key's entries are doubly linked, so an entry leaves both in constant time. The tables grow at half load and are freed when their last entry goes, so an exiting worker leaves none. 200,000 keys now take 0.24 s (Node 0.30 s). Closes vercel-labs#1702 Co-authored-by: ejc3 <1694850+ejc3@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1619.
WeakMapandWeakSetfound a key by walking every entry of the collection (scr_weak_findovermap->entriesinpackages/runtime/src/scr_json.c), andscr_weak_remove_implwalked the list again to unlink an entry, so filling a collection with n keys took O(n²) time. 200,000 object keys took 64.6 s on Linux arm64 where Node takes 0.12 s.buckets, a power-of-two length) that doubles when its entries outnumber its buckets.get,set,hasanddeletelook up and unlink through it in expected constant time, and tracing, release and collection walk its buckets.scr_weak_hash), which spreads keys over any table size; the old(bits >> 4) ^ (bits >> 14)was fitted to a fixed 1,024-bucket table.Tests:
tests/harness/weak-collection-scaling.test.tsruns the same work with 25,000 and 200,000 keys and bounds the ratio of the two times at 32. With this change it measures 11x to 14x, in both lanes; on main the 200,000-key run is stopped at its 60 s timeout. It followsclosure-environment-scaling.test.ts.Testing
On main at 1ee5c84. Environment: Linux arm64 (glibc), Node 24.22.0, pnpm 11.1.3, LLVM 22.1.8, Zig 0.16.0, with the llvm-linux-arm64-gnu and runtime-linux-arm64-gnu native artifacts built locally from this tree. I have no Vercel Sandbox credentials, so these are the local fallback lanes from CONTRIBUTING.md. The host was shared with other test runs.
weak-collection-scaling.test.tsfails at its 60 s timeout on the 200,000-key run. With this branch it passes, plain and withSCRIPTC_SAN=1. Corpus 1619 passes indifferential.test.tsandllvm-differential.test.ts, plain and sanitized.SCRIPTC_TEST_WORKERS=4 pnpm test: 262 failed, 12620 passed, 126 skipped; main gives 263 failed, 12616 passed on this host, one of them this branch's scaling test, which I had copied there to see it fail. No test fails on this branch only, andlibrary-contract.test.ts's identity-hash case failed on main only. The shared failures: those that target wasm32-wasi (wasi-differential,three,static-spatial,wasm-library,library-dispatch), since no wasm32-wasi runtime pack is built here; 40 innative-toolchain.test.ts, most with ENOENT on scratch cache directories; the 600 s timeout ofcoverage.test.ts's all-static check; the size bound inruntime-tree-shaking.test.ts; and one ASan case inlibrary-mode.test.ts.SCRIPTC_SAN=1: a full local lane with four workers would have taken more than a day here, so I ran it in two parts on this head and on main. The corpus files (differential.test.tsandllvm-differential.test.ts, in 12SCRIPTC_TEST_SHARDshards): 7016 passed, 0 failed (main: 7014 passed, 0 failed). Everything else (--excludethose two, 12 workers): 266 failed, 5571 passed, 155 skipped; main gives 266 failed, 5570 passed, and the failing tests are the same: the ones above, plus three ASan cases inlibrary-multi.test.tsand one inself-hosting-native-driver.test.ts.pnpm lint,pnpm format:checkandpnpm test:ts7 --baselines-onlypass.