Repository navigation
Wait for bursts of external writes before writing the Terminal status - #2170
Conversation
5b00d8a to
286024f
Compare
286024f to
bd0c69d
Compare
Metro review: Looks goodRisk: low · Commit: The PR makes No findings. Verified
Benchmarks (base vs this PR)
Method
Not run: Flow: not checked; Automated review by Claude. Advisory only: it never approves or blocks, and a maintainer makes the call. |
Summary: When output was written outside of `Terminal` (e.g. `console.log`) in a burst, a status update could land between the burst's lines. In a TTY it can't be erased there, so copies of the status were left scattered in the output. - Before writing the status, `Terminal` waits until no output is written outside of it for 333ms (`EXTERNAL_WRITES_QUIET_MS`), in interactive and non-interactive terminals. Outside output is detected through the stream's `bytesWritten`. Without outside output, the status is written synchronously. `flush()` skips the wait. - Status writes wait on a delay promise set after each write: 33ms in a TTY, 3500ms otherwise. `flush()` ends the delay early. - Status delay timers are unreferenced, so they don't keep the process alive. An ongoing status update is flushed on `beforeExit`, so the latest status is still written and `waitForUpdates()` resolves. - `lodash.throttle` is removed from `metro-core` and `metro`: - `Terminal` limits non-TTY status lines through the same status delay. - `TerminalReporter` updates bundle progress on each `bundle_transform_progressed` event; `Terminal` limits the redraws. The `bundle_transform_progressed_throttled` event type is removed from `TerminalReportableEvent`. - The status is erased with ANSI escape sequences written through `Terminal`'s own write, so every write tracks its completion and the bytes written after it. `node:readline` is no longer used. - `isUpdating` and `hasPendingUpdate` are removed: an update is ongoing while `#updatePromise` is set, and pending while the written status differs from the latest one. `metro build` no longer waits for status timers before exiting: on a generated app of 5,000 modules whose build logs 3 lines per transformed module, wall time drops by 28% in prod and 37% in dev (about 2-3s, the trailing `lodash.throttle` timer on `main`), with no significant CPU change and byte-identical bundles. Changelog: [Breaking] Terminal no longer prints the status in the middle of bursts of output written outside of it, and `bundle_transform_progressed_throttled` is removed from `TerminalReportableEvent` Test Plan: - New test "writes the status after a burst of external writes" fails before this change, with a 33ms quiet period, and without non-TTY outside output detection, and passes after. - New test "flush does not wait for the next status update" fails when `flush()` doesn't end the status delay. - New test "flushes the pending status before the process exits" fails without the `beforeExit` listener. - New `TerminalReporter` test "shows the latest bundle transform progress" fails with the throttled reporter. - `Terminal-test.js` uses fake timers. - `yarn jest packages/metro-core packages/metro/src/lib` - `yarn flow check`, `yarn eslint packages/metro-core/src packages/metro/src/lib`, `yarn verify-api-snapshots` - `node scripts/benchmark-patches/benchmark-patches.js bundling_speed --runs=10`
…minal's own writes - The wait for external writes to stop is capped at 2000ms (`EXTERNAL_WRITES_MAX_WAIT_MS`), so sustained external output doesn't hide the status. - Bytes written outside of `Terminal` are counted separately from its own writes, so `Terminal.log()` calls don't extend the wait once it started.
74c4905 to
afd65a3
Compare
There was a problem hiding this comment.
I had a closer look at this, I'm a fan of removing the dependency and the dead wait at the end of bundling, but there are a couple of regressions that need fixing - flickering and too much main thread use:
- Every terminal.log line erases the status, and it only reappears after a 333ms pause.
- It renders the status on every transform progress event, vs previous throttling limiting renders. I measured that at 11% slower warm builds (worse on slow terminals)
But both are fairly easily fixed with two changes: limit progress renders to one per 100ms and have log() redraw the status in the same write. That eliminates flickering and makes main thread load negligible.
Full report, with recommendations and benchmark methodology: https://robhogan.github.io/reports/metro-2170-terminal-status/
|
FWIW, I disagree with the review bot about this being a breaking change. Reporter events have never been considered part of the semver contract (IMO). We can land this in a minor release once it's not regressing. |
…s in Terminal.log() - `TerminalReporter` updates the status on `bundle_transform_progressed` at most every 100ms (`PROGRESS_STATUS_INTERVAL_MS`). `_log` and `_updateState` still run on every event. - `Terminal.log()` writes an erased status again below the line in the same write, so the status stays on the screen, unless it is cleared.
Summary: Publish Metro 0.88.0 from `main`. Changelog: [Internal] Test plan: ``` node scripts/updateVersion.js 0.88.0 sl diff --stat ``` Only the 17 `packages/*/package.json` manifests change, and every `0.87.3` reference is replaced. Flow, ESLint, Prettier, TypeScript, API snapshots, Markdown lint and Jest pass on this commit. Notable breakage in third parties, for the notes or a heads-up before React Native adopts this: - `@rnx-kit/metro-config` 2.2.6 adds `path.dirname(resolver.emptyModulePath)` as a watch folder when there are none, which is now `metro:runtime/modules`, so the dev server never serves a bundle for a single-package app. - `@tamagui/metro-plugin` 3.x refuses `metro-resolver` outside 0.84-0.87, and then imports the removed `metro-resolver/private/createDefaultContext`. - `@rnx-kit/metro-resolver-symlinks` calls the removed `context.redirectModulePath`. End to end (AI-driven), with the packed tarballs against 0.87.3 from npm, in a React Native 0.89.0-rc.0 template app with a few popular dependencies: dev and release bundles, source maps and assets for iOS and Android are byte-identical, except where #2174 changes resolution. `rxjs/operators` now resolves to the same ESM build as `rxjs`, dropping 190 duplicate modules, and `require('nanoid/non-secure')` gets nanoid's CJS build. With #2186, only the module system and `asyncRequire` differ besides those. Release builds reach the first screen on a Galaxy A16 and the iOS simulator. The dev server passes the same checks on both, with and without Watchman (edits, added and deleted files, HMR, `/symbolicate`, assets and lazy `import()`), and Fast Refresh works on the simulator. The end-to-end checks above were on 9de42bbed3b2, before #2170, #2157, #2158 and #1990 landed. I'm re-running them now and will update this. Draft release notes (from the 19 commits in v0.87.3...main - internal-only changes omitted): ``` - **[Breaking]**: Resolver: Read `"exports"` from the package root, and apply `"browser"` redirections to the file it resolves to, by default. Set `resolver.unstable_resolveExportsFromPackageRoot` and `resolver.unstable_applyBrowserSpecAfterExports` to `false` for the previous behavior (#2174 by @robhogan) - **[Breaking]**: Default `transformer.asyncRequireModulePath` and `resolver.emptyModulePath` to `metro:runtime/...`. A custom `resolver.resolveRequest` must delegate `metro:` specifiers it doesn't handle to `context.resolveRequest`, and code that reads either option as a file path must resolve it instead (#2180 by @robhogan) - **[Breaking]**: Resolve the module system via `metro:runtime/polyfills/require`. A custom `resolver.resolveRequest` must delegate `metro:` specifiers it doesn't handle to `context.resolveRequest` (#2184 by @robhogan) - **[Breaking]**: `transformer.globalPrefix` now also prefixes `__r`, `__c`, `__registerSegment` and `__accept`, and the default run statement. Code that calls these by name must apply the prefix (#2186 by @robhogan) - **[Breaking]**: Remove the deprecated `redirectModulePath` from `ResolutionContext`. A custom resolver should drop any call to it and delegate to `context.resolveRequest` or `resolve`, which already apply `"browser"` redirections and return `{type: 'empty'}` for `false` (#2181 by @robhogan) - **[Breaking]**: Remove the deprecated `doesFileExist` from `ResolutionContext`, and the `DoesFileExist` type from `metro-resolver` (use `fileSystemLookup`) (#2181 by @robhogan) - **[Breaking]**: `ResolutionContext.isESMImport` is required (#2182 by @robhogan) - **[Breaking]**: Async bundle and HMR module URLs keep the module's file extension (`foo.js.bundle`) and carry the bundle's `platform`, fixing lazy loading and HMR source maps for files whose extension isn't in `sourceExts` (e.g. `.cjs`, `.mjs`) or that share a name with a sibling (e.g. `foo.json` next to `foo.js`). Old URLs still work (#1990 by @dasjideepak) - **[Breaking]**: Babel transformers can no longer return a `functionMap`. Apply the `plugins` passed by Metro and return the resulting Babel `metadata` to keep function names in stack traces (#2157 by @robhogan) - **[Breaking]**: `/symbolicate` no longer reads `req.rawBody`, and fails if a middleware has already read the request body (#2158 by @robhogan) - **[Breaking]**: Remove the `bundle_transform_progressed_throttled` event from `TerminalReportableEvent`. `TerminalReporter` subclasses should handle `bundle_transform_progressed`, which their `_log` now receives for every progress update (#2170 by @vzaidman) - **[Breaking]**: Source and JSON files containing the reserved string `_$$_METRO_`, including in comments and string literals, now fail to transform (#2155 by @robhogan) - **[Breaking]**: Remove the deprecated, always-`true` `hot` option from `getTransformOptions` (#2159 by @robhogan) - **[Feature]**: Custom serializers receive the reserved string as `options.reservedString` (#2155 by @robhogan) - **[Fix]**: Don't print the terminal status in the middle of bursts of output written outside of it, and don't wait for a pending status update before exiting (#2170 by @vzaidman) - **[Fix]**: Fall back to file-based resolution, instead of throwing a `TypeError`, when a package's `"exports"` field is not a string, array or object (#2187 by @kwy404) - **[Fix]**: Fix relative symlinks on a different Windows drive than the project root (#2161 by @ramya-2196) - **[Performance]**: ~5-10% faster resolution (#2181 by @robhogan) > NOTE: Experimental features are not covered by semver and can change at any time. - **[Experimental]**: Remove `unstable_incrementalResolution` from `ResolutionContext`, and `metro-resolver/private/createDefaultContext` (#2181 by @robhogan) **Full Changelog**: v0.87.3...v0.88.0 ```
Summary: Publish Metro 0.88.0 from `main`. Changelog: [Internal] Test plan: ``` node scripts/updateVersion.js 0.88.0 sl diff --stat ``` Only the 17 `packages/*/package.json` manifests change, and every `0.87.3` reference is replaced. Flow, ESLint, Prettier, TypeScript, API snapshots, Markdown lint and Jest pass on this commit. Notable breakage in third parties, for the notes or a heads-up before React Native adopts this: - `@rnx-kit/metro-config` 2.2.6 adds `path.dirname(resolver.emptyModulePath)` as a watch folder when there are none, which is now `metro:runtime/modules`, so the dev server never serves a bundle for a single-package app. - `@tamagui/metro-plugin` 3.x refuses `metro-resolver` outside 0.84-0.87, and then imports the removed `metro-resolver/private/createDefaultContext`. - `@rnx-kit/metro-resolver-symlinks` calls the removed `context.redirectModulePath`. End to end (AI-driven), with the packed tarballs against 0.87.3 from npm, in a React Native 0.89.0-rc.0 template app with a few popular dependencies: dev and release bundles, source maps and assets for iOS and Android are byte-identical, except where #2174 changes resolution. `rxjs/operators` now resolves to the same ESM build as `rxjs`, dropping 190 duplicate modules, and `require('nanoid/non-secure')` gets nanoid's CJS build. With #2186, only the module system and `asyncRequire` differ besides those. Release builds reach the first screen on a Galaxy A16 and the iOS simulator. The dev server passes the same checks on both, with and without Watchman (edits, added and deleted files, HMR, `/symbolicate`, assets and lazy `import()`), and Fast Refresh works on the simulator. The end-to-end checks above were on 9de42bbed3b2, before #2170, #2157, #2158 and #1990 landed. I'm re-running them now and will update this. Draft release notes (from the 19 commits in v0.87.3...main - internal-only changes omitted): ``` - **[Breaking]**: Resolver: Read `"exports"` from the package root, and apply `"browser"` redirections to the file it resolves to, by default. Set `resolver.unstable_resolveExportsFromPackageRoot` and `resolver.unstable_applyBrowserSpecAfterExports` to `false` for the previous behavior (#2174 by @robhogan) - **[Breaking]**: Default `transformer.asyncRequireModulePath` and `resolver.emptyModulePath` to `metro:runtime/...`. A custom `resolver.resolveRequest` must delegate `metro:` specifiers it doesn't handle to `context.resolveRequest`, and code that reads either option as a file path must resolve it instead (#2180 by @robhogan) - **[Breaking]**: Resolve the module system via `metro:runtime/polyfills/require`. A custom `resolver.resolveRequest` must delegate `metro:` specifiers it doesn't handle to `context.resolveRequest` (#2184 by @robhogan) - **[Breaking]**: `transformer.globalPrefix` now also prefixes `__r`, `__c`, `__registerSegment` and `__accept`, and the default run statement. Code that calls these by name must apply the prefix (#2186 by @robhogan) - **[Breaking]**: Remove the deprecated `redirectModulePath` from `ResolutionContext`. A custom resolver should drop any call to it and delegate to `context.resolveRequest` or `resolve`, which already apply `"browser"` redirections and return `{type: 'empty'}` for `false` (#2181 by @robhogan) - **[Breaking]**: Remove the deprecated `doesFileExist` from `ResolutionContext`, and the `DoesFileExist` type from `metro-resolver` (use `fileSystemLookup`) (#2181 by @robhogan) - **[Breaking]**: `ResolutionContext.isESMImport` is required (#2182 by @robhogan) - **[Breaking]**: Async bundle and HMR module URLs keep the module's file extension (`foo.js.bundle`) and carry the bundle's `platform`, fixing lazy loading and HMR source maps for files whose extension isn't in `sourceExts` (e.g. `.cjs`, `.mjs`) or that share a name with a sibling (e.g. `foo.json` next to `foo.js`). Old URLs still work (#1990 by @dasjideepak) - **[Breaking]**: Babel transformers can no longer return a `functionMap`. Apply the `plugins` passed by Metro and return the resulting Babel `metadata` to keep function names in stack traces (#2157 by @robhogan) - **[Breaking]**: `/symbolicate` no longer reads `req.rawBody`, and fails if a middleware has already read the request body (#2158 by @robhogan) - **[Breaking]**: Remove the `bundle_transform_progressed_throttled` event from `TerminalReportableEvent`. `TerminalReporter` subclasses should handle `bundle_transform_progressed`, which their `_log` now receives for every progress update (#2170 by @vzaidman) - **[Breaking]**: Source and JSON files containing the reserved string `_$$_METRO_`, including in comments and string literals, now fail to transform (#2155 by @robhogan) - **[Breaking]**: Remove the deprecated, always-`true` `hot` option from `getTransformOptions` (#2159 by @robhogan) - **[Feature]**: Custom serializers receive the reserved string as `options.reservedString` (#2155 by @robhogan) - **[Fix]**: Don't print the terminal status in the middle of bursts of output written outside of it, and don't wait for a pending status update before exiting (#2170 by @vzaidman) - **[Fix]**: Fall back to file-based resolution, instead of throwing a `TypeError`, when a package's `"exports"` field is not a string, array or object (#2187 by @kwy404) - **[Fix]**: Fix relative symlinks on a different Windows drive than the project root (#2161 by @ramya-2196) - **[Performance]**: ~5-10% faster resolution (#2181 by @robhogan) > NOTE: Experimental features are not covered by semver and can change at any time. - **[Experimental]**: Remove `unstable_incrementalResolution` from `ResolutionContext`, and `metro-resolver/private/createDefaultContext` (#2181 by @robhogan) **Full Changelog**: v0.87.3...v0.88.0 ```
Summary
When output is written outside of
Terminal(e.g.console.log) in a burst, a status update can land between the burst's lines. In a TTY it can't be erased there, so copies of the status are left scattered in the output.Terminalwaits until no output is written outside of it for 333ms (EXTERNAL_WRITES_QUIET_MS), in interactive and non-interactive terminals, for at most 2000ms (EXTERNAL_WRITES_MAX_WAIT_MS), so sustained output doesn't hide the status. Outside output is detected through the stream'sbytesWritten, counted separately fromTerminal's own writes. Without outside output, the status is written synchronously.flush()skips the wait.flush()ends the delay early.beforeExit, so the latest status is still written andwaitForUpdates()resolves.lodash.throttleis removed frommetro-coreandmetro:Terminallimits non-TTY status lines through the same status delay.TerminalReporterupdates bundle progress state on eachbundle_transform_progressedevent, and updates the status at most every 100ms (PROGRESS_STATUS_INTERVAL_MS). Thebundle_transform_progressed_throttledevent type is removed fromTerminalReportableEvent.TerminalReportersubclasses that handled it should handlebundle_transform_progressedinstead, and their_logoverrides now run on every progress event (e.g. Expo CLI'sMetroTerminalReporter).Terminal.log()writes an erased status again below the line in the same write, so the status stays on the screen, unless it is cleared. Aconsole.logright after aTerminal.log()leaves a copy of the restored status.Terminal's own write, so every write tracks its completion and the bytes written after it.node:readlineis no longer used.isUpdatingandhasPendingUpdateare removed: an update is ongoing while#updatePromiseis set, and pending while the written status differs from the latest one.The performance impact was measured with the benchmark in #2171, which builds a generated app and supports a base patch applied to every variant (results below).
Changelog: [Breaking] Terminal no longer prints the status in the middle of bursts of output written outside of it, and
bundle_transform_progressed_throttledis removed fromTerminalReportableEvent;TerminalReportersubclasses should handlebundle_transform_progressedinsteadTest plan
flush()doesn't end the status delay.beforeExitlistener.TerminalReportertest "updates the bundle transform progress status at most every 100ms" fails without the limit.log(). "keeps logs in order with output written outside of Terminal" expects the restored status copy before the outside line.Terminal-test.jsuses fake timers.yarn jest packages/metro-core packages/metro/src/libyarn flow check,yarn eslint packages/metro-core/src packages/metro/src/lib,yarn verify-api-snapshotsBenchmark
node scripts/benchmark-patches/benchmark-patches.js bundling_speed --runs=10, with abase.patchthat makesmetro buildlog 3 lines per transformed module (30,000 lines per build, interleaved with progress updates), and this PR'sTerminal.jsandTerminalReporter.jschanges as the variant patch. Measured on e7abf93, with this PR's files as of afd65a3, before the 100ms progress limit and the status restore inlog(), which were measured in this review report:Values are mean ± sample sd over 10 runs per variant. t is Welch's t vs unpatched.
The saving is a roughly flat 2-3s per build, not faster bundling: the process no longer waits for status timers before exiting (on
main, the trailinglodash.throttletimer, up to 3.5s after the last status line in non-interactive output). The generated app builds in about 5s, so the percentages are large. Longer builds, like real apps, save about the same seconds, which is a much smaller share of their wall time. CPU time doesn't change significantly, and bundles are byte-identical.base.patch
Benchmark on an internal app
bundling_speedas onmain, which takes an app as a target (5 runs per variant), on an internal app, built cold with Metro from source, withDEBUG=Metro:*, a patch that logs 3 lines per transformed module in the main process, and output piped. Its Metro didn't include #2162 yet, somain-terminaladds #2162, andterminal-external-write-burstsadds #2162 and #2170: #2170's effect is the difference between the two.Values are mean ± sample sd. t is Welch's t vs unpatched; at 5 runs, |t| < 2.3 is not significant (p > 0.05).
No wall time or CPU change is significant, and bundles are byte-identical. This app's Meta internal build command already flushes its terminal when the build finishes, so there's no exit wait to save; the burst handling adds no measurable cost on a large app under heavy logging.