Repository navigation
Conversation
|
Hi @Largo, hope you're doing well. Oddly enough, I have merge access to this repo, so I can merge this PR. It hasn't seen any merges in ages, and I'm not sure how many people still use IronRuby, but it would be a shame for such a substantial contribution to get lost. It'd be great to get this project back to life in the age of agents. Let's see what we can do. |
|
@dotnet-policy-service agree |
|
Hey @amiralles thanks for your comment, I haven't seen it until now. That would be very cool. It's a huge change that I'm sure is hard to review by looking at it. |
The method JIT declines a body in which something that can deopt follows an effect, because its one way out of a failed guard re-runs the generic body from the top. Richards' Scheduler#hold_self is such a body: an instance variable increment, then an attribute write and an attribute read on another object. IR_JIT_VERBOSE=1 now names the reason on the `declined' line. With -X:JITResume (off by default) an attribute read or write on another object after the first effect guards itself and, when its guard fails, makes the source's call once on the operands already evaluated; the typed body goes on. Only bodies with no parameters and no locals, and only self-guarded attribute operations, literals, self and instance variable reads after the first fallback. Kept as the design's stage 1, for the mechanism stage 2 builds on, not as a speed-up: with the switch hold_self is 13-14% faster (0.87 and 0.86 of the same build without it, Util/perf/ab.rb, two runs), under perf-step's 15%, and no AWFY benchmark moves (geomean of all thirteen 1.009 and 1.013 of the unchanged build). Without the switch nothing is compiled differently (1.018 and 0.996). Checked: IronRuby.Tests has the one failure the unchanged build has; Util/jit-matrix.rb prints what CRuby 4.0.9 prints with the switch on, off and with -X:NoJIT, RS_CHECK and ED_CHECK pass; six other matrices print what the unchanged build prints; a ruby/spec sweep of all five suites with the switch on has no failure Util/spec-known-failures.txt does not list. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
class Node; attr_accessor :val; end
class Hidden < Node; private :val, :val=; end
def peek(o) = o.val
Once peek was hot, peek(Hidden.new) answered with the variable where CRuby
4.0 (and IronRuby with -X:NoJIT) raises "private method 'val' called for an
instance of Hidden". The same for `protected', for the writer, and for a
Struct member made private in a subclass of the Struct class.
`private :val' in a subclass leaves an entry there that only forwards to the
inherited method (SuperForwarderInfo). Method resolution follows it, so the
method it hands back is the superclass's, which is public, and the JIT's
three binders of an attribute operation with an explicit receiver
(SelfAttrSite.ReaderVariable and WriterVariable, StructMemberSite.Member)
asked that method whether it was public. They now ask the class's lookup
(RubyModule.GetMethodLookup), which keeps the forwarder's visibility and is
what ResolveMethod answers from for a class anyway. The JIT's other checks
of a method's visibility (inlined calls, ==, the built-in Array, Hash and
String methods, new, Math.sqrt) gave CRuby's answer for the same shapes
and are unchanged.
Found while gating -X:JITResume, whose fallbacks bind through the same site:
with the switch a hold_self-shaped body wrote through the private writer too.
Util/jit-matrix.rb section 12 has the rows (ten of them differ on the build
before this change; CRuby, the JIT, -X:NoJIT, -X:JITResume and -X:JITCalls
print the same now), and JitAttrVisibility1 the test, with and without
-X:JITResume.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
…question Records the experiment merged in d809920: what stage 1 of Docs/jit-resume-deopt.md compiles, that it is 13-14% on hold_self with the switch and nothing measurable on the benchmarks, so it does not meet perf-step's rule and is kept for the mechanism as the design says, and what the numbers say about gap 1 (a leaf body of this size is mostly its entry). Records the fault the gate's probes found on master and d4a9724 fixed (an accessor a subclass makes private), why the experiment's own private and protected rows passed all the same, and adds the question to the perf-gate skill: make the method private where it is defined and with `private :x' in a subclass. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
… body) Four methods modelled on Richards' Scheduler and TaskControlBlock, each one the method JIT's screen rejects today for a call it makes: the measurement stage 2 of Docs/jit-resume-deopt.md is judged by. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
…l mode" and rejected Stage 2 of Docs/jit-resume-deopt.md was built on perf/resume2 as a second compile of the bodies the typed compiler declines: every operation self-guarded or not speculative, all values objects, general calls through a light-scope site, the body in the generic frame's place. It was correct on the matrices, the test suite, ruby/spec language and the benchmarks, and it was not faster: the four shapes it was aimed at measured the same with the switch on and off in both column orders (ab.rb, 7 rounds), and all thirteen benchmarks were 1.028 of the base in both orders, Json, Towers and Bounce 7-13% slower. The step's rule asks for 15% on the shape or 3% on the geomean, so the branch is not merged. The ledger's gap 1 now says what the attempt showed (the generic body with its call-site rules already is a compiled body of guarded object operations with a frame, so the gain has to come from dropping the frame, the boxes or the dispatch between two compiled bodies) and the Rejected list has the numbers and the two faults found on the way. The design document's stage 2 points there. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
-X:JITCalls, the prototype that lets a method the JIT compiles call the typed bodies of the other methods it calls on self, was measured on its own shapes only (Util/bench/jit-methods.md) and left off. On the thirteen Are-We-Fast-Yet benchmarks it makes List 28% faster (`tail` calls `is_shorter_than`, a callee with a loop that the inliner does not take, 39.2 -> 28.3 ms in six alternating processes per build) and leaves the other twelve inside their spread, geomean 0.976. The default of the JITCalls option is now true and -X:NoJITCalls turns it off; -X:JITCalls is still accepted. JitCalls1 checks that the default engine binds calls and one with the option off binds none, and JitMethods1 sets the option both ways. The matrix header, the bench notes, the README's performance section and the two perf skills say what the default is. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
Round thirteen of Util/perf/LEDGER.md. With the default on, List is 26-29% faster in four awfy.rb runs and 28% in six alternating processes per build (39.2 -> 28.3 ms); the other twelve benchmarks stay inside their spread (geomean 0.976 to 0.986), no micro line is slower in both orders and start-up is the same. The gate: IronRuby.Tests fails the same one test on both trees; sixteen differential matrices print what the baseline prints, with direct calls on and with -X:NoJITCalls; a sweep of all five ruby/spec suites (32,285 examples) has no failure the list of known ones lacks; probes of deep recursion through a bound callee, an effect before a callee's deopt, truth against value and a callee overridden or made private in a subclass give CRuby 4.0's output. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
… by default Records what fd13114 measured and what its gate ran: List 28% faster, the other twelve benchmarks unchanged, nothing slower; the eight methods the wider screen now offers and the compiler declines, and that Richards, which has three of them, is the same in ten alternating processes. Gap 1 says what direct calls reach and what they do not (callees without a typed body; calls on another receiver), and the entry notes a difference from CRuby found on master while probing: a caller inside its compiled loop does not see a callee that another thread redefines until it returns. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
awfy.rb ran thirteen of AWFY's fourteen benchmarks: Havlak was left out because its som.rb reopens Set, which CRuby 4.0 made a core class, so CRuby fails it. IronRuby and TruffleRuby run it, and a comparison of two IronRuby trees does not need CRuby's column. `--whole` (or the name after `--`) adds it, with at least a minute per interpreter since one iteration takes seconds; CRuby's cell is FAIL and stays out of its geomean. The default is unchanged. The perf-step and perf-gate skills ask for one whole run per decision. Checked: the default prints the thirteen as before; `--whole -- Havlak Sieve` gives IronRuby a time for Havlak and CRuby a FAIL with its last lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
The method JIT declined a whole method at `s.size' or `@input.length' on a
receiver the profile saw hold a String ("size on a MutableString"): the
size/length intrinsic knew only Array and Hash. In Json that kept
HashIndexTable#string_hash out, and with it, through calls on self,
hash_slot_for, add and get.
A receiver the profile saw hold a String now compiles to
JitRuntime.StringLength: for an object whose CLR type is MutableString itself
(not a subclass) the library's character count in the String's encoding, a
broken byte counting as one; anything else deopts. String#length/#size must
be the public library method and no String may have a singleton one
(HasPlainInstances), checked when compiling; a redefinition later moves the
method version the entry guards on.
Json: string_hash, hash_slot_for, HashIndexTable#add and #get specialize
(specialized 11 -> 15). Util/bench/jit-core-shapes.rb str_size 95.9 -> 49.4
ns (CRuby 53.0), str_length 58.1 -> 48.5. jit-matrix section 13 (multi-byte,
broken, binary, UTF-16 Strings, subclasses, a singleton length, an Array,
nil, length redefined mid-run, made private, refined) is identical under
CRuby, this build and -X:NoJIT -X:NoOSR; test JitCore_StringLength1.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
`n.abs' on a number declined the whole method ("call: abs/0"): Bounce's
Ball#bounce, and Util/perf/micro.rb's `abs' shape (`@v = 0 - @v.abs'), the
per-operation gap the ledger has at 1.4-1.8 of CRuby.
On an Integer or Float operand, with Integer#abs or Float#abs the public
library method, the body computes it: an Int32's absolute value as an Int64
(-2**31 has none in an Int32), an Int64's the same except at -2**63, whose
absolute value is a Bignum (JitRuntime.AbsLong deopts to the generic code),
and a Float's with Math.Abs, so -0.0.abs is 0.0 and NaN stays NaN, as in
CRuby.
Util/bench/jit-core-shapes.rb (load 2.9-4.6): abs 63.9 -> 48.4 ns, fabs 62.4
-> 49.8 ns (CRuby 32.6 and 50.0). Bounce's `bounce' itself now gets past abs
and is declined for a deopt after an effect (the ivar writes before the
guarded reads), which is the effect rule's, not this change's. jit-matrix
section 14 (Int32 and Int64 edges, Bignums, -0.0, NaN, infinities, a
Rational, nil, a Float-hot body given Integers, both abs redefined mid-run,
made private, refined) is identical under CRuby, this build and -X:NoJIT
-X:NoOSR; test JitCore_Abs1.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
som.rb's Vector#at and #capacity (`@storage == nil'), used by DeltaBlue, CD and Json, were declined at "== on a RubyArray": the method JIT took `v == w' on an object as identity only for receivers whose == is BasicObject's. For `v == nil' or `v != nil' (a nil literal) on a receiver the profile saw hold an Array or a Hash, JitRuntime.CollectionEqualsNil answers false for an exact Array or Hash: Array#== and Hash#== of an object of another type are false unless it responds to to_ary / to_hash (rb_ary_equal, rb_hash_equal), which nil does not while NilClass defines neither (checked when compiling, with Array#== / Hash#== (and #!=) the public library methods and no singleton ones). Any other receiver, nil included, goes to IsIdentical, which answers for BasicObject#== and deopts otherwise. Vector#at and #capacity specialize in DeltaBlue (26 -> 28), CD and Json. Util/bench/jit-core-shapes.rb (load 2.4-5.4): vec_at 94.8 -> 57.1 ns, vec_capacity 72.9 -> 47.7 (CRuby 79.6, 73.9). AWFY at 12 s: DeltaBlue, CD, Json 0.99, 0.96, 0.99 of the base, inside the noise. jit-matrix section 15 (nil, Hashes, Strings, subclasses and an object with an == of their own, a singleton ==, nil given to_ary and == mid-run, Array#== redefined, Hash#!= defined, Array#== made private) is identical under CRuby, this build and -X:NoJIT -X:NoOSR; test JitCore_CollectionEqualsNil1. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
…rejected The first experiment of round fourteen made a restart after an effect invisible by logging stores of instance variables, attributes and Struct members and taking them back on a deopt, as typed bodies already do for element writes. It was correct on the JIT matrix and compiled Richards' hold_self and DeltaBlue's remove_first, and the compiled bodies were 14-31% slower than the generic ones: the log's bracket at the entry and the checked stores cost more than the inline-cached stores of the generic body. Its author reverted it on the branch; the ledger keeps the design, the numbers and the lesson. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
…t::NAN Each IronRuby.Tests test runs without ruby4.rb, which defines String#b and Float::NAN, so JitCore_StringLength1 and JitCore_Abs1 failed with a NoMethodError and a NameError before reaching the JIT. They now build the binary String with force_encoding and NaN and -Infinity by division; the expected output is unchanged (checked with ruby -w), and the three JitCore tests pass. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
…d block loop
The block-loop tier outlined only innermost blocks: its screen rejected a block
that itself held a call with a block, so NBody's `each_index { (i + 1...n).each
{ } }' in advance and energy ran both blocks through yield, every Float crossing
the inner block's boundary boxed.
The screen now admits, as a statement of the block, `(a...b).each', `n.times',
`a.upto(b)' and `a.downto(b)' with a literal block that passes the same screen
(NestedBlockLoop), and the loop emitter writes that loop in place
(JitCompiler.Nest.cs): bounds evaluated once in source order and checked to be
Int32 Integers (a deopt otherwise, which restarts the outer iteration in the
generic code), the iterator checked to be the library's at compile time (the
method version guard keeps it so), the inner block's parameter and locals CLR
locals of the copy, its own safe point. A block whose last statement runs in
place is taken only by the kinds that drop the block's value.
Checked with a new section of Util/block-loop-matrix.rb (hot nested shapes with
late Float, Bignum, String, empty and backwards bounds, jumps, raise and caller,
throw, procs, binding, three levels, a frozen body, Range#each and
Integer#times redefined, Thread#raise, TracePoint, recursion a million deep):
identical under CRuby, this build and -X:NoJIT -X:NoOSR. New test
BlockLoopNested1. NBody's harness result verifies.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
…ect to the call A review found three faults in the core-method operations of this branch. `private :!=' (or protected) in Array or Hash, where the method is BasicObject's, stores a forwarder that ResolveMethod follows to BasicObject#!=, whose own visibility is still public. IsPublic read that, so `a != nil' on an Array compiled onto CollectionEqualsNil and answered true where CRuby raises NoMethodError. IsPublic and IdentityCallSite.Resolves now read the visibility a call sees (RubyModule.GetMethodLookup), which also closes the same hole for the baseline's String and identity comparisons. `v.abs' on an object whose attr_reader is named abs bailed with `abs on Obj' before the attribute read was tried, so methods the baseline compiled were declined (about 15% slower on that shape). The abs operation is now tried only when the receiver's profile is a number or unknown. jit-matrix.rb section 16 and JitCore_Review1 add rows for both, and for abs of an Int64-typed value and -2**63 (JitRuntime.AbsLong), which no row reached. The matrix equals CRuby's with and without -X:NoJIT -X:NoOSR; IR_JIT_VERBOSE shows rv_la, rv_attr specialized (rv_ne/rv_neh specialize when the section is run alone; in the full matrix an earlier section leaves them declined). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
…rator per outer iteration
Review of the nested-block change found two wrong results that the baseline
does not show on `n.times { m.times { } }`:
- Thread#raise stops the inner loop at its safe point by a deopt, which takes
the outer iteration back to its top and rolls its locals back; a global
stored earlier in that iteration stays stored, so it was left ahead of the
local it was stored from (21 of 30 runs). An outer block that stores a global
(directly or through a bound callee) beside an iterator run in place is now
not specialized.
- The copy checked the method version on entry only, so an Integer#times
redefined by another thread kept running the old loop in later outer
iterations. Each outer iteration now checks the version where it reaches the
in-place iterator (JitRuntime.NestedIteratorOk) and deopts if it moved.
The matrix rows that never reached the nested copy (outer runs below the
block's floor of 5) now use longer outer runs; the redefinition rows use an
each_index outer loop over 8 elements. New rows: Thread#raise after a global
store, Integer#times redefined by another thread; the same two cases are in
BlockLoopNested1. IR_OSR_VERBOSE shows the rows reaching the copy (or, for the
global row, the new bail). block-loop-matrix: CRuby, this build and
-X:NoJIT -X:NoOSR identical; all six matrices identical with the baseline;
the nested shapes are still 0.27-0.32 of the baseline; thirteen AWFY
benchmarks run without FAIL; IronRuby.Tests fails only
Exception_CauseRaisedInClause1, as on the baseline.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
… inside a block loop Round fourteen of Util/perf/LEDGER.md, from a workflow on TruffleRuby's JIT techniques (blocks inlined into the loop that yields to them). The block-loop tier took only innermost blocks; an outer block may now hold `(a...b).each`, `n.times`, `a.upto(b)` or `a.downto(b)` with a literal block, whose loop is emitted inside the outer one over the same typed locals, so a Float that crosses the inner block is not boxed. NBody's `advance` is such a loop: NBody runs in 0.59 of its time in five alternating processes per build (1447 -> 859 ms) and 0.58 and 0.60 in two runs of all fourteen benchmarks, which are otherwise inside their spread (Richards and Sieve, 4-6% slower in both of those runs, are 1.004 and 1.014 in six alternating processes). An independent review found two thread-timed faults, fixed on the branch: an outer block that stores a global beside the in-place iterator is not taken (a Thread#raise at the inner safe point restarts the outer iteration, which takes locals back but not a global), and each outer iteration re-reads the method version where it reaches the iterator. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
… in typed bodies Round fourteen of Util/perf/LEDGER.md, from a workflow on TruffleRuby's JIT techniques (core methods as specialized operations of the compiled caller). The method JIT declined a whole method for `s.length`, `s.size`, `x.abs` as a call, and `a == nil` / `a != nil` on an Array or a Hash. Each is now an operation of the typed body, checked when it compiles against the public library method. Shapes, ab.rb with five rounds in both column orders, CRuby / before / after in ns: String#size 60 / 118 / 58, #length 41 / 71 / 54, Integer#abs 43 / 76 / 54, Float#abs 65 / 78 / 58, Vector#at 108 / 115 / 68, Vector#capacity 96 / 84 / 53. Json compiles five more methods, DeltaBlue two and CD one; the fourteen benchmarks are within their spread (geomean 0.993 and 0.992). An independent review found that `a != nil` answered true where `!=` had been made private or protected, because the guard read the visibility of the method a forwarder resolves to; the guard now reads the lookup's visibility, which also corrects the same test on the String and identity comparisons that were compiled before. `v.abs` through an attr_reader named abs compiles as an attribute read again. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
…n typed bodies Records the two experiments merged in c256bd7 and 8081395: what each does, my measurements (NBody 0.59 of its time; the six core shapes 0.48-0.77; the fourteen benchmarks 0.960 with both), the gate's runs on the merged tree, and the merged build against CRuby and TruffleRuby (geomean 0.629 of CRuby, NBody 0.39). Gaps 1 and 2 say what is left behind each. A new section, "Caught by the review", keeps what the independent reviews of this round found: three candidate-only faults that the authors' matrices had passed, fixed before the gate, and the differences from CRuby that were on master already and are still open - first among them that a compiled loop which is running does not see a method another thread redefines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
The method JIT declined every method that read a constant whose value
is nil ("constant without a value"), because an autoload that has not
run also resolves to no value. Richards' NO_WORK = nil declined
add_input_and_check_priority, append and two initialize methods.
A constant set to nil has a storage whose weak value is
ConstantSiteCache.WeakNull, as the generic constant site's cache keeps
it; an autoload not yet run has no such storage. The JIT now takes the
first and still bails on the second. ObservedType answers for a
constant with the class of the value the constant version guard keeps,
and `NONE == x', `NONE != x' on such a constant is `x is nil' while
NilClass#== (and BasicObject#!=) are the public library methods and no
refinement is active.
Richards: the two initialize methods are now specialized;
add_input_and_check_priority now waits on append, which is declined for
"a deopt after an effect" (it starts with a store), the rule that
stays. No benchmark moved (Richards 1.049 of base, ranges overlapping,
load 5); the nil_const shape in Util/bench/jit-core-shapes.rb went from
60.4 to 38.1 ns (0.63, CRuby 32.1).
Checked with Util/jit-matrix.rb section 17 (reassigned, removed,
private_constant, deprecate_constant, lexical and inherited lookup,
Object:: and ::, NilClass#== redefined and NilClass#!= made private
mid-run, a refinement, an autoload), the same under CRuby, this build
and -X:NoJIT -X:NoOSR, each JIT row specialized per IR_JIT_VERBOSE; and
the new test JitCore_NilConstant1.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
…n entry A compiled loop - an outlined while/until, a block loop over times or each, a typed method body with a loop - checked the global method version when it was entered and never again. So a loop that was already running went on with the old Array#size, String#length, Integer#abs, an inlined Integer#to_i or an inlined or bound callee after another thread redefined it, until the loop or its method returned. CRuby calls the new method at the next call; so does IronRuby with -X:NoJIT -X:NoOSR. The safe-point counter (RubyUtils) becomes the low half of a 64-bit word whose high half is a code epoch, bumped after every method table version bump and every ConstantAccessVersion bump (RubyContext.ConstantsChanged). The loop's existing safe point compares the whole word with the epoch the code was compiled at, an immediate, so one compare sees both a waiting interrupt and a redefinition, as TruffleRuby's safepoints do. - An outlined loop hands the loop back at the iteration boundary when the word moved (JitRuntime.OsrHandBack); the path out is a throw, so the CLR JIT keeps nothing live across it (a returning slow path made it spill the loop counter, an each over an Array 15-30% slower). The site (OsrLoopSite.Run) finds the copy stale, or its epoch moved, and builds one for the new epoch without counting a deopt - also when the prologue's version guard fails, which used to spend one of the three retypes. A rebuild waits until the loop has run eight times as long as the last compilation took, so a thread defining thousands of methods beside a running loop cannot keep it in the compiler, and the loop is never retired for it. - Inner loops and perf/nest's in-place iterators use the same check, which replaces NestedIteratorOk: they already deopt to the outer iteration's top. - A typed method body's loop keeps its slow path, which runs a trap handler in place: it now also checks the versions after it, and a body that may deopt (it can deopt anyway, so it is pure, or it logs its element writes) deopts when a method or a folded constant changed, by another thread or by the handler. An impure body cannot, and goes on. Checked with the reviewers' probes (q_thread_redef3, p07b, flight.rb pure and impure): CRuby's output on this build, not on the baseline; osr-array-matrix and block-loop-matrix unchanged under CRuby, this build, and -X:NoJIT -X:NoOSR. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
…runs osr-array-matrix.rb section 10 (outlined while loops), block-loop-matrix.rb's last section (times, each, a nested times) and jit-matrix.rb section 17 (typed method bodies): another thread redefines what the running loop was built on - Array#size, String#length, Integer#abs, Integer#+, a folded constant, an inlined and a bound callee; by def, alias, define_method and prepend; from a Signal.trap handler and from the rescue of a Thread#raise'd exception, after which the loop is entered again - then sets a flag, and the loop counts only iterations after the one that saw the flag, so the answer is CRuby's every run. The last OSR row defines 10,000 methods beside a running loop. Each prints the same under CRuby, this build and -X:NoJIT -X:NoOSR; the baseline differs on 8 of the OSR rows, 4 of the typed-body rows and the each row. IR_OSR_VERBOSE=1 / IR_JIT_VERBOSE=1 show every row's loop specialized and then handed back (or deopted) at its safe point, except the outlined callee row, dropped: an outlined copy inlines no Ruby callee, so that case is the typed body's. JitRunningLoopRedefined1 is the registered test. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
MethodsUpdated visits every class that depends on the module changed - a method defined on Object visits all of them - and IncrementMethodVersion bumped the code epoch for each, one interlocked add per class. The epoch only has to move once, after every version the update bumped; it now does, at the end of MethodsUpdated and AllDependentMethodsUpdated. Checked: osr-array-matrix, block-loop-matrix and jit-matrix's section 17 as before; a loop running while another thread redefines String#length 10,000 times (each followed by a call that binds a site to it again, so every one bumps the version): the loop is rebuilt 127-136 times, spends 465-490 ms of about 4.3 s compiling (StaleRebuildSpacing's ninth), ends compiled, and the run takes 8.1 s against the baseline's 7.5 s, which keeps the stale copy and computes the same only because nothing it inlined changed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
With the epoch in the high half, the loop's word with no work waiting
was a 64-bit immediate: the safe point became a mov into a register and
a compare, and the CLR JIT moved the loop's counter between registers
around it. The empty-bodied n.times { |j| s ^= j } loop was 14% slower
than the baseline's (ab.rb, 15 rounds, 1.1 against 1.3 ns), an each over
an Array 6%.
The epoch is now the low half, kept below 2^31 (it wraps within its 31
bits, by a compare-and-swap, never into the count), and the count of
waiting safe-point work the high half, which the interrupt test reads
as an int. The loop's test is one cmp qword [word], imm32 - the
outlined times loop's code is the baseline's but for that instruction -
and the interrupt test is the 32-bit compare with 0 it was.
Checked: the three matrices as before (CRuby, this build, -X:NoJIT
-X:NoOSR identical); loop shapes one per process, 11 rounds, new/base
0.95-0.99 for the outlined while, nested while, times, each and nested
times, 0.80 and 0.81 for the typed method bodies.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
The epoch moved on every method table update and every constant version bump, though a method table update moves GlobalMethodVersion only when it reaches a class, and most constant bumps (a new constant, include, extend, prepend, an autoload starting or ending, the Ractor and namespace bumps) change nothing a loop folded. Each move handed every running loop in the process back and kept it generic for StaleRebuildSpacing: a thread looping Object.new.extend(M), const_set of new names or a second engine's extend made a running loop 8-15x slower than the baseline. Now MethodsUpdated and AllDependentMethodsUpdated move the epoch only if GlobalMethodVersion moved in the update - after which every typed body and loop copy is stale at its next entry on the baseline too - and of the constant changes only an existing constant assigned again or removed moves it (RubyContext.ConstantReassigned); the rest bump ConstantAccessVersion alone, as before this branch. A copy that is not stale is thrown away only after such a move, which is now rare by construction. The matrix rows after the first redefinition of Array#size and String#length shared their loop with an earlier row, whose site was no longer specializable by then, so they did not reach the copy: they get loops of their own and now show the hand-back. Checked: the review's probes (b2, b1 in all eight modes, a6, e1), base and this build alternating, five runs each: medians within noise, stale=0 except Struct.new (26, as it moves the method version); the three matrix sections the same under CRuby, this build and -X:NoJIT -X:NoOSR, each row reaching the hand-back or deopt; JitRunningLoopRedefined1 passes; loops.rb shapes 0.96-1.03 against the baseline, typed bodies 0.79-0.84. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
Both branches added a section 17 to Util/jit-matrix.rb. The one about a typed body's loop that meets a redefinition stays 17 (osr-array-matrix.rb refers to it by number) and runs first; the one about a constant whose value is nil becomes 18 and runs after it, because it ends with a `using`, and a refinement active for the rest of the file would keep the methods of a later section out of the paths that section is about. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPwZf7WbR4bihMQfHYjG4L
C API: ruby.h compiles as C++
C API: $~ and $_ are the calling frame's; rb_Hash, rb_Rational, rb_Complex, finalizers, arithmetic sequences
C API: rb_io_t for GetOpenFile, IO waits, fd sets
C API: keyword calls, $! in rb_rescue/rb_ensure, rb_get_kwargs
C API: C globals follow $stdout/$/, constants, errinfo, symbols, Data
C API: encodings beyond ASCII and UTF-8
C API: st tables, rb_f_notimplement, enumerator sizes, autoloads
The check hardcoded 4.0.0, so the move to 4.0.10 failed it. What it tests is that puts reaches stdout, not the version, which RUBY_VERSION's own specs cover.
RUBY_VERSION 4.0.10: IronRuby matches CRuby 4.0.10
ruby/debug.h had no implementation: rb_tracepoint_new and friends raised
NotImplementedError. rb_tracepoint_new now makes an ordinary TracePoint (the
RUBY_EVENT_* bits are TraceEvents') whose hook calls the C function with the
TracePoint; rb_tracearg_* answer from the TracePoint's readers, so
rb_tracearg_from_tracepoint hands back the TracePoint's own handle.
rb_debug_inspector_open lists the frames caller_locations reports. The C method
that opened it and its Ruby caller get self, class and a binding. IronRuby keeps no
chain of the callers' scopes, so frames further up have only their location.
Also:
- rb_iter_break called from a Ruby block (C method inside `3.times { ... }`)
breaks out of the innermost block around the call, as a break in eval'd code
does. It used to raise NotImplementedError.
- Kernel#binding called from C (rb_funcall(self, "binding")) raises
RuntimeError, as CRuby does, instead of capturing the C method's caller.
- ObjectSpace.memsize_of a typed T_DATA object adds what its dsize function
reports. IMemorySized moves into IronRuby.dll, so DataObject can implement it.
- rb_time_new, rb_time_nano_new and rb_time_timespec_new no longer go through
Time.at, which a program may redefine (CRuby makes the Time directly). They
add an exact Rational to the epoch instead.
- A FREEZE bit that C code wrote into RBASIC(obj)->flags itself now freezes the
object. The receiver's and arguments' slots are checked when a C method returns.
- rb_fd_fix_cloexec maps IronRuby's IO#fileno number to the native descriptor
before fcntl, so the flag lands on the right descriptor.
spec/optional/capi failures and errors go from 21 to 1 (digest_spec, next
commit), with none new. IronRuby.Tests passes, and so do spec/core/tracepoint,
Kernel#binding, Time.at and spec/library/objectspace.
A C extension can bring its own hash function to Digest: it wraps an
rb_digest_metadata_t with rb_digest_wrap_metadata, which ruby/digest.h finds
through rb_ext_resolve_symbol("digest.so", ...), sets it as the "metadata" of a
Digest::Base subclass, and digest.so's Base then runs the plugin's
init/update/finish functions on a context of ctx_size bytes. On IronRuby
rb_ext_resolve_symbol answered NULL, so such an extension failed to load with
"rb_digest_wrap_metadata not found", and a Ruby subclass of Digest::Base could not
be instantiated at all.
The shim now provides the wrapper (a typed object of type "digest/metadata").
Digest::Base gets a constructor for Ruby subclasses: like digest.so's allocator,
it looks for "metadata" up the class chain, raises "Digest::Base cannot be
directly inherited in Ruby" when there is none, and otherwise creates and
initializes the native context (NativeDigest). #update, #finish (which starts the
context over, as digest.so's does), #reset, #initialize_copy, #block_length and
#digest_length then go to the plugin. The prebuilt AOT subclass of Base gains the
matching constructor (Util/aot/gen-subclasses.rb).
spec/optional/capi/digest_spec.rb passes, so the C API specs now pass in full
(1660 examples). spec/library/digest and IronRuby.Tests pass.
C API: TracePoint, debug inspector, digest plugins; capi specs pass in full
Keep the cause of a C#-raised error in a rescue clause once the method is compiled
Installing and running real C extension gems turned up functions the spec suite never
calls:
- rb_ext_ractor_safe (RB_EXT_RACTOR_SAFE), which most current extensions call
first thing in Init_ (jaro_winkler, msgpack, oj...). It was a stub that raised,
so the extension failed to load. The rest of ruby/ractor.h comes along:
- The Ractor's stdin, stdout and stderr are $stdin & co.
- rb_ractor_make_shareable(_copy) and the rb_ractor_shareable_p fallback go to
Ractor.
- Ractor-local storage keys hold one value each, since an extension only runs
in the main Ractor here.
- rb_obj_freeze_inline (RB_OBJ_FREEZE) and rb_must_asciicompat: bigdecimal, oj.
- rb_hash_size_num (RHASH_SIZE): msgpack.
- rb_opts_exception_p, object.c's handling of the exception: keyword, which
bigdecimal looks for with have_func: bigdecimal, oj.
- rb_memhash: oj.
With these, and the two commits that follow, these gems build with `gem install`,
load and give the same results as on CRuby 4.0 for a script exercising each:
- fast_blank, redcarpet, jaro_winkler, escape_utils, racc (cparse),
digest-murmurhash and unf_ext (C++).
- With IRONRUBY_NATIVE_GEMS: bigdecimal, msgpack, bcrypt and oj.
spec/optional/capi still passes in full.
IronRuby raised TypeError "Cannot undefine 'initialize' method". CRuby 4.0 allows it. `undef`/undef_method warn "undefining 'initialize' may cause serious problems", as they do for object_id and __send__, and C's rb_undef_method says nothing. racc's C parser does it in its Init (rb_undef_method(CparseParams, "initialize")), so racc/cparse failed to load. new on such a class allocates the object and calls #initialize anyway, which ends in method_missing: NoMethodError "undefined method 'initialize' for an instance of D" unless the class defines method_missing, as in CRuby. Object construction had asserted that initialize always resolves; it now builds that call when it does not. Scenario_RubyInitializers6 asserted the old TypeError and now checks CRuby's behaviour and its warning. IronRuby.Tests, spec/language, spec/core/class, spec/core/module and spec/core/basicobject pass, apart from the known into/to wording errors.
Two problems with installing real C extension gems: - Gem::Specification#build_extensions was redefined in the class body, with a `super` meant for RubyGems' own method. That method was what it replaced, so `super` reached BasicSpecification, which has none. `gem install oj` failed with NoMethodError as soon as it got to oj's installed bigdecimal dependency. The override is now a prepended module, as the installer's already was. - With IRONRUBY_NATIVE_GEMS naming msgpack (or oj, or bcrypt), the gem was activated ahead of IronRuby's built-in on $LOAD_PATH, and its compiled extension still never loaded: `require "msgpack/msgpack"` takes a .rb from anywhere on the load path before a .so, as CRuby's does, and Src/StdLib/ironruby/msgpack/msgpack.rb is IronRuby's implementation under the extension's name. The three built-in files that sit under an extension's name (msgpack/msgpack.rb, oj/oj.rb, bcrypt_ext.rb) now ask Gem.ironruby_native_extension_for first, and load the native gem's .so instead when there is one. Without IRONRUBY_NATIVE_GEMS the lookup returns at once and nothing changes. Checked by installing msgpack, oj and bcrypt with IRONRUBY_NATIVE_GEMS='*' and comparing a script that exercises them with CRuby 4.0, and by running the same script without the variable (the built-ins load as before). The C API README lists the gems tried.
Real C extension gems: install, load and match CRuby
UndefineStatement built its expression without a sequence point, unlike every other statement, so an undef that is the first statement of a class body ran with the line of the `class` keyword still current. The NameError for an undefined method pointed there, and so did the warning for undefining object_id. Its expression now carries its location, so both report the undef's line, as in CRuby.
#250 made undef of initialize, object_id and __send__ warn, but the other half of CRuby's warnings was missing, and the warning came at the wrong time: - CRuby's rb_undef warns before it looks for the method, so undefining one that is already gone warns and then raises NameError. IronRuby checked first and raised without the warning. - remove_method of those three warns "removing 'X' may cause serious problems" (rb_remove_method_id); IronRuby said nothing. - A new entry for object_id or __send__ that is not a C function (def, define_method, alias, attr_reader, a visibility change, undef) warns "redefining 'X' may cause serious problems" when the class already has the method, defined or undefined, itself or from an ancestor (rb_method_entry_make). So `undef object_id` in a class gives two warnings, as in CRuby. A module defining __send__ has no ancestor with it, and gets none. BasicObject#initialize could not be removed: it was kept "to simplify object construction logic". Object construction handles a class without initialize since #250, so it is now removed like any other method, with the warning, and Object.new then raises NoMethodError, as in CRuby. IronRuby also defined an initialize of its own on Object, which CRuby does not have (Object.private_instance_methods(false) listed it). It is gone, so the BasicObject's is the one every class inherits. Checked against CRuby 4.0 case by case (undef twice, undef_method, def, define_method, alias, attr_reader, private, a singleton def, remove_method, module __send__, removing BasicObject#initialize). New tests UndefCoreMethodWarnings1 and RemoveBasicObjectInitialize1. RubyInitializersCaching3 asserted the old NameError and now checks CRuby's behaviour.
- Ractor-local storage keys are shared by every thread, and there is no GVL to order their reads and writes. Setting a key now stores the value before the flag that says there is one (release), and reading checks the flag first (acquire). Before, a reader on a weakly ordered CPU could see the flag before the value. - rb_hash_start started every process from the same constant, and rb_memhash passed a seed of 0, so an extension's hash values were the same from run to run. CRuby seeds both from a random per-process value (rb_hash_seed). rb_hash_start now adds such a seed, and rb_memhash starts from rb_hash_start(0), as CRuby's does.
msgpack/msgpack.rb, oj/oj.rb and bcrypt_ext.rb each carried the same five lines
that look up the native gem's extension and load it in their place. That is now
Gem.ironruby_require_native(feature), true when it loaded one, so each file says
`return if ... Gem.ironruby_require_native("msgpack/msgpack")`. The behaviour is
unchanged.
…t CRuby ruby/spec's capi suite never calls the functions #250 added for real gems (rb_opts_exception_p, rb_must_asciicompat, RHASH_SIZE, OBJ_FREEZE, rb_memhash, Ractor-local storage, rb_ractor_make_shareable...), and they were checked only by installing gems by hand. Util/capi/selftest is a small extension calling each. Util/capi/selftest.rb builds it with mkmf and checks the results. They are CRuby 4.0's: `ruby Util/capi/selftest.rb` passes too. On IronRuby it also checks IRONRUBY_NATIVE_GEMS's handoff: with the variable naming a gem, Gem.ironruby_require_native loads that gem's compiled extension. Without the variable, it leaves the feature to IronRuby's own implementation. release.yml runs it next to the existing C extension smoke test, on the Linux archive. CI's jobs do not build the C API's native shim, which this needs.
nokogiri's extconf.rb compiles its own libxml2 and libxslt through mini_portile,
and gives their configure `--host` from RbConfig. It reads
`CONFIG["host_alias"].empty?`, which raised NoMethodError on IronRuby: CONFIG
had no host, host_alias, build or target at all. Build scripts read more of what
CRuby's configure records, so these are filled in, with CRuby's values for the
same platform:
- host, build, target and their _cpu, _os, _vendor ("pc") and _alias (empty,
since IronRuby is not cross-built). For example host "x86_64-pc-linux".
- NM, RANLIB, LD, OBJDUMP, OBJCOPY, PKG_CONFIG, GNU_LD, LN_S, GREP, EGREP and
SOEXT: the tools configure found, by name.
- datarootdir, includedir, docdir, mandir, sysconfdir, localstatedir and the
other autoconf directories, under prefix. platform is RUBY_PLATFORM.
With this, `gem install nokogiri --platform ruby` (IRONRUBY_NATIVE_GEMS=nokogiri)
builds the extension with its packaged libxml2, libxslt and gumbo. A script
exercising it gives the same output as on CRuby 4.0: HTML, HTML5, XML with
namespaces, Builder, fragments, SAX, XSLT, parse errors, and 2000 parses
through a GC. spec/library/rbconfig and RbConfig_ExecutableExts1 pass.
`class X; def object_id = 1; private :object_id; end` warned "redefining 'object_id' may cause serious problems" twice. Once is right, for the def. The second came from private: IronRuby copies the entry with the new visibility, and that copy went through the new-entry check. MRI changes an own entry's visibility in place, without a new method entry, so it does not warn. Making an inherited method private does add an entry in the class, and still warns, as it does in MRI. Found by CodeRabbit on #251.
…flag The flag orders only a key's first store. Once it is set, a reader could see a later writer's value or pointer, stored relaxed, without the writes the writer made before it. For a pointer, those writes are the pointee's initialization. Each store of the value and the pointer is now a release, and each load an acquire. Found by CodeRabbit on #251.
C API review fixes: core method warnings, Ractor storage, hash seed, selftest
Nineteen more C extension gems were installed with gem install --platform ruby and compared with CRuby 4.0 by a script exercising each, and the real nokogiri with IRONRUBY_NATIVE_GEMS once RbConfig had what its extconf.rb reads. The README's list of gems known to work now has them, and says why digest-crc runs its Ruby code: its Rakefile builds the extensions on RUBY_ENGINE == "ruby" only.
RbConfig: triplets, binutils and install dirs (nokogiri builds)
Hi, I updated IronRuby to be Ruby 4.0 compatible. It's using prism now. I managed to create working dlls as well. I don't expect you to merge this huge change, but I would like to make you aware, that there is a working ruby 4.0 version.