Repository navigation
Conversation
Consensus tells the view changer the view it starts in, which marks the view as entered, so the node doesn't act on a new view message for it again after a restart. Restoring a view change still moves to the next view. The Restore field of the view changer is documented. Part of hyperledger#765. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Hagar Meir <hagar.meir@ibm.com>
|
I have a little doubt.
|
The view changer's run loop picked between informChan and incMsgs at random, so a new view message that arrived right after a restart could be processed before the start view was marked as entered. Consensus now sets StartViewEntered before Start, which marks the view before any message is processed. The test also sends the new view message again right after the restart, waits for the view start before the restart so that it is not counted after it, and checks that the node doesn't act on the message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Hagar Meir <hagar.meir@ibm.com>
|
Thanks, you were right on both points.
(Drafted with Claude's help.) |
| // The node starts in a view it already entered, so it must not act on a new view message for it again. | ||
| // A restored view change instead moves to the next view, and waits for a new view message for it. | ||
| c.viewChanger.StartViewEntered = c.viewChanger.Restore == nil | ||
| c.viewChanger.Start(view) |
There was a problem hiding this comment.
Should the parameter be passed to the function call? Otherwise, now it's all done on the discipline of the calling code. This contract can be easily violated in the future.
There was a problem hiding this comment.
Good point. It's now a parameter, Start(startViewNumber, startViewEntered bool), so every caller has to decide, and the field is gone. Consensus passes Restore == nil. The unit tests pass false, which keeps what they test: a new view message for the view they start in. (Drafted with Claude's help.)
Every caller of Start now says whether the node already entered the view it starts in, instead of relying on a field being set before. Consensus passes Restore == nil, and the unit tests pass false, which keeps what they test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Hagar Meir <hagar.meir@ibm.com>
| c.viewChanger.Start(view) | ||
| // Unless the node restores a view change, it starts in a view it already entered, | ||
| // so it must not act on a new view message for it again. | ||
| c.viewChanger.Start(view, c.viewChanger.Restore == nil) |
There was a problem hiding this comment.
When the node restores a view change, it still starts in the view it already entered and then left, and acceptedNewView stays false there until the restore runs. setViewAndSeq sets the start view to the ViewChange record's NextView, which holds the view being left, not the next one. run() picks at random among ready select cases, so a NewView for that old view that was queued in incMsgs while ControllerStartedWG was blocking can be handled before <-v.Restore. The node then goes through processNewViewMsg and Controller.ViewChanged(v) for a view whose view data it already sent for v+1. Passing true every time looks safe: processViewChangeMsg(true) clears the flag when it moves to v+1. The other option is to handle Restore before anything else in run().
| v.MetricsViewChange.CurrentView.Set(float64(v.currView)) | ||
| v.MetricsViewChange.RealView.Set(float64(v.realView)) | ||
| v.MetricsViewChange.NextView.Set(float64(v.nextView)) | ||
| v.acceptedNewView = startViewEntered |
There was a problem hiding this comment.
Nothing persists the flag, so a restart can bring it back as false even though the node already acted on a NewView. processNewViewMsg sets acceptedNewView = true before commitInFlightProposal. If the in-flight commit times out, or the node crashes before the NewView record is saved, the last WAL entry is still the ViewChange record. After a restart, Restore is non-nil, the restore moves the node to v+1 with acceptedNewView=false, and it can act on a second, different NewView for v+1 from a faulty leader. Inferring the flag from Restore == nil covers only the case where a NewView record was written. Persisting the acceptance in the WAL when the flag is set would fix this at the root.
| c.viewChanger.Start(view) | ||
| // Unless the node restores a view change, it starts in a view it already entered, | ||
| // so it must not act on a new view message for it again. | ||
| c.viewChanger.Start(view, c.viewChanger.Restore == nil) |
There was a problem hiding this comment.
On reconfig, Restore, and so startViewEntered, comes from stale state. c.state.Entries is still WALInitialContent from startup, because PersistedState.Save never updates it. setViewAndSeq in reconfig() therefore looks at the WAL as it was at startup, not now. If startup ended with a ViewChange record whose NextView is at least the checkpoint view, every later reconfig re-arms Restore and starts with startViewEntered=false. A node that is in the middle of a view change at reconfig time gets Restore == nil and true instead.
| func (v *ViewChanger) Start(startViewNumber uint64) { | ||
| // Start starts the view changer in the given view. startViewEntered says whether the node already entered that view, | ||
| // so that it acts on no new view message for it. It is set before any message is processed. A node that restores | ||
| // a view change didn't enter the view it starts in: the restore moves it to the next view, to wait for a new view |
There was a problem hiding this comment.
This comment is inaccurate. A node that restores a view change starts in the view named by the ViewChange record, which is the view it was in and then left (NextView: v.currView is saved before currView is incremented). So it did enter that view. Only the next view, which the restore moves it to, is one it hasn't entered. As written, the comment argues for false in exactly the case where true is the safe value.
| // so that it acts on no new view message for it. It is set before any message is processed. A node that restores | ||
| // a view change didn't enter the view it starts in: the restore moves it to the next view, to wait for a new view | ||
| // message for it. | ||
| func (v *ViewChanger) Start(startViewNumber uint64, startViewEntered bool) { |
There was a problem hiding this comment.
The new parameter can be derived from what the view changer already has: the only production caller passes c.viewChanger.Restore == nil, and v.Restore is visible in Start. Setting the flag inside Start (or just v.acceptedNewView = true, since the restore path resets it) would keep the signature as it was, avoid touching about 30 test call sites, and keep a future caller from passing a wrong value.
| } | ||
|
|
||
| vc.Start(0) | ||
| vc.Start(0, false) |
There was a problem hiding this comment.
Every unit test now calls Start(..., false), while production passes true unless it restores. So the internal/bft tests run a configuration production almost never uses, and no unit test checks that Start(v, true) drops a NewView for v, or that a restore with true still accepts the NewView for v+1. The integration test also covers only the path that doesn't restore.
| MetricsViewChange *api.MetricsViewChange | ||
| MetricsBlacklist *api.MetricsBlacklist | ||
| MetricsView *api.MetricsView | ||
| // Restore, when it holds a token, makes the view changer restore a view change once it starts. |
There was a problem hiding this comment.
The doc says Consensus sets Restore when the WAL ends with a view change record. setViewAndSeq sets it only when viewChange.GetNextView() >= view; an older record is ignored. Start now depends on this field, so the doc should state the condition.
| return ignored.Load() > 0 | ||
| }, 10*time.Second, 100*time.Millisecond, "the node did not ignore the new view message for the view it restarted in") | ||
| require.Zero(t, viewChanges.Load(), "the node acted on the new view message after it restarted in the view") | ||
| require.Equal(t, int32(1), starts.Load(), "the node started the view again after it restarted in it") |
There was a problem hiding this comment.
This assertion may be flaky. The substring "Starting view with number 1" also matches views 10 to 19. And a legitimate restart of the view after the restart (for example one triggered by a sync when the node falls behind on heartbeats, in the waitForTheView=false case) makes starts equal 2 even if the NewView was ignored. Matching "Starting view with number 1," and checking that the count did not grow after the message was handled would be more robust.
Written by Claude (Anthropic's AI assistant) on behalf of Hagar Meir.
Part of #765.
Consensus.startComponentstellsViewChanger.Startwhether the node already entered the view it starts in. It did, unless it restores a view change.Startmarks the view as entered before it processes any message, so the node doesn't act on a new view message for that view again after a restart. Restoring a view change still moves to the next view and waits for a new view message for it, as before.Restorefield of the view changer is documented: it makes a node restore a view change it may have been in the middle of, instead of starting in a view it entered.Tests:
TestNewViewIgnoredAfterRestart: the leader goes away, the others change the view, and node 4 restarts in view 1. Node 4 is then sent the new view message for view 1 again, either once it is back in the view or right after the restart returns. It ignores the message, doesn't act on it (noViewChanged), and doesn't start the view again. Onmain, it acts on the message.🤖 Generated with Claude Code