Skip to content

Do not act on a new view message for the view a node restarts in - #771

Open
HagarMeir wants to merge 3 commits into
hyperledger:mainfrom
HagarMeir:new-view-after-restart
Open

HagarMeir wants to merge 3 commits into
hyperledger:mainfrom
HagarMeir:new-view-after-restart

Conversation

@HagarMeir

@HagarMeir HagarMeir commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Written by Claude (Anthropic's AI assistant) on behalf of Hagar Meir.

Part of #765.

  • Consensus.startComponents tells ViewChanger.Start whether the node already entered the view it starts in. It did, unless it restores a view change. Start marks the view as entered before it processes any message, so the node doesn't act on a new view message for that view again after a restart. Restoring a view change still moves to the next view and waits for a new view message for it, as before.
  • The Restore field of the view changer is documented: it makes a node restore a view change it may have been in the middle of, instead of starting in a view it entered.

Tests:

  • TestNewViewIgnoredAfterRestart: the leader goes away, the others change the view, and node 4 restarts in view 1. Node 4 is then sent the new view message for view 1 again, either once it is back in the view or right after the restart returns. It ignores the message, doesn't act on it (no ViewChanged), and doesn't start the view again. On main, it acts on the message.

🤖 Generated with Claude Code

Consensus tells the view changer the view it starts in, which marks the
view as entered, so the node doesn't act on a new view message for it
again after a restart. Restoring a view change still moves to the next
view. The Restore field of the view changer is documented.

Part of hyperledger#765.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Hagar Meir <hagar.meir@ibm.com>
@pfi79

pfi79 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

I have a little doubt.

  1. Is a synchronous guarantee needed at startup? If, after a restart, a new-view message can be delivered or processed before run() actually enters its working loop, then the current approach (enqueueing into informChan) does not guarantee the protection. It would be more reliable to set the "already acted on the startup view" flag in the view changer state before messages for that view can be processed (taking into account that run() may still be waiting on ControllerStartedWG), or to guarantee that informNewView is processed before incMsgs in this case.

  2. Reproducibility. It is worth running TestNewViewIgnoredAfterRestart many times with the race detector (-race, a high -count) and considering a variant where the repeated new-view message is injected as early as possible after the restart: that is exactly the area where the non-determinism of select may show up.

The view changer's run loop picked between informChan and incMsgs at
random, so a new view message that arrived right after a restart could
be processed before the start view was marked as entered. Consensus now
sets StartViewEntered before Start, which marks the view before any
message is processed.

The test also sends the new view message again right after the restart,
waits for the view start before the restart so that it is not counted
after it, and checks that the node doesn't act on the message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Hagar Meir <hagar.meir@ibm.com>
@HagarMeir

Copy link
Copy Markdown
Contributor Author

Thanks, you were right on both points.

  1. The informChan approach wasn't guaranteed: run() picks between informChan and incMsgs at random when both are ready. The view changer now has a StartViewEntered field that Consensus sets before Start, next to Restore. Start copies it into acceptedNewView before it starts run(), so it's set before any message is processed.
  2. TestNewViewIgnoredAfterRestart now has a subtest that sends the new view message again right after the restart returns. With the informChan approach, that subtest failed 14 of 40 runs. With the fix, both subtests passed 60 of 60 under -race. The stress run also showed a race in the test itself: the view start from before the restart could be counted as one after it. That's fixed too, and the test now also checks that the node doesn't call ViewChanged for the view after the restart.

(Drafted with Claude's help.)

Comment thread pkg/consensus/consensus.go Outdated
Comment on lines 545 to 548
// The node starts in a view it already entered, so it must not act on a new view message for it again.
// A restored view change instead moves to the next view, and waits for a new view message for it.
c.viewChanger.StartViewEntered = c.viewChanger.Restore == nil
c.viewChanger.Start(view)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the parameter be passed to the function call? Otherwise, now it's all done on the discipline of the calling code. This contract can be easily violated in the future.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. It's now a parameter, Start(startViewNumber, startViewEntered bool), so every caller has to decide, and the field is gone. Consensus passes Restore == nil. The unit tests pass false, which keeps what they test: a new view message for the view they start in. (Drafted with Claude's help.)

Every caller of Start now says whether the node already entered the
view it starts in, instead of relying on a field being set before.
Consensus passes Restore == nil, and the unit tests pass false, which
keeps what they test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Hagar Meir <hagar.meir@ibm.com>
c.viewChanger.Start(view)
// Unless the node restores a view change, it starts in a view it already entered,
// so it must not act on a new view message for it again.
c.viewChanger.Start(view, c.viewChanger.Restore == nil)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the node restores a view change, it still starts in the view it already entered and then left, and acceptedNewView stays false there until the restore runs. setViewAndSeq sets the start view to the ViewChange record's NextView, which holds the view being left, not the next one. run() picks at random among ready select cases, so a NewView for that old view that was queued in incMsgs while ControllerStartedWG was blocking can be handled before <-v.Restore. The node then goes through processNewViewMsg and Controller.ViewChanged(v) for a view whose view data it already sent for v+1. Passing true every time looks safe: processViewChangeMsg(true) clears the flag when it moves to v+1. The other option is to handle Restore before anything else in run().

v.MetricsViewChange.CurrentView.Set(float64(v.currView))
v.MetricsViewChange.RealView.Set(float64(v.realView))
v.MetricsViewChange.NextView.Set(float64(v.nextView))
v.acceptedNewView = startViewEntered

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing persists the flag, so a restart can bring it back as false even though the node already acted on a NewView. processNewViewMsg sets acceptedNewView = true before commitInFlightProposal. If the in-flight commit times out, or the node crashes before the NewView record is saved, the last WAL entry is still the ViewChange record. After a restart, Restore is non-nil, the restore moves the node to v+1 with acceptedNewView=false, and it can act on a second, different NewView for v+1 from a faulty leader. Inferring the flag from Restore == nil covers only the case where a NewView record was written. Persisting the acceptance in the WAL when the flag is set would fix this at the root.

c.viewChanger.Start(view)
// Unless the node restores a view change, it starts in a view it already entered,
// so it must not act on a new view message for it again.
c.viewChanger.Start(view, c.viewChanger.Restore == nil)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On reconfig, Restore, and so startViewEntered, comes from stale state. c.state.Entries is still WALInitialContent from startup, because PersistedState.Save never updates it. setViewAndSeq in reconfig() therefore looks at the WAL as it was at startup, not now. If startup ended with a ViewChange record whose NextView is at least the checkpoint view, every later reconfig re-arms Restore and starts with startViewEntered=false. A node that is in the middle of a view change at reconfig time gets Restore == nil and true instead.

func (v *ViewChanger) Start(startViewNumber uint64) {
// Start starts the view changer in the given view. startViewEntered says whether the node already entered that view,
// so that it acts on no new view message for it. It is set before any message is processed. A node that restores
// a view change didn't enter the view it starts in: the restore moves it to the next view, to wait for a new view

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is inaccurate. A node that restores a view change starts in the view named by the ViewChange record, which is the view it was in and then left (NextView: v.currView is saved before currView is incremented). So it did enter that view. Only the next view, which the restore moves it to, is one it hasn't entered. As written, the comment argues for false in exactly the case where true is the safe value.

// so that it acts on no new view message for it. It is set before any message is processed. A node that restores
// a view change didn't enter the view it starts in: the restore moves it to the next view, to wait for a new view
// message for it.
func (v *ViewChanger) Start(startViewNumber uint64, startViewEntered bool) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new parameter can be derived from what the view changer already has: the only production caller passes c.viewChanger.Restore == nil, and v.Restore is visible in Start. Setting the flag inside Start (or just v.acceptedNewView = true, since the restore path resets it) would keep the signature as it was, avoid touching about 30 test call sites, and keep a future caller from passing a wrong value.

}

vc.Start(0)
vc.Start(0, false)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every unit test now calls Start(..., false), while production passes true unless it restores. So the internal/bft tests run a configuration production almost never uses, and no unit test checks that Start(v, true) drops a NewView for v, or that a restore with true still accepts the NewView for v+1. The integration test also covers only the path that doesn't restore.

MetricsViewChange *api.MetricsViewChange
MetricsBlacklist *api.MetricsBlacklist
MetricsView *api.MetricsView
// Restore, when it holds a token, makes the view changer restore a view change once it starts.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc says Consensus sets Restore when the WAL ends with a view change record. setViewAndSeq sets it only when viewChange.GetNextView() >= view; an older record is ignored. Start now depends on this field, so the doc should state the condition.

return ignored.Load() > 0
}, 10*time.Second, 100*time.Millisecond, "the node did not ignore the new view message for the view it restarted in")
require.Zero(t, viewChanges.Load(), "the node acted on the new view message after it restarted in the view")
require.Equal(t, int32(1), starts.Load(), "the node started the view again after it restarted in it")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assertion may be flaky. The substring "Starting view with number 1" also matches views 10 to 19. And a legitimate restart of the view after the restart (for example one triggered by a sync when the node falls behind on heartbeats, in the waitForTheView=false case) makes starts equal 2 even if the NewView was ignored. Matching "Starting view with number 1," and checking that the count did not grow after the message was handled would be more robust.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants