Skip to content

ci: measure coverage at the PR head that Codecov receives - #27205

Closed
alamb wants to merge 2 commits into
apache:mainfrom
alamb:alamb/coverage-head-sha
Closed

alamb wants to merge 2 commits into
apache:mainfrom
alamb:alamb/coverage-head-sha

Conversation

@alamb

@alamb alamb commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The Codecov reports on our PRs are unreliable: they regularly mark comments and blank lines as "Missing" and show unexecuted code as covered (exmaples below)

The problem is that the coverage job compares the PR to the base branch (e.g. main), but Codecov attributes the report to the PR head commit

Examples of Codecov results that refer to lines not in the PR

For each PR below I took Codecov's per-line coverage for the head commit (via the Codecov API), the PR diff (merge-base..head) and the merge tree the job actually built (git merge-tree of the head and the main tip at the time the run was created).

#25455 (datafusion/optimizer/src/push_down_filter.rs): the Codecov comment reports 70.73% | 15 Missing and 9 partials.

See the report here.

Among the lines it marks as Missing are L4593-L4594, L4613-L4615, which are a /// doc comment, a blank line and two // comments:

4593:    /// A filter is not moved below a pure extraction projection, even when its
4594:    /// predicate only references pass-through columns.
...
4613:
4614:        // `b` is a plain pass-through column, so without the precedence rule
4615:        // this predicate would reach the scan as a `full_filters` entry.

Out of 243 PRs merged to main between 2026-09-25 and 2026-10-20 that touch Rust files, 26 had at least one changed file whose coverage was read at shifted line numbers.

Why the line numbers are wrong, and which PRs are affected today

For a pull_request run, actions/checkout checks out the merge of the PR head into the base branch, but codecov-action uploads the report under the PR head SHA. Whenever the base branch has commits the PR lacks, Codecov reads the merge tree's coverage at the head's line numbers.

Today this affects pull_request runs in the apache repo: PRs to branch-* (backports) and non-fork PRs to main. Fork PRs to main run in the fork on push since #25591, where the head is already checked out, so they are not affected. Before #25591 every PR was affected, which is where the examples above come from.

What changes are included in this PR?

Update coverage job to check out github.event.pull_request.head.sha on pull_request runs, and github.sha otherwise (merge queue, pushes to branch-*).

The trade-off is the same as in the sqlparser PR: the cargo test (amd64) job now tests the PR head rather than the merge with the base branch; I actually think this is more intuitive as it is the same commit that gets checked out locally.

Note that the merge queue still tests the merged result before anything lands on main.

What is the testing strategy for this PR?

CI on this PR. The Codecov comment on this PR cannot show the difference since it changes no Rust code; the examples above were verified against Codecov's per-line data and the reconstructed merge trees.

Are there any user-facing changes?

No.

The coverage job checks out the merge of the PR head into the base
branch, but codecov-action uploads the report under the PR head SHA.
When the base branch has commits the PR lacks, Codecov reads the merge
tree's coverage at the head's line numbers and flags the wrong lines.

Same change as apache/datafusion-sqlparser-rs#2832.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions github-actions Bot added the development-process Related to development process of DataFusion label Nov 7, 2026
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@alamb
alamb marked this pull request as ready for review November 7, 2026 21:18
@alamb

alamb commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

I can't make sense of the codecov report. Closing for now

@alamb alamb closed this Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

development-process Related to development process of DataFusion

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant