Skip to content

help: advertise the vfs-sparse-index capability in build options - #1026

Draft
Tyrie Vella (tyrielv) wants to merge 5 commits into
microsoft:vfs-2.56.1from
tyrielv:tyrielv/sparse-index/vfs-capability
Draft

help: advertise the vfs-sparse-index capability in build options#1026
Tyrie Vella (tyrielv) wants to merge 5 commits into
microsoft:vfs-2.56.1from
tyrielv:tyrielv/sparse-index/vfs-capability

Conversation

@tyrielv

@tyrielv Tyrie Vella (tyrielv) commented Nov 14, 2026 •
edited
Loading

Copy link
Copy Markdown

Important

Merge order: this PR must merge after #1024 and #1025. It cannot be published for review until both merge.

Review only the last commit (help: advertise the vfs-sparse-index capability in build options). The first four commits come from #1025 (three commits) and #1024 (one commit). They are here only because Git PRs do not stack. After both PRs merge, this branch will be rebased so that it contains only the last commit.

Problem and Context

A virtual file system such as VFS for Git can use a sparse index only when the Git that it runs supports a sparse index under core.virtualFilesystem. A Git without this support does not fail. It expands the sparse index to a full index on each read, and the index stays full. The virtual file system cannot see this, so it must ask Git before it enables a sparse index.

A version floor cannot answer the question today, because no released Git contains the support. Also, a behavior probe needs a scratch repository and several processes. One line in git version --build-options answers the question with one process.

The line is a promise. It is true only with the behavior of #1024 and #1025. The commit message lists each part of that behavior, so that a later change cannot remove part of it without noticing that the line becomes false.

Changes

  • help.c: get_version_info() prints feature: vfs-sparse-index in git version --build-options, after feature: fsmonitor--daemon. The line is unconditional, because the support is always compiled in and configuration controls it at run time. A comment next to the line lists each part of the promise.
  • Documentation/config/core.adoc: the description of core.virtualFilesystem names the line.
  • Add t/t1097-vfs-sparse-index-capability.sh, and register it in t/meson.build. Its tests check that git version --build-options and git --version --build-options print the line, and that git version without the option does not print it. Its description names t1094, t1095, and t1096, which test the promised behavior.
  • t/meson.build in the combined branch: sparse-index: do not expand the index during the working-tree walk under a virtual filesystem #1024 adds t1094 and Support a cone-mode sparse index under a virtual file system #1025 adds t1095 and t1096 at the same place. The PR that merges second must keep all three entries in numeric order.

Known gaps and deliberate decisions

  • Sparse-directory root with a different case on disk, without a virtual file system. With core.ignoreCase, if the root of a sparse directory has a different case on disk than in the index (OUT/ for out/), a plain sparse index with sparse.expectFilesOutsideOfPatterns reports the tracked files inside it as untracked, and git clean -d offers to remove them. Under a virtual file system, sparse-index: do not expand the index during the working-tree walk under a virtual filesystem #1024 handles this case, and the line covers only that configuration. The plain sparse-index case comes from upstream code and is not changed here.
  • git sparse-checkout clean is not part of the promise. Only the automatic cleanup after set, add, and reapply skips the out-of-cone directories. The explicit clean command still deletes them. A virtual file system that owns those directories must block or guard that command itself.
  • No test ties the line to the behavior. t1097 checks only the output. t1094, t1095, and t1096 test the behavior. The comment next to the line and the commit message list each part of the promise, so that a change that removes a part also removes the line.

@tyrielv
sparse-index: let the name-hash find entries that expand_index() reuses
fe3f397 
expand_index() builds a new cache array that reuses the cache entries of
the sparse index. It then discards the name-hash of the index and leaves
it uninitialized. The reused entries keep the CE_HASHED flag, so the next
lazy_init_name_hash() skips them: hash_index_entry() returns early for an
entry that has CE_HASHED. After the expansion, index_file_exists() and
index_dir_find() cannot find these entries, and the dir_hash does not
contain their directories.

This happens when something initializes the name-hash before the index
expands. With core.ignoreCase, the untracked-file walk of "git status"
does this. For example, take a cone that contains "zin/", with
sparse.expectFilesOutsideOfPatterns set and an untracked file in the
out-of-cone directory "out/". The walk initializes the name-hash, then
the lookup of "out/.gitignore" expands the index. Then the walk cannot
find "zin/" in the dir_hash and reports this tracked directory as
untracked:

	?? out/new
	?? zin/

Clear CE_HASHED on each entry that expand_index() reuses, so that the
next lazy_init_name_hash() adds it to the new name-hash.

The test for the "git status" case also exposes a leak in the same
code. expand_index() discards the dir_hash with hashmap_clear(), which
frees the table but not the dir_entry structs, so they leak whenever
the name-hash was initialized with core.ignoreCase before the
expansion. Use free_name_hash(), which also frees the dir_entry
structs. Without this, the new test fails under the leak checker.

Add a "--expand-and-find=<path>" option to "test-tool read-cache". It
looks up a path in the name-hash, expands the index to a full index, and
looks up the path again. Add a t1092 test that uses it, and a t1092 test
for the "git status" case above. The order in which readdir() returns
the directories can hide the "git status" case on some platforms, so the
test-tool test is the deterministic check.

Assisted-by: Claude Opus 5.5
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv
virtualfilesystem: do not expand a sparse index
828fa67 
apply_virtualfilesystem() runs on every index read. It sets
CE_SKIP_WORKTREE on every cache entry, then clears the bit for each
path that the virtual file system hook reports. To find these paths,
clear_ce_flags_virtualfilesystem_1() uses index_name_pos(), or the
name-hash helpers when core.ignoreCase is set.

In a sparse index, a reported path can be inside a sparse-directory
entry. Such a path has no cache entry of its own, and index_name_pos()
then expands the whole index to a full index. Because this happens on
every index read, a virtual file system repository can never keep a
sparse index. A directory wildcard (for example "dir/") has a second
problem: the loop that clears the bit on all entries below the
directory also clears it on a nested sparse-directory entry.

Use index_name_pos_sparse() instead. It does not expand the index, so a
path inside a sparse-directory entry is not found, and the
sparse-directory entry keeps its CE_SKIP_WORKTREE bit. This is correct:
the contents of the sparse directory are outside of the sparse-checkout
cone. With a full index, index_name_pos_sparse() and index_name_pos()
are the same, so a full index behaves as before. The core.ignoreCase
lookups do not change: the name-hash does not contain sparse-directory
entries, so they do not expand the index. In the directory wildcard
loop, skip sparse-directory entries.

apply_virtualfilesystem() runs only when core.virtualFilesystem is set,
so a repository without a virtual file system behaves as before.

A virtual file system that uses a sparse index must keep modified
paths inside the cone. Git does not look at the working tree for a
path inside a sparse-directory entry.

Add t1095. The tests use the trace2 "ensure_full_index" and
"expand_index" regions and the "expansion-reason" data to assert that
the index does not expand. They check the skip-worktree bits with
"git ls-files -t --sparse" for reported paths inside sparse
directories, for a directory wildcard that contains a nested sparse
directory, and for case-insensitive matches of in-cone paths. They
also check that "git status" reports an in-cone change, and that a
full index keeps the original behavior.

Assisted-by: Claude Opus 5.5
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv
dir.c: consult cone patterns for sparse-checkout under a virtual file…
4c6b9eb 
…system

With core.virtualFilesystem, path_in_sparse_checkout_1() reports every
path as inside the sparse-checkout. convert_to_sparse() collapses a
directory only when path_in_sparse_checkout() reports it as outside, so
an out-of-cone directory never collapses into a sparse-directory entry,
and a virtual file system repository cannot have a sparse index.

When the repository uses cone-mode sparse-checkout and sets index.sparse
to true in its configuration, let the cone patterns decide. The new
helper vfs_sparse_index_configured() checks the configuration, and
vfs_uses_cone_patterns() also checks the pattern file. The default value
of index.sparse is true, but it does not count here: a virtual file
system must turn this on explicitly, because a virtual file system that
cannot read a sparse index must never get one. These helpers run for
each path, so they do not use is_sparse_index_allowed(istate, 0), which
reads the environment and can write index.sparse when
GIT_TEST_SPARSE_INDEX is set.

 - path_in_sparse_checkout_1() no longer returns early under a virtual
   file system when vfs_uses_cone_patterns() is true.

 - path_matches_pattern_list() skips its virtual file system block for
   a cone-mode pattern list when vfs_sparse_index_configured() is true.
   The list is already in cone mode, so this checks only the
   configuration and does not load patterns into the temporary index
   of expand_index(). For a directory that the virtual file
   system does not report, that block returns 1. For the cone patterns,
   1 means "in the cone", so such a directory would still never
   collapse. Pattern lists that do not use cone mode keep the virtual
   file system block. The path-walk API, for example in "git backfill
   --sparse", also uses a cone-mode pattern list. Under a virtual file
   system, it now also matches only the patterns, as it does without a
   virtual file system.
   This change also fixes "git sparse-checkout reapply" on a collapsed
   index, which died with "is_excluded_from_virtualfilesystem passed
   unhandled dtype" when expand_index() passed a sparse directory with
   DT_UNKNOWN to that block.

 - do_read_index() no longer collapses a full index under a virtual
   file system. It calls ensure_correct_sparsity() before
   post_read_index_from() calls apply_virtualfilesystem(), so
   convert_to_sparse() would use the skip-worktree bits that are stored
   in the index file. Under a virtual file system these bits are stale:
   the hook can now report a path that had the bit when the index was
   written. Then the directory collapses, and apply_virtualfilesystem()
   cannot clear the bit of a path inside a sparse directory. The index
   collapses when it is written, after the bits are correct.

 - clean_tracked_sparse_directories() returns early under a virtual
   file system. After "git sparse-checkout set", "add", or "reapply",
   it deletes each out-of-cone directory that has no untracked files.
   A virtual file system shows these directories in the working tree,
   and the virtual file system owns them. Before this change, nothing
   collapsed under a virtual file system, so this function deleted
   nothing.

All new behavior requires core.virtualFilesystem. The cone patterns
are used only when the repository also uses cone mode and sets
index.sparse to true. Without a virtual file system, nothing changes. A
virtual file system repository without these settings also behaves as
before. Update the description of core.virtualFilesystem to match.

Add t1096. The out-of-cone files are on disk, as a virtual file system
shows them. The tests check that nothing collapses with index.sparse
false or with the default value of index.sparse, and that the
out-of-cone directory collapses with index.sparse true. They check that
"git status" reports an in-cone change without an expansion, and that
"git sparse-checkout add" and "reapply" keep all files on disk. They
check that a directory with a reported path does not collapse, that
non-cone mode under a virtual file system does not change, and that
cone mode without a virtual file system does not change.

Assisted-by: Claude Opus 5.5
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv
Tyrie Vella (tyrielv) force-pushed the tyrielv/sparse-index/vfs-capability branch 2 times, most recently from c61714a to 7fb7378 Compare November 14, 2026 08:28
@tyrielv
sparse-index: do not expand the index in the working-tree walk under VFS
4158772 
A sparse index collapses each directory outside the sparse-checkout cone
into one sparse-directory entry. Several lookups in the working-tree walk
use index_name_pos(). When the path is inside a sparse-directory entry,
index_name_pos() expands the whole index to answer a question about one
path.

A plain sparse checkout does not get to these lookups. The files outside
the cone are not on disk, so the walk does not go into those directories.
Under core.virtualFileSystem, the working tree shows the full tree while
the index stays sparse, and the walk goes into them. In a repository with
about 2.2 million index entries, an earlier version of this change made
"git status" go from about 20 seconds to less than half a second. When
the command writes the index, it expands the index a second time
("re-expanding after write").

Five lookups are involved. read_skip_worktree_file_from_index() reads a
.gitignore from the index when the file is not on disk. add_patterns()
looks up the object ID of a .gitignore on disk for the untracked cache.
directory_exists_in_index() decides if a directory is tracked.
get_index_dtype() gives the type of a path to resolve_dtype() when the
caller does not know it. index_name_is_other() removes tracked paths
from the untracked list in "git status" and "git clean".

Add vfs_sparse_dir_containing(). If core.virtualFileSystem is set, the
index is sparse, and a sparse-directory entry contains the path, it
returns that entry without an expansion. Otherwise it returns NULL. Each
of the five lookups calls it first. Under core.virtualFileSystem, a NULL
result means that index_name_pos() cannot expand the index for that
path, so the existing code runs. Without core.virtualFileSystem, the
helper always returns NULL, and the behavior does not change.

When a sparse-directory entry contains the path, each lookup gives the
answer of the full index. Everything inside a sparse-directory entry has
the skip-worktree bit, so read_skip_worktree_file_from_index() reads the
blob from the tree of that entry. The untracked-cache lookup computes the
object ID from the file on disk, as it does for any path that has no
usable index entry. directory_exists_in_index() and
index_name_is_other() look up the path in the tree. An index has entries
for files only, so index_name_is_other() reports a directory in the tree
as "other". directory_exists_in_index() does its lookup before the
case-insensitive lookup, because the name hash does not contain the
directories inside a sparse-directory entry. get_index_dtype() returns
DT_UNKNOWN, and the caller uses lstat(). An index that was just expanded
returns DT_UNKNOWN too, because expansion does not mark the new entries
up to date.

Add sparse_dir_get_tree_entry() for the tree lookups. If core.ignoreCase
is set, directory_exists_in_index() and index_name_is_other() compare
names without case, because with a full index the walk finds tracked
paths with case-insensitive lookups. If a tree cannot be read, the
helper dies. A failure to read must not make a tracked path look
untracked, because "git clean" could then delete it.

For these two lookups, vfs_sparse_dir_containing() also finds the
sparse-directory entry when its name has a different case on disk, for
example "OUT/a" for the entry "out/". The index sorts names with case,
so such a path does not sort next to its sparse-directory entry, and
the name hash does not contain the paths inside the entry. Without this,
"git status" reports the tracked files below "OUT/" as untracked, and
"git clean -f -d" deletes them. The helper looks up each leading
directory of the path in the case-insensitive dir_hash, which contains
the directory of each sparse-directory entry, and checks if the name
that the index uses for it is a sparse-directory entry. In a test
repository with about 41,000 paths on disk, this made the walk of "git
status -uall" with core.ignoreCase take 0.245 seconds instead of 0.22.
The other three lookups compare with case, as index_name_pos() does.

Add t1094, which puts files on disk inside a sparse directory and uses a
virtual file system hook. The tests compare "git status", "git clean
-n", and "git add --ignore-missing" with the results in a clone that has
a full index. They also use trace2 to make sure that the index does not
expand and stays sparse. Other tests check directories spelled with a
different case on disk (inside a sparse directory, the sparse directory
itself, and a nested sparse directory), and an unreadable tree. A last
test shows that without a virtual file system, the walk still expands
the index.

Assisted-by: Claude Opus 5.5
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv
help: advertise the vfs-sparse-index capability in build options
ea4666d 
A virtual file system such as VFS for Git can use a sparse index only
when the Git that it runs supports a sparse index under
core.virtualFilesystem. A Git without this support does not fail. It
expands the sparse index to a full index on each read, and the index
stays full. The virtual file system cannot see this, so it must ask Git
before it enables a sparse index.

Print "feature: vfs-sparse-index" in "git version --build-options". A
virtual file system can run this command once and read the line. It
must not enable a sparse index when the line is not present. A Git
without this commit does not print the line, so the check fails safe.
The support is always compiled in, and configuration controls it at
run time, so the line is unconditional.

The line is a promise. When core.virtualFilesystem is set, a build that
prints the line has all of the behavior that follows. The working-tree
walk does not expand a sparse index: its lookups for .gitignore files,
tracked directories, path types, and untracked paths find the
sparse-directory entry that contains the path, and answer without an
expansion. apply_virtualfilesystem() does not expand a sparse index,
and a path that the hook reports inside a sparse-directory entry does
not clear the skip-worktree bit of that entry. When expand_index()
expands the index, the name-hash still finds the entries that it
reuses.

When the repository uses cone-mode sparse-checkout and sets
index.sparse to true in its configuration, the cone patterns decide
which directories collapse into sparse-directory entries. The default
value of index.sparse does not count, so the virtual file system must
turn this on explicitly. A full index does not collapse when Git reads
it, because the skip-worktree bits are not correct until
apply_virtualfilesystem() runs. The index collapses when Git writes it.
Last, the automatic cleanup after "git sparse-checkout set", "add", and
"reapply" does not delete the out-of-cone directories, because the
virtual file system owns them. The explicit "git sparse-checkout clean"
command is not part of the promise.

If a later change removes any part of this behavior, it must also
remove the line. Otherwise a virtual file system will enable a sparse
index that Git does not handle correctly. The comment next to the line
lists the same parts. Also name the line in the description of
core.virtualFilesystem.

Add t1097. Its tests check that "git version --build-options" and
"git --version --build-options" print the line, and that "git version"
without the option does not print it. Its description names t1094,
t1095, and t1096, which test the promised behavior.

Assisted-by: Claude Opus 5.5
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv
Tyrie Vella (tyrielv) force-pushed the tyrielv/sparse-index/vfs-capability branch from 7fb7378 to ea4666d Compare November 14, 2026 09:29

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tyrielv