[Tyrie Vella (tyrielv)]

Problem and Context

A sparse index collapses each directory outside the sparse-checkout cone into one sparse-directory entry. Several lookups in the working-tree walk use index_name_pos(). If a sparse-directory entry contains the path, index_name_pos() expands the whole index.

A plain sparse checkout does not get to these lookups, because the files outside the cone are not on disk. Under core.virtualFileSystem, the working tree shows the full tree while the index stays sparse. Thus the walk goes into the collapsed directories, and the first lookup there expands the index. In a repository with about 2.2 million index entries, an earlier version of this change made git status go from about 20 seconds to less than half a second. When the command writes the index, it expands the index a second time ("re-expanding after write").

This change applies only when core.virtualFileSystem is set and the index is sparse. Without a virtual file system, the code paths do not change.

After this change, these commands leave the index sparse under a virtual file system. Before, the first expansion made git write a full index.

Changes

• Add vfs_sparse_dir_containing() to sparse-index.c. If core.virtualFileSystem is set, the index is sparse, and a sparse-directory entry contains the path, it returns that entry. Otherwise it returns NULL. It never expands the index. Under core.virtualFileSystem, a NULL result means that index_name_pos() cannot expand the index for that path.
• Add sparse_dir_get_tree_entry(), which finds the rest of the path in the tree of a sparse-directory entry. It can compare names without case. If a tree cannot be read, it dies, so that an unreadable tree cannot make a tracked path look untracked to git clean.
• Call vfs_sparse_dir_containing() first in each lookup that the walk uses. When it returns NULL, the existing code runs. When it returns an entry, the lookup gives the answer of the full index:
  • read_skip_worktree_file_from_index() reads a missing .gitignore from the tree.
  • The untracked-cache lookup in add_patterns() hashes the .gitignore on disk, as it does for any path without a usable index entry.
  • directory_exists_in_index() uses the tree, before the case-insensitive lookup. The name hash does not contain directories inside a sparse-directory entry, so without this, a tracked directory looks untracked when core.ignoreCase is set.
  • get_index_dtype() returns DT_UNKNOWN, so the caller uses lstat(). An index that was just expanded also returns DT_UNKNOWN, because expansion does not mark the new entries up to date.
  • index_name_is_other() uses the tree, and reports a directory as "other", as the full index does.
  • directory_exists_in_index() and index_name_is_other() compare without case when core.ignoreCase is set, because with a full index the walk finds tracked paths case-insensitively.
  • For these two lookups, vfs_sparse_dir_containing() also finds the sparse-directory entry when its name has a different case on disk (for example OUT/a for the entry out/). The index sorts names with case, so such a path does not sort next to its entry. The helper looks up each leading directory of the path in the case-insensitive dir_hash, which contains the directory of each sparse-directory entry. Without this, git status reports the tracked files below OUT/ as untracked, and git clean -f -d deletes them. In a test repository with about 41,000 paths on disk, the walk of git status -uall took 0.245 seconds with core.ignoreCase instead of 0.22.
• Add t1094-virtualfilesystem-sparse-walk.sh. The setup makes a cone-mode sparse index, puts the out-of-cone files and some untracked and ignored files back on disk, and installs a virtual file system hook. A second clone with the same working tree and a full index is the reference.
  • These tests compare git status (several -u and --ignored modes, with core.ignoreCase false and true, and with the untracked cache), git clean -n, and git add --ignore-missing with the full-index clone. Each test also uses trace2 to make sure that the index does not expand and stays sparse.
  • A test checks the exact git clean -n -d -x output.
  • These tests rename a directory on disk to a different case and check that git status and git clean -n -d give the results of the full index: a directory inside the sparse directory, the sparse directory itself (out/ to OUT/), and a nested sparse directory (out/deep/ to OUT/Deep/, with out/ in the cone).
  • A test removes a tree object and checks that git clean -n -d fails and lists nothing.
  • A last test shows that without a virtual file system, the walk still expands the index.

Known gaps and deliberate decisions

• Hook rows inside a sparse directory. apply_virtualfilesystem() still uses index_name_pos() for each path that the hook lists, so a hook row inside a sparse directory still expands the index. A separate change handles that function. The new tests list only the exact sparse-directory name out/.
• Skip-worktree bits inside a sparse directory. The index now stays sparse, so the files inside a sparse-directory entry keep the skip-worktree bit, and git status does not report changes to them. The virtual file system layer must bring a changed path into the cone. Before this change, the first expansion wrote a full index, which removed the benefit of the sparse index.
• Case difference in a sparse-directory name without a virtual file system. Without core.virtualFileSystem, a sparse-directory name with a different case on disk (for example OUT/ for the entry out/) still makes tracked files below it look untracked. This is not new, and it is rare without a virtual file system, because a plain sparse checkout does not put those files on disk. This PR changes only the virtual file system case. No follow-up is planned in this PR.
• Type and object ID from disk. get_index_dtype() returns DT_UNKNOWN, and the untracked cache hashes the .gitignore on disk. A full index that git status refreshed would answer from the index instead. The answers differ only when a skip-worktree path on disk differs from the index, and then the new answer describes the disk.