Add opt-in cross-source package deduplication by AmelBawa-msft · Pull Request #6590 · microsoft/winget-cli

AmirMS (AmelBawa-msft) · 2026-10-12T20:12:52Z

📖 Description

Add opt-in cross-source package deduplication for single-package install, download, and ordinary show commands.

Enable it with experimentalFeatures.packageDeduplication: true. It is disabled by default and independent of interactive package selection.

Compare two or three complete candidates from distinct sources, after source priority and before interactive selection.
Require matching resolved versions/channels and installer SHA256 hashes, plus compatible installer families, scope, operation restrictions, and relevant dependency behavior.
Select the earliest participating source in the opened source order and reuse its exact version, manifest, and installer without merging metadata or probing older releases.
Preserve existing prompt/error behavior when equivalence cannot be established. Also propagate source-reported truncation so incomplete candidate sets are not treated as complete.

This does not merge search rows or apply to show --versions, upgrade flows, multi-package operations, or COM/PowerShell/DSC callers. Payload equivalence does not guarantee identical installer behavior; the selected source's instructions remain authoritative.

🎞️ Demo

WinGet cross-source package deduplication: before and after

Installed winget v1.30.180-preview before, and the PR's wingetdev build with deduplication enabled after. Actual read-only CLI output is replayed with excerpts and condensed timing.

PuTTY: equivalent installers resolve automatically. Both sources report version 0.85.0.0 and the same full installer SHA256, with compatible MSI/WiX installer types. winget reports ambiguity; wingetdev selects one candidate without prompting. The Store candidate wins because msstore came first in the opened source order at equal priority, not because the heuristic prefers the Store. Result-table order does not determine the selection.
OBS Studio: different versions remain separate. The sources report 32.2.2 from winget and 32.0.1 from msstore, with different installer hashes. Both builds retain ambiguity. Deduplication does not choose an older release just to obtain a match.

🔗 References

Related to #5345.

🔍 Validation

From src\x64\Debug\AppInstallerCLITests:

.\AppInstallerCLITests.exe '[PackageDeduplication]' -r compact

Passed: 43 test cases, 7,718 assertions.

Live comparisons with the deployed PR build:

Command	Result
`winget show PuTTY --disable-interactivity`	Expected ambiguity
`wingetdev show PuTTY --disable-interactivity`	Success; equivalent installers detected and Store candidate selected
`winget show "OBS Studio" --disable-interactivity`	Expected ambiguity
`wingetdev show "OBS Studio" --disable-interactivity`	Expected ambiguity retained

Also verified both scenarios on the same PR build with deduplication disabled and enabled to isolate this feature's effect. Source-specific metadata confirmed PuTTY's matching version/hash and OBS Studio's differing versions/hashes.

Development settings were restored after capture. No packages were installed by the demo.

The full native suite and installation/download E2E tests were not run.

✅ Checklist

Signed the Contributor License Agreement
Linked to an issue
Updated Release Notes (if applicable)
Updated documentation (if applicable)
Added or updated tests (or noted why not applicable)
Updated Copilot instructions (if build, architecture, or conventions changed)

🤖 AI Assistance

AI assistance was used and has been disclosed in this PR
No AI assistance was used

AI assisted with code refactoring, validation, spelling fixes, and demo/PR preparation.

📋 Issue Type

Bug fix
Feature
Task

Microsoft Reviewers: Open in CodeFlow

Copilot

🔵 Needs a closer look

Automatic package-source selection affects install security and reliability across several core workflows and warrants final human validation.

0 open findings

What changed in this PR

Adds opt-in cross-source package deduplication while preserving source priority, trust, agreements, and installer verification.

Changes:

Adds the packageDeduplication experimental setting.
Compares versions, hashes, applicability, and dependencies before automatic selection.
Adds extensive workflow tests, documentation, resources, and truncation handling.

File	Description
`src/AppInstallerRepositoryCore/CompositeSource.cpp`	Propagates child-source truncation.
`src/AppInstallerCommonCore/UserSettings.cpp`	Validates the new setting.
`src/AppInstallerCommonCore/Public/winget/UserSettings.h`	Defines the setting mapping.
`src/AppInstallerCommonCore/Public/winget/ExperimentalFeature.h`	Adds the feature flag.
`src/AppInstallerCommonCore/ExperimentalFeature.cpp`	Implements feature discovery and enablement.
`src/AppInstallerCLITests/SourceFlow.cpp`	Adds deduplication workflow coverage.
`src/AppInstallerCLITests/PromptFlow.cpp`	Tests truncated composite results.
`src/AppInstallerCLITests/ExperimentalFeature.cpp`	Tests setting and policy behavior.
`src/AppInstallerCLITests/CompositeSource.cpp`	Tests truncation propagation.
`src/AppInstallerCLIPackage/Shared/Strings/en-us/winget.resw`	Adds user-facing messages.
`src/AppInstallerCLICore/Workflows/WorkflowBase.h`	Declares selection flags and prepared matches.
`src/AppInstallerCLICore/Workflows/WorkflowBase.cpp`	Implements comparison and selection.
`src/AppInstallerCLICore/Workflows/UpdateFlow.cpp`	Handles prepared install matches.
`src/AppInstallerCLICore/Workflows/ShowFlow.cpp`	Enables prepared show matches.
`src/AppInstallerCLICore/Workflows/InstallFlow.h`	Exposes argument compatibility checking.
`src/AppInstallerCLICore/Workflows/InstallFlow.cpp`	Implements non-mutating argument checks.
`src/AppInstallerCLICore/Resources.h`	Registers new resources.
`src/AppInstallerCLICore/ExecutionContextData.h`	Stores prepared package matches.
`src/AppInstallerCLICore/Commands/ShowCommand.cpp`	Opts `show` into deduplication.
`src/AppInstallerCLICore/Commands/InstallCommand.cpp`	Opts `install` into deduplication.
`src/AppInstallerCLICore/Commands/DownloadCommand.cpp`	Opts `download` into deduplication.
`schemas/JSON/settings/settings.schema.0.2.json`	Documents the setting schema.
`doc/windows/package-manager/winget/source.md`	Documents source behavior.
`doc/specs/#5345 - Interactive package selection.md`	Describes interaction with prompting.
`doc/specs/#5345 - Cross-source package deduplication.md`	Adds the feature specification.
`doc/Settings.md`	Documents configuration.
`doc/ReleaseNotes.md`	Announces the experimental feature.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

ranm-msft

Thanks for this - the implementation is careful and the opt-in is genuinely surgical. I read WorkflowBase.cpp/.h, the three command opt-in sites, and the InstallFlow/ShowFlow/UpdateFlow/PromptFlow hunks. The THROW_HR_IF identity/operation/version/manifest asserts, the single-use reset of Data::PreparedPackageMatch in SelectInstaller, the GetDetails().Priority skip that preserves source precedence, and the fact that OpenSource and InstallSinglePackage are untouched (so no source or package agreement is bypassed) all look right to me.

One question is holding me back from approving.

Is byte-identical SHA-256 sufficient to call two installers equivalent?

GetEquivalentInstallers requires the preferred installer hash to be present, correctly sized, non-zero and identical across candidates, plus matching version/channel, an allowlisted installer type, and non-conflicting explicit scope. That proves the two candidates point at the same payload. It does not appear to prove they would execute the same way.

A single bootstrapper or unified installer can install different products, editions, workloads or channels purely as a function of its command line. Two manifests from different sources can legitimately share a hash and a nominal version while differing in InstallerSwitches, generated arguments, expected return codes, or ExpectedReturnCodes interpretation. In that case the dedup path would silently pick one source's execution contract and suppress the disambiguation prompt that exists precisely to let the user choose.

Concretely, I did not find a comparison of the fields that determine what the installer actually does - switches and generated arguments in particular - only of the payload identity and the surrounding eligibility gates.

Two ways this could be closed out, either of which would let me approve:

Extend the equivalence predicate to cover the effective execution contract (switches / generated arguments / scope and elevation behaviour), and add a negative test using the same hash and version but product-selecting switches that differ, asserting fallback to the normal prompt.
If the shared-bootstrapper false positive is considered acceptable in practice, say so explicitly in the spec and I will take that as the answer - it is a product call, not a code defect.

I appreciate that the feature is experimental and off by default, which keeps the blast radius small. I did not want to lean on that to wave through a predicate I could not evidence, since the user-facing message describes the results as equivalent.

Everything else I read I am comfortable with, and the test coverage across SourceFlow.cpp, CompositeSource.cpp, PromptFlow.cpp and ExperimentalFeature.cpp is substantial.

Kaleb Luedtke (Trenly) · 2026-10-21T18:17:21Z

+
+### Comparison rules
+
+Use existing command-specific version, channel, pin, and installer-applicability rules. Compare the preferred installer, not any installer with a matching hash. For install without an explicit version, evaluate only the first permitted version; an inapplicable installer or missing requested version leaves the set unresolved. Never probe older manifests.


What if the sources differ in preferred installer? For example, if Source A has installer entries for msi and exe but Source B has only installer entries for exe - then the preferred comparison would be msi vs exe.

Even if the exe installers have byte-for-byte identical SHA256, the sources wouldn't be de-duplicated, despite Source B being a subset-type duplicate of Source A ?

Kaleb Luedtke (Trenly) · 2026-10-21T18:22:59Z

+On selection, emit localized text, then continue normal command output:
+
+```text
+Equivalent installers found. Selected Contoso Editor [Contoso.Editor] from source Alpha.


Will logs show the equivalent installers that were compared without needing to enable verbose logging?

JohnMcPMS · 2026-10-21T17:53:58Z

+    <comment>{Locked="{0}","{1}","{2}"} {0} is the selected package name, {1} its identifier, and {2} its source name. Explains automatic selection across sources.</comment>
+  </data>
+  <data name="PackageDeduplicationFailed" xml:space="preserve">
+    <value>Could not compare packages across sources. Refine the search or choose a package.</value>


Do we really need a message that states: "I couldn't do the new thing so here is the old thing."? Presumably the second sentence here is redundant anyway.

I think logging should be good enough, no output required.

JohnMcPMS · 2026-10-21T17:56:35Z

 using namespace AppInstaller::Repository;
 using namespace AppInstaller::Settings;

+namespace


Why are these tests in this file? They are more install flow or generic workflow, not source specific (as evidenced by not changing the product source flow file).

JohnMcPMS · 2026-10-21T18:23:51Z

+        template <>
+        struct DataMapping<Data::PreparedPackageMatch>
+        {
+            using value_t = std::optional<Workflow::PreparedPackageMatch>;


Suggested change

using value_t = std::optional<Workflow::PreparedPackageMatch>;

using value_t = Workflow::PreparedPackageMatch;

I really don't like having an optional inside an optional item.

JohnMcPMS · 2026-10-21T18:33:28Z

+        GetManifestFromPackage(bool considerPins, std::optional<OperationType> preparedOperation = {}) :
+            WorkflowTask("GetManifestFromPackage"), m_considerPins(considerPins), m_preparedOperation(preparedOperation) {}


Why do we need this "secret phrase" type value whose only purpose is to allow us to use the prepared data if it matches? Since it is a hard error when not set properly, it doesn't appear to be intended as control flow. It only serves to double check that we expected the possibility of prepared data for the operation type.

It should be removed unless there is some justification.

JohnMcPMS · 2026-10-21T18:42:57Z

+                        Utility::LocIndView{ prepared->PackageVersion->GetSource().GetDetails().Name }) << std::endl;
+                    Logging::Telemetry().LogAppFound(name, manifest.Id);
+                    context.Add<Execution::Data::Package>(std::move(package));
+                    context.Add<Execution::Data::PreparedPackageMatch>(std::move(prepared));


I really don't like this alternate data item that we then have to merge back in when the alternate data it carries is requested. I'm assuming that the goal is to optimize on the fact that we needed all of this data to make the choice, so lets use it.

I would prefer that we set the data items that are needed and update the code that would normally choose them to detect their presence and no-op.

JohnMcPMS · 2026-10-21T19:01:41Z

+
+        if ((operation != OperationType::Install && operation != OperationType::Download && operation != OperationType::Show) ||
+            context.Args.Contains(Execution::Args::Type::Manifest) || context.Args.Contains(Execution::Args::Type::ListVersions) ||
+            context.Args.GetCount(Execution::Args::Type::MultiQuery) > 1 ||


Can we even get here with a multi-query? Shouldn't we be in a subcontext with a single query by this point?

JohnMcPMS · 2026-10-21T19:04:10Z

+            WI_IsAnyFlagSet(context.GetFlags(), Execution::ContextFlag::InstallerExecutionUseUpdate | Execution::ContextFlag::InstallerExecutionUseRepair) ||
+            (context.Contains(Execution::Data::InstalledPackageVersion) && context.Get<Execution::Data::InstalledPackageVersion>()))
+        {
+            return skip("unsupported operation or context");


We need to break this down into individual checks and messages because I don't want to have to reverse engineer which of these was not allowed after the fact.

JohnMcPMS · 2026-10-21T21:37:03Z

+                if (priority && *priority != opened->GetDetails().Priority)
+                {
+                    return skip("results have not been filtered by source priority");
+                }


I don't understand how this is relevant.

JohnMcPMS · 2026-10-21T21:44:46Z

+            return std::nullopt;
+        }
+
+        std::optional<size_t> GetDeduplicationCandidateIndex(const SearchResult& result, const Source& source)


This whole function seems off-base. The only way the deduplication should work is if, for the highest priority level with packages, there are either 0 or 1 packages per-source.

I might be missing something here, but this seems like this code could support 2 sources with 2 packages from one of them. There is no point trying to deduplicate multiple packages from the same source.

The code that handles priority in EnsureOneMatchFromSearchResult already creates a "highest priority only" result set. It explicitly excludes anything that targets installed items right now, but you could build the list in more cases and then only use it when necessary. That gets you the priority reduced set to use here, but then it becomes simply a filter rather than this attempt selection ("are there no more than 1 package per source"). As long as that is true, the candidate index will always be 0.

JohnMcPMS · 2026-10-21T21:48:37Z

+
+Every candidate must have:
+
+1. Equal resolved versions under existing version comparison (`2.4` equals `2.4.0`) and case-insensitively equal channels. Reject empty, `Unknown`, `Latest`, or approximate versions. Unknown search metadata is acceptable if retrieval resolves a concrete version.


I think I would be ok with a little more flexibility in versions, like:

Choose the lower of the latest versions, then do the comparison at that version

Then always install from the source with the highest version if deduped

AmirMS (AmelBawa-msft) added 5 commits October 12, 2026 11:50

Preserve child source truncation in composite searches

44de348

Add installer equivalence checks for cross-source deduplication

7270918

Add bounded cross-source resolution for equivalent packages

952589b

Integrate opt-in cross-source deduplication into CLI commands

302fce8

Document opt-in cross-source package deduplication behavior

ea7ac8c

AmirMS (AmelBawa-msft) requested a review from a team as a code owner October 12, 2026 20:12

This comment has been minimized.

Sign in to view

Refactor package deduplication into focused helper functions

776bca5

AmirMS (AmelBawa-msft) requested a balanced review from Copilot October 12, 2026 20:28

AmirMS (AmelBawa-msft) marked this pull request as draft October 12, 2026 20:28

Copilot started reviewing on behalf of AmirMS (AmelBawa-msft) October 12, 2026 20:29

View session

This comment has been minimized.

Sign in to view

Fix spelling checks for cross-source package deduplication

fc5e3e9

Copilot AI reviewed Oct 12, 2026

View reviewed changes

AmirMS (AmelBawa-msft) marked this pull request as ready for review October 12, 2026 21:45

ranm-msft reviewed Oct 13, 2026

View reviewed changes

Kaleb Luedtke (Trenly) reviewed Oct 21, 2026

View reviewed changes

JohnMcPMS reviewed Oct 21, 2026

View reviewed changes


		### Comparison rules

		Use existing command-specific version, channel, pin, and installer-applicability rules. Compare the preferred installer, not any installer with a matching hash. For install without an explicit version, evaluate only the first permitted version; an inapplicable installer or missing requested version leaves the set unresolved. Never probe older manifests.

	using value_t = std::optional<Workflow::PreparedPackageMatch>;
	using value_t = Workflow::PreparedPackageMatch;

		GetManifestFromPackage(bool considerPins, std::optional<OperationType> preparedOperation = {}) :
		WorkflowTask("GetManifestFromPackage"), m_considerPins(considerPins), m_preparedOperation(preparedOperation) {}


		Every candidate must have:

		1. Equal resolved versions under existing version comparison (`2.4` equals `2.4.0`) and case-insensitively equal channels. Reject empty, `Unknown`, `Latest`, or approximate versions. Unknown search metadata is acceptable if retrieval resolves a concrete version.

Conversation

AmirMS (AmelBawa-msft) commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📖 Description

🎞️ Demo

🔗 References

🔍 Validation

✅ Checklist

🤖 AI Assistance

📋 Issue Type

Microsoft Reviewers: Open in CodeFlow

Uh oh!

This comment has been minimized.

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

ranm-msft left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

AmirMS (AmelBawa-msft) commented Oct 12, 2026 •

edited

Loading