[Keith McGrane (Keithland89)]

For 0.6.8. Keith decides the merge.

Summary

Workday org data now installs on Azure, from its CSV export, at parity with Fabric. After each
Entra ID load, the jobs add the newest export's columns to dbo.copilot_org_data, matched on
Primary Work Email, with the Fabric notebook's rules and checks. This is Keith's decision,
"Workday CSV on Azure: port it in 0.6.8 via W4", so it's part of workstream W4 of the 0.6.8 Azure
parity plan. No linked issue.

Stacked on #131 (Product feedback by CSV); merge after #131. This PR reuses the drop folder,
CSV landing and installer steps from #129 and #131, and edits the same lines, so its base is
#131's branch and its diff is one commit. Once #131 merges, this PR's base moves to main; if #131
is squash-merged, this branch also rebases onto main (git rebase --onto main 968628e, #131's
head).

As on Fabric, Workday is a data source that comes with org data, not a module (workday in
installer/src/uploads.js, csv or skip). So the catalog's modules, AZURE_SUPPORTED_MODULES
and the Azure module list in installer/README.md don't change.

• Core. The new valuelens_core.workday ports the rules of the Fabric notebook
  Copilot_Org_Data_Workday_Lander (fabric/notebooks/workday-org-data/): its settings and their
  defaults, its column-name rules, its checks and its messages.
  • MODE auto (the default) enriches the org data when there is some, and otherwise lands the
    export on its own. enrich and standalone force one or the other.
  • The export's work email (Primary Work Email by default) is trimmed, lowercased and matched
    to the org data's PersonId. The export adds only the columns the org data doesn't have. The
    org data's people and values win, even blank ones, people only in Workday are left out, and
    exact duplicate rows are kept once.
  • Before anything is written, the step fails on: an export with no rows; a missing or blank
    email; two different rows for one email; org data with a blank or repeated PersonId; a match
    rate under the minimum (0.5); or an empty result. Only the Workday step fails: the org data
    stays as the Entra ID load left it, and the run carries on.
• Jobs.
  • The new workday collector (collect/workday.py) runs right after the org data, whatever its
    outcome, as Fabric's pipeline runs the notebook. It runs when VALUELENS_MODULES has both
    orgData and workday.
  • It reads the newest .csv or .txt (by modified time, then name) in the drop folder's
    workday/: the landing container, or the SharePoint drop folder with private networking. It
    logs how many older files it ignores. The drop folder's SUBDIRS gains workday.
  • VALUELENS_WORKDAY_OUTPUT_TABLE says where the result goes. The default, as in the notebook,
    is a copy to review, dbo.copilot_org_data_workday_preview, with a new snapshot publish
    target. The installer sets dbo.copilot_org_data with
    VALUELENS_WORKDAY_ALLOW_BASE_OVERWRITE=true, as Fabric's installer does
    (installer/src/catalog.js:215-216), so each run enriches raw/copilot_org_data in place
    before publish.
  • If the Entra ID load fails, the last enriched org data is the baseline, so its Workday columns
    keep their values. With no export, nothing is written.
  • The other settings are VALUELENS_WORKDAY_MODE, VALUELENS_WORKDAY_MIN_MATCH_RATE and
    VALUELENS_WORKDAY_EMAIL_COLUMN, with the notebook's defaults. The step's messages name these
    variables rather than the notebook's settings.
• SQL. No migration: publish adds the export's columns to dbo.copilot_org_data as
  NVARCHAR(4000) NULL, and creates the review table when it's used.
• Infra. main.bicep gains workdayOutputTable (default
  dbo.copilot_org_data_workday_preview) and workdayAllowBaseOverwrite (default false), which
  containerapps.bicep passes to the jobs as VALUELENS_WORKDAY_OUTPUT_TABLE and
  VALUELENS_WORKDAY_ALLOW_BASE_OVERWRITE. installer/src/azure/main.arm.json is rebuilt from it.
• Installer.
  • Azure now offers Workday, with org data. Under the advanced settings, Workday org data?
    offers Upload a CSV export or Leave it out, and --data workday=csv sets it too. On
    Repair or change, it's under Module connections. Demo mode leaves it out, with a note
    if it was chosen.
  • The plan says how to export it and where to drop it. With private networking, the existing
    SharePoint drop-folder question now also covers it.
  • A new install step, "Workday export", grants access to where the export goes, as for the
    Agent 365 export:
    • Public endpoints: you get Storage Blob Data Contributor on the storage account, so you
      can upload it.
    • Private networking: the jobs' managed identity gets Sites.Selected read on the SharePoint
      site.
  • The deployment adds workday to the jobs' modules, and always passes
    workdayOutputTable = dbo.copilot_org_data and workdayAllowBaseOverwrite = true.
  • Confirm, status and the summary show it. Uninstall's note that the SharePoint folder is left
    alone (from Azure: Agent 365 registry by CSV (V009) #129) covers it.
• Docs.
  • azure/README.md has a new Workday org data section.
  • Also updated: azure/jobs/README.md (a new section and four settings rows), azure/DESIGN.md
    (the Phase 2 row), docs/DATA-DICTIONARY.md (the Workday section), installer/README.md and
    CHANGELOG.md (## Unreleased).

Scope

Parts affected (installer / app / Fabric notebooks and pipeline / Azure infra, jobs or web / core / Power BI models):
installer; Azure infra and jobs; core; docs. No app, Fabric, Power BI model or SQL migration
changes.

Source contracts, optional inputs, install record or deployed item names changed:

• New drop subfolder: landing/workday/ with public endpoints, or workday/ under the
  SharePoint drop folder with private networking. The jobs read the newest export there and never
  move or delete files.
• Jobs:
  • VALUELENS_MODULES can now include workday (it needs orgData).
  • New settings: VALUELENS_WORKDAY_MODE, VALUELENS_WORKDAY_OUTPUT_TABLE,
    VALUELENS_WORKDAY_ALLOW_BASE_OVERWRITE, VALUELENS_WORKDAY_MIN_MATCH_RATE and
    VALUELENS_WORKDAY_EMAIL_COLUMN. Bicep sets the output table and the overwrite.
  • For the review table, used only without the installer's settings: the publish target
    copilot_org_data_workday_preview and the Parquet path
    raw/copilot_org_data_workday_preview/part-0.parquet.
• Deployment parameters: workdayOutputTable and workdayAllowBaseOverwrite, in main.bicep
  and installer/src/azure/main.arm.json. The installer's list of required template parameters
  has them.
• Azure SQL: dbo.copilot_org_data gains the export's columns, as NVARCHAR(4000) NULL.
• Install record:
  • dataSources.workday (csv or skip) now applies on Azure.
  • azure.drop (the SharePoint folder) is now also asked for when only the Workday export uses
    it.
• Model: unchanged. As on Fabric, it reads the export's Function and Location columns when
  it has them.

Validation

• Ran python -B -m pytest tests -q and recorded results below.
• Ran npm test (and npm run typecheck for the installer) in each changed Node project.
• Updated directly related documentation and checked changed links.
• No credentials, tenant exports, prompts or customer names are included.
• If .pbit files changed, documented the rebuild method and Desktop refresh / visual checks. (Not applicable: no .pbit files changed.)

Results, skipped checks and limitations:

• One commit, ef63b96, on Azure: Product feedback by CSV (V010) #131 (968628e, on Azure: Agent 365 registry by CSV (V009) #129's db08544, on main 61c9997). In CHANGELOG.md's
  ## Unreleased, Azure: Agent 365 registry by CSV (V009) #129's entry comes first, then Azure: Product feedback by CSV (V010) #131's, then this PR's, then main's entries
  verbatim, so the stack's diff against main only adds lines there.

• ARM drift check (the arm-drift job in .github/workflows/tests.yml, run locally):
  az bicep build --file azure/infra/main.bicep --outfile <temp>/main.arm.json (Bicep 0.47.16),
  then the job's comparison with installer/src/azure/main.arm.json: passed, "ARM template
  matches the Bicep source."

• python -B -m pytest tests -q -p no:cacheprovider (repo root; the template's command, plus
  -p no:cacheprovider so the run leaves no cache in the worktree): 1061 passed, 5 skipped,
  12168 subtests passed, exit 0. The 1,599 warnings are all sqlite3's date and datetime adapter
  deprecations from the existing Azure publish tests; none come from the new tests.

  • That includes the new tests/test_azure_workday.py, 30 test functions (82 cases) on:
    • the core, against the notebook's own source (its settings, name rules and messages) and
      against the Fabric regression's fixtures and expectations
      (tests/fabric_workday_enrichment.py, imported as it is);
    • the CSV reader;
    • the collector, from storage and SharePoint: the newest file, the review table and in-place
      outputs, bad settings, long and awkward headers, and a failed Entra ID load;
    • the wiring: collect order, publish targets, settings from the environment, the Bicep and ARM
      parameters, and a publish that adds the export's columns in sqlite.
  • tests/test_workday_enrichment.py, the Fabric notebook's tests, is unchanged and passes.
  • tests/test_docs.py checks the changed links and anchors.

• CI's pytest job doesn't install PySpark, so the Workday tests also ran without it:
  python -B -c "import sys; sys.modules['pyspark'] = None; sys.modules['py4j'] = None; import pytest; sys.exit(pytest.main(['tests/test_azure_workday.py', 'tests/test_workday_enrichment.py', '-q', '-p', 'no:cacheprovider']))"
  (repo root): 91 passed, 13 subtests passed, exit 0.

• npm run typecheck (installer/): passed (exit 0).

• npm test (installer/): 901 tests, 900 passed, 1 failed (exit 1). It includes 6 new
  Workday tests in test/azure.test.js and the new question in test/advanced.test.js. The
  failure, test/script.test.js:115, isn't in code this PR touches. It's environmental: this
  machine's PowerShell execution policy blocks generate.ps1 ("running scripts is disabled on this
  system"), and the policy wasn't bypassed.

• With Installer: recommend Azure and start new installs on it #128 (at a76fb80), which merges after this PR. An offline integration run, ef63b96 with
  git merge --no-ff --no-commit a76fb80, merged clean. Then:

  • npm test (installer/): 924 tests, 923 passed, 1 failed, the same script.test.js:115;
  • npm run typecheck (installer/): passed (exit 0);
  • python -B -m pytest tests/test_consolidated_release.py tests/test_executive_query_measures.py tests/test_template_dax_fixes.py tests/test_docs.py tests/test_parked_ideas.py tests/test_azure_workday.py tests/test_workday_enrichment.py -q -p no:cacheprovider
    (repo root): 331 passed, 35 subtests passed, exit 0.

  Installer: recommend Azure and start new installs on it #128's afeb4e9 already takes workday: ['csv'] out of the --data modes it makes Fabric only,
  so --data workday=csv reaches Azure with both merged.

• CI on ef63b96: passed.

  • The tests workflow's jobs all passed:
    • pytest (python -B -m pytest tests -q -ra, without PySpark): 1061 passed, 5 skipped, 12168
      subtests passed;
    • installer (npm test): 901 tests, 897 passed, 0 failed, 4 skipped; then npm run typecheck;
    • app, arm-drift and valuelens-web.
  • The azure-images workflow built both images and smoke-tested the jobs image. On pull
    requests it doesn't push them.

• Not done (offline only):

  • no deployment, live tenant or Azure run;
  • no Sites.Selected grant on a real site;
  • no Power BI refresh;
  • no image build. Don't dispatch azure-images.yml from this branch: it would push images tagged
    with installer/package.json's version (0.6.7).

  Suggested live check, once approved, on a test install with 0.6.8 images:

  1. Under the advanced settings, answer Upload a CSV export to Workday org data?.
  2. Take fabric/notebooks/workday-org-data/samples/org_workday/workday_org_sample.csv and
     replace its fictional emails with the test tenant's user principal names. Otherwise fewer
     than half match, and the step fails by design.
  3. Drop it in landing/workday/, and run the job.
  4. Check that dbo.copilot_org_data gains the export's columns, that its other columns and rows
     are unchanged, and that the model refreshes.
  5. Add a Department column and run again, to check the first item under "Found while porting"
     below. Then drop a newer export: the next run uses it. The Department column stays in
     dbo.copilot_org_data (see that item), so remove it before step 6.
  6. Repeat with private networking and a SharePoint folder.

Compatibility

• Existing Azure installs. Workday is off by default (skip), so nothing changes on Repair
  or change until you choose Upload a CSV export. Choosing it:

  • adds workday to the jobs' modules;
  • grants the upload or SharePoint read access;
  • adds the export's columns to dbo.copilot_org_data, from the first run after it lands.

  A record that already has dataSources.workday = 'csv' turns it on at the next Repair or
  change. The advanced settings show it, already chosen.

• The deployment parameters. The installer always passes dbo.copilot_org_data and true.
  They do nothing unless workday is in the jobs' modules. A redeploy resets them, so a review
  table set by hand on the job goes back to in-place enrichment at the next Repair or change.
  A deployment from the Bicep alone keeps its defaults and writes the review table.

• Where Azure differs from the notebook. These are deliberate, and the tests pin them:

  • Only the newest export is read. The notebook combines every file in its folder, where the
    Fabric installer keeps one, and two exports of the same people would conflict.
  • A file that isn't UTF-8 fails the step with a message saying so, where Spark would replace
    what it can't read.
  • A header longer than 128 characters (Azure SQL's limit for a column name) is dropped, with a
    warning.
  • As for every Azure table, publishing cuts text to 4,000 characters.

• Turning it off keeps its columns in dbo.copilot_org_data, empty from the next run.

• Unchanged:

  • The Fabric path.
  • Upload exports, the installer's upload command and --csv, which stay Fabric only.
  • Demo mode, which doesn't load a Workday sample.
  • The catalog's modules, AZURE_SUPPORTED_MODULES and the Azure module list in
    installer/README.md: Workday is a data source.

• Migrations. None, so this PR uses no migration number.

• ValueLens templates. No matching change is needed: the model reads the same
  copilot_org_data columns as on Fabric.

• Installer: recommend Azure and start new installs on it #128 (recommend Azure). Its afeb4e9 takes workday: ['csv'] out of the --data modes it
  makes Fabric only. It merges after this PR. The two merge clean, and the integration run above
  passed.

• Lines shared with other 0.6.8 PRs. As a data source, this PR doesn't touch the catalog's
  azure flags, AZURE_SUPPORTED_MODULES, the config.modules line in planAzure or the Azure
  module list in installer/README.md. It adds an entry to CHANGELOG.md's ## Unreleased.

• Known overlaps. Checked offline with git merge-tree: this branch (ef63b96) against the
  current head of each open PR, compared with Azure: Product feedback by CSV (V010) #131 (968628e) against the same head. Line numbers
  are this branch's. In each, whichever PR lands second keeps both sides. The ones this PR adds to
  Azure: Product feedback by CSV (V010) #131's are marked new.

  • CHANGELOG.md's ## Unreleased: this stack's entries (lines 23-41) are at the top of the
    section, above main's entries as main has them. So they conflict there with the PRs that also
    add an entry at the top: Azure: Agent Evaluator jobs collector, parser and V008 (W2, 1 of 3) #133, Azure: Agent Evaluator model on Azure SQL, with org-scoped access (W2, 2 of 3) #140, Add an Azure by-hand setup path #115 and Managed app: one command sets it up by hand #114 (Add an Azure by-hand setup path #115 and Managed app: one command sets it up by hand #114 already conflict with main
    there). Azure: Consumption Central parity with Fabric (fixes, docs and a parity runbook) #125, Fix the 0.6.7 Azure test findings, build release candidates, and credit the app's creator #127, Installer: recommend Azure and start new installs on it #128 and Size the Azure path for tenants above 10,000 seats #130 add theirs lower down and don't conflict there.
  • Azure: Agent Evaluator jobs collector, parser and V008 (W2, 1 of 3) #133 (Agent Evaluator jobs, 1 of 3, at 67db2a1) and Azure: Agent Evaluator model on Azure SQL, with org-scoped access (W2, 2 of 3) #140 (2 of 3, at dced5c7, which
    includes Azure: Agent Evaluator jobs collector, parser and V008 (W2, 1 of 3) #133):
    • azure/jobs/valuelens_jobs/__main__.py:20-21, the docstring's Parquet list: this stack's
      raw/agents_365/part-0.parquet and raw/user_feedback/part-0.parquet lines replace the old
      raw/agents_365/*.parquet (Phase 2, optional) line, and Azure: Agent Evaluator jobs collector, parser and V008 (W2, 1 of 3) #133's agent_* lines go after
      them.
    • __main__.py:326-329, the publish targets: this stack's agent365 and productFeedback
      lines and Azure: Agent Evaluator jobs collector, parser and V008 (W2, 1 of 3) #133's agentEvaluator lines.
    • azure/jobs/valuelens_jobs/publish.py:87-93, after the core targets: this stack's
      agents_365, user_feedback and copilot_org_data_workday_preview targets and Azure: Agent Evaluator jobs collector, parser and V008 (W2, 1 of 3) #133's
      Agent Evaluator targets.
    • azure/jobs/README.md:262-266, the settings table: this stack's VALUELENS_DROP_* row and
      the four VALUELENS_WORKDAY_* rows first, then Azure: Agent Evaluator jobs collector, parser and V008 (W2, 1 of 3) #133's five rows.
    • New: azure/jobs/valuelens_jobs/config.py:36-41 and :82-86. Both add Settings
      fields and their from_env lines: this PR's workday_*, and Azure: Agent Evaluator jobs collector, parser and V008 (W2, 1 of 3) #133's
      dataverse_environments and transcript_*.
    • Azure: Agent Evaluator jobs collector, parser and V008 (W2, 1 of 3) #133's migration test checks only the first eight versions ([:8]), so V009 and V010
      don't affect it.
  • Azure: Agent Evaluator model on Azure SQL, with org-scoped access (W2, 2 of 3) #140 only:
    • installer/src/config.js:286-287, the azure record's property docs: Azure: Agent Evaluator model on Azure SQL, with org-scoped access (W2, 2 of 3) #140's
      agentEvaluatorDatasetId line goes before this stack's drop and storageRole lines.
    • installer/src/steps/azure/index.js:533-572: this stack's azureAgent365On,
      azureProductFeedbackOn, azureWorkdayOn, azureReadsDrop and azureDropContents, and
      Azure: Agent Evaluator model on Azure SQL, with org-scoped access (W2, 2 of 3) #140's azureAgentEvaluatorOn.
    • index.js:2023-2025, the end of the summary: this stack's Agent 365:, Feedback: and
      Workday: lines and Azure: Agent Evaluator model on Azure SQL, with org-scoped access (W2, 2 of 3) #140's Evaluator: line.
  • Azure: Consumption Central parity with Fabric (fixes, docs and a parity runbook) #125 (Consumption Central parity, at 81813ec):
    • installer/test/azure.test.js:13, the import list.
    • azure/DESIGN.md:467, the Phase 2 row: Azure: Consumption Central parity with Fabric (fixes, docs and a parity runbook) #125 rewrites the "follow." sentence; keep this
      stack's three "done by CSV" sentences too.
    • New: installer/src/steps/azure/index.js:1005-1006, in azureDeployment's parameters:
      this PR's modules: line, which adds workday, and Azure: Consumption Central parity with Fabric (fixes, docs and a parity runbook) #125's semanticModels: line.
    • index.js:2022-2025, the end of the summary: Azure: Consumption Central parity with Fabric (fixes, docs and a parity runbook) #125's Credits: line (now shown when
      azureConsumptionModelDeployed(config)), then this stack's Agent 365:, Feedback: and
      Workday: lines.
  • Fix the 0.6.7 Azure test findings, build release candidates, and credit the app's creator #127 (the W1 fixes, at fe7ffd7):
    • installer/README.md:158, the list of modules supported on Azure: "core, orgData,
      m365Activity, agent365, productFeedback, consumption and defender".
    • installer/src/steps/azure/index.js:2007, the end of azureUninstall: Azure: Agent 365 registry by CSV (V009) #129's
      if (az.drop?.folderUrl) ui.note(…) goes before Fix the 0.6.7 Azure test findings, build release candidates, and credit the app's creator #127's new if (!deleting) return true;.
  • Azure: least-privilege install into a resource group your Azure admin creates #126 (least privilege, at 92ab976): index.js:45, after isPrivate. Keep Azure: least-privilege install into a resource group your Azure admin creates #126's
    azureProviders and this stack's AZURE_SUPPORTED_MODULES. The Storage Blob Data
    Contributor grant, which this PR also uses for the Workday upload, is already in Azure: least-privilege install into a resource group your Azure admin creates #126's
    role-assignment condition.
  • New: Size the Azure path for tenants above 10,000 seats #130 (large-tenant sizing, at c85c83e): installer/src/azure/main.arm.json:8, the
    templateHash only. Both change the Bicep, so whichever lands second runs
    npm run build:arm (installer/) and the ARM drift check.
  • Add an Azure by-hand setup path #115 (Azure by hand, at 8fc73d6): azure/README.md:287-288, the end of the "ARM can't do
    these, so the installer:" list: keep both bullets. Also CHANGELOG.md, README.md and
    docs/PREREQUISITES.md, the files Add an Azure by-hand setup path #115 already conflicts with main in. This stack adds nothing
    to those beyond CHANGELOG.md's.
  • Managed app: one command sets it up by hand #114 (managed app by hand, a draft): only CHANGELOG.md. Its other conflicts are with
    main itself.
  • Clean: Installer: recommend Azure and start new installs on it #128 (recommend Azure, at a76fb80), App: the Executive summary's headline figures don't wait for saved task times #138 and App: count agents in use and people with an organization at once, so the Executive summary loads sooner #139 (app), and the Dependabot bumps
    Bump the npm group across 2 directories with 7 updates #134, Bump the github-actions group with 3 updates #135 and Bump python from 3.12-slim-bookworm to 3.14-slim-bookworm in /azure/jobs #45.

• Found while porting (not changed here; the same on Fabric). For Keith to decide:

  1. A Department column probably breaks the model refresh, and on Azure it stays broken.
     Confirmed by reading the code. Not run: that needs Power Query.
     • The rename. The Chat + Agent Org Data query (its lines 33-52) renames the first
       column whose name normalizes to department to Organization, without checking whether
       Organization exists. Three templates have it: ValueLens - Fabric.pbit,
       ValueLens - Fabric OneLake.pbit and Agent Evaluator.pbit. The Azure models are built
       from them and keep the rename: the main model from ValueLens - Fabric.pbit
       (installer/src/steps/azure/index.js:1621-1628), and Azure: Agent Evaluator model on Azure SQL, with org-scoped access (W2, 2 of 3) #140's Agent Evaluator model from
       Agent Evaluator.pbit.
     • The collision. Both baselines already have Organization (from Graph's department)
       and no Department: BASE_COLUMNS in core/valuelens_core/org.py:15 on Azure, and
       fabric/notebooks/Copilot_Org_Data_Direct_Ingester.ipynb:208 on Fabric. So an export with
       a Department column adds one when it enriches, and the refresh probably fails with a
       duplicate column.
     • On Azure the failure persists. Publish never drops a column: ensure_table only adds
       (azure/jobs/valuelens_jobs/publish.py:121-135), and dbo.copilot_org_data is a snapshot
       target, replaced by DELETE then INSERT (:196-197). Nothing else in the jobs, the
       migrations or the installer drops one. So once Department has been published, it stays,
       as NULLs, and the refresh keeps failing even after Workday is turned off or the column
       leaves the export.
     • On Fabric it clears itself. The Entra ID ingester rewrites the table with
       overwriteSchema (Copilot_Org_Data_Direct_Ingester.ipynb:445-447), so Department is
       gone until the Workday lander adds it back.
     • So the model fix is the better one. Renaming only when Organization is missing also
       stops a leftover column colliding. Leaving Department out in both Workday paths doesn't
       heal a table that already has it: on Azure that also needs
       ALTER TABLE dbo.copilot_org_data DROP COLUMN [Department].
  2. Coverage advice. core/valuelens_core/org.py:179-180, docs/DATA-DICTIONARY.md:222 and
     installer/src/steps/run.js:508 suggest Workday for people without a department, but
     enrichment never fills Organization: Entra ID's values win, even blank ones.
  3. "Adds or overrides". installer/src/uploads.js:65 and
     installer/src/transform/pipeline.js:618 say Workday "adds or overrides" org fields. It only
     adds.

• Images. The collector ships in the valuelens-jobs image, so it needs the 0.6.8 images.