[tnazarew]

Stacked on #353 (base validation/no-false-greens; both change the Spark/Hive workflows). Retarget after #353 merges.

Why

Groundwork for running each producer locally the same way CI does (follow-up PRs per producer). Spark and Hive each carry a copy of runner/dataproc_workflow.py; Hive's is a strict subset of Spark's (the cluster, job, GCS and credentials code is identical; Spark adds run-scenarios and a configurable master machine type, Hive adds run-hive-query).

What (no behaviour change)

• One runner: dev/dataproc/dataproc_workflow.py = Spark's file (moved with git mv) plus Hive's run-hive-query; the two identical requirements.txt become dev/dataproc/requirements.txt. The Spark and Hive workflows call the new paths.
• Helpers in dev/, used by CI steps and local runs (wired in by the producer PRs):

  Path
  defaults.env	project / region / bucket defaults
  dataproc/create_cluster.sh, delete_cluster.sh, gcs_upload.sh	cluster create/delete (--list-mine for leftovers), upload-unless-present
  dataproc/install_jars.sh	one Dataproc init action: copies the jars listed in the metadata (JARS → JARS_TARGET_DIR on every node, MASTER_JARS → master only), no defaults; a failed copy fails cluster creation
  lib/status.sh	writes status.json
  validation/validate_locally.sh	validates one run like CI (cached OpenLineage clone, specs, venv), status per scenario, exit 1 on failures

• dev/README.md (contents, prerequisites: gcloud auth application-default login, no key file needed) and a "Testing a scenario before committing" section in producer/README.md.
• .gitignore had a Python-packaging lib/ pattern that would ignore dev/lib/; added an exception.
• The PR path map runs Spark and Hive for dev/dataproc/, dev/lib/, dev/defaults.env (dbt for dev/lib/).

Verification

• pytest scripts/tests → 78 passed, including:
  • test_dataproc_workflow_cli.py: all five subcommands, and every invocation the Spark and Hive workflows make today parses to the right handler.
  • test_install_jars.py: runs the init action with stub metadata and gsutil (every-node vs master-only copies, default target dir and fallback, failing copy fails, empty metadata). Found and fixed an empty-list set -u error under macOS bash.
  • test_run_scripts.py: --help, unknown option and missing arguments for every dev/ script.
• pyflakes on the merged runner: no undefined names (only an unused variable both copies already had).
• validate_locally.sh on the stored Hive events of the last release run (OL 1.55.0): aggregate_job SUCCESS, exit 0; with a status.json marking the job FAILED: producer job FAILURE, exit 1.
• shellcheck clean; actionlint: no new findings.
• This PR moves dataproc_workflow.py, so its own check runs Spark and Hive for real.

🤖 Generated with Claude Code