Skip to content

Don't hardcode deferrable mode in Athena Spark system test - #77173

Open
seanghaeli wants to merge 1 commit into
apache:mainfrom
aws-mwaa:ghaeli/athena-spark-system-test-follow-default
Open

seanghaeli wants to merge 1 commit into
apache:mainfrom
aws-mwaa:ghaeli/athena-spark-system-test-follow-default

Conversation

@seanghaeli

@seanghaeli seanghaeli commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #75915. Same fix as #70206 made for the Glue crawler system test.

  • The Athena Spark system test had a second calculation task hard-coded to deferrable=True, and the original task was pinned to deferrable=False.
  • Forcing deferral makes the Dag hang in system-test runs with executors.
  • This restores the test as it was before Add deferrable support to AthenaSparkOperator #75915: one calculation task with deferrable unset, so it follows operators.default_deferrable like the other Amazon system tests. Both modes still get covered by flipping that setting.
  • Docs: drops the exampleinclude of the removed snippet. The "Deferrable mode" section still explains deferrable=True.

Ran the system test against AWS with default_deferrable set to true (task deferred and resumed) and false (synchronous). Both passed and cleaned up after themselves.

The deferrable example task forced deferrable=True, so the Dag hangs
when it runs with an executor and no triggerer. Like the other Amazon
system tests, keep one calculation task and leave deferrable unset so it
follows operators.default_deferrable.
@seanghaeli
seanghaeli marked this pull request as ready for review November 16, 2026 19:41
@seanghaeli seanghaeli changed the title Let the Athena Spark system test follow default_deferrable Don't hardcode deferrable mode in Athena Spark system test Nov 16, 2026
@vincbeck
vincbeck enabled auto-merge (squash) November 16, 2026 21:36

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants