Repository navigation
Add deferrable support to AthenaSparkOperator - #75915
Conversation
|
Requesting review for for this. |
shahar1
left a comment
There was a problem hiding this comment.
The deferrable flow looks right: trigger error events reach execute_complete, which stops the calculation before failing, and terminal states go through the same _handle_terminal_state path as sync mode. I have two minor points inline (deferrable mode needs an extra IAM action, and the new waiter's acceptors are untested) and four nits. The real CI has not run yet (workflow approval pending), so none of this is CI-verified.
Smaller observations
All six are inline:
- minor
providers/amazon/src/airflow/providers/amazon/aws/waiters/athena.json:86: the waiter pollsGetCalculationExecutionStatus, a separate IAM action from theGetCalculationExecutionthat sync mode uses. - minor
providers/amazon/tests/unit/amazon/aws/triggers/test_athena.py:79: nothing tests theFAILED/CANCELED→successacceptor mapping. - nit
providers/amazon/src/airflow/providers/amazon/aws/triggers/athena.py:114: thehook()override repeats whataws_hook_classalready provides. - nit
providers/amazon/src/airflow/providers/amazon/aws/triggers/athena.py:78: the trigger docstring does not documentregion_name,verifyorbotocore_config. - nit
providers/amazon/src/airflow/providers/amazon/aws/operators/athena_spark.py:61: stray blank line in the docstring. - nit
providers/amazon/src/airflow/providers/amazon/aws/operators/athena_spark.py:168: raise-then-catch inexecute_complete; one guard would be simpler.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. The findings
below are observations, not blockers; an Apache Airflow
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.More on how Apache Airflow handles maintainer review:
Contributing guide.
1216c01 to
63bc33d
Compare
|
Fork CI: green. CI for the latest commit passed in your fork. The pull request is ready for maintainer review. Pull requests from contributors run their CI in the contributor's own fork. GitHub Actions is free and has no minute limit on public forks, while the CI capacity of the Apache Software Foundation is shared by every ASF project and is finite. Latest run: https://github.com/SameerMesiah97/airflow-oss-sm97/actions/runs/43595592941 How this works: CI in your fork. This comment was posted automatically by the fork CI monitor from templates drafted with an AI-assisted tool, so it may contain mistakes. An Apache Airflow maintainer, a real person, will review the pull request once its fork CI is green. Why the first pass is automated. |
63bc33d to
a8f2954
Compare
a8f2954 to
b98af76
Compare
Monitor calculations asynchronously with AthenaSparkTrigger, releasing worker slots while waiting. Preserve metadata and failure handling, with tests, documentation, and a system test example.
…on.Simplify trigger event validation, remove the redundant hook override, and document the remaining AWS configuration parameters.
b98af76 to
f9222da
Compare
Description
This change adds deferrable execution support to
AthenaSparkOperator, allowing calculations to be monitored asynchronously without occupying a worker slot.Set
deferrable=Trueto enable this behavior. Both execution modes return the same execution metadata and raise an exception if the calculation fails or is canceled.Rationale
Athena Spark calculations can run for extended periods. Deferrable execution moves polling to the triggerer, releasing the worker slot while the calculation runs.
The
waiter_delayandwaiter_max_attemptsparameters control polling in both execution modes.Tests
Added operator tests verifying that:
RuntimeError.RuntimeError.Added trigger tests verifying that:
AthenaHookand the expected Spark waiter, API arguments, status queries, and return value.Example DAGs
Updated the existing Athena Spark system test DAG to exercise synchronous and deferrable execution sequentially within the same session.
Documentation
Added a deferrable execution example, prerequisites, polling behavior, and cancellation limitations to the Athena Spark guide. Updated operator and trigger docstrings.
Backwards Compatibility
The existing synchronous execution path and metadata return structure are preserved.
The
deferrableparameter defaults to theoperators.default_deferrableconfiguration value, falling back toFalse. DAG authors can explicitly setdeferrable=Falseto retain synchronous execution regardless of that configuration.Was generative AI tooling used to co-author this PR?
Generated-by: [GPT 6.1] following the guidelines