Skip to content

Add deferrable support to AthenaSparkOperator - #75915

Merged
shahar1 merged 2 commits into
apache:mainfrom
SameerMesiah97:Add-AthenaSpark-Deferrable-Mode
Nov 14, 2026
Merged

shahar1 merged 2 commits into
apache:mainfrom
SameerMesiah97:Add-AthenaSpark-Deferrable-Mode

Conversation

@SameerMesiah97

Copy link
Copy Markdown
Contributor

Description

This change adds deferrable execution support to AthenaSparkOperator, allowing calculations to be monitored asynchronously without occupying a worker slot.

Set deferrable=True to enable this behavior. Both execution modes return the same execution metadata and raise an exception if the calculation fails or is canceled.

Rationale

Athena Spark calculations can run for extended periods. Deferrable execution moves polling to the triggerer, releasing the worker slot while the calculation runs.

The waiter_delay and waiter_max_attempts parameters control polling in both execution modes.

Tests

Added operator tests verifying that:

  • Deferrable execution passes the calculation ID, polling settings, and AWS configuration to the trigger without synchronous polling.
  • Successful completion returns the expected execution metadata.
  • Failed and canceled calculations raise RuntimeError.
  • Missing or unsuccessful trigger events and unexpected calculation states request cancellation and raise RuntimeError.
  • Cancellation failures preserve the original trigger error.

Added trigger tests verifying that:

  • The trigger serializes with the expected classpath and calculation ID.
  • Polling settings and AWS configuration are preserved during serialization.
  • The trigger can be reconstructed from its serialized arguments.
  • The trigger uses AthenaHook and the expected Spark waiter, API arguments, status queries, and return value.

Example DAGs

Updated the existing Athena Spark system test DAG to exercise synchronous and deferrable execution sequentially within the same session.

Documentation

Added a deferrable execution example, prerequisites, polling behavior, and cancellation limitations to the Athena Spark guide. Updated operator and trigger docstrings.

Backwards Compatibility

The existing synchronous execution path and metadata return structure are preserved.

The deferrable parameter defaults to the operators.default_deferrable configuration value, falling back to False. DAG authors can explicitly set deferrable=False to retain synchronous execution regardless of that configuration.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [GPT 6.1] following the guidelines

@SameerMesiah97
SameerMesiah97 marked this pull request as ready for review October 29, 2026 18:23
@SameerMesiah97

Copy link
Copy Markdown
Contributor Author

Requesting review for for this.

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The deferrable flow looks right: trigger error events reach execute_complete, which stops the calculation before failing, and terminal states go through the same _handle_terminal_state path as sync mode. I have two minor points inline (deferrable mode needs an extra IAM action, and the new waiter's acceptors are untested) and four nits. The real CI has not run yet (workflow approval pending), so none of this is CI-verified.

Smaller observations

All six are inline:

  • minor providers/amazon/src/airflow/providers/amazon/aws/waiters/athena.json:86: the waiter polls GetCalculationExecutionStatus, a separate IAM action from the GetCalculationExecution that sync mode uses.
  • minor providers/amazon/tests/unit/amazon/aws/triggers/test_athena.py:79: nothing tests the FAILED/CANCELED → success acceptor mapping.
  • nit providers/amazon/src/airflow/providers/amazon/aws/triggers/athena.py:114: the hook() override repeats what aws_hook_class already provides.
  • nit providers/amazon/src/airflow/providers/amazon/aws/triggers/athena.py:78: the trigger docstring does not document region_name, verify or botocore_config.
  • nit providers/amazon/src/airflow/providers/amazon/aws/operators/athena_spark.py:61: stray blank line in the docstring.
  • nit providers/amazon/src/airflow/providers/amazon/aws/operators/athena_spark.py:168: raise-then-catch in execute_complete; one guard would be simpler.

This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. The findings
below are observations, not blockers; an Apache Airflow
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.

More on how Apache Airflow handles maintainer review:
Contributing guide.

@SameerMesiah97
SameerMesiah97 force-pushed the Add-AthenaSpark-Deferrable-Mode branch from 1216c01 to 63bc33d Compare November 13, 2026 21:58
@github-actions github-actions Bot added the fork ci required Gated by AIP-120: CI must pass in the contributor's fork. Managed by the fork CI reconciler. label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Fork CI: green. CI for the latest commit passed in your fork. The pull request is ready for maintainer review.

Pull requests from contributors run their CI in the contributor's own fork. GitHub Actions is free and has no minute limit on public forks, while the CI capacity of the Apache Software Foundation is shared by every ASF project and is finite.

Latest run: https://github.com/SameerMesiah97/airflow-oss-sm97/actions/runs/43595592941

How this works: CI in your fork.


This comment was posted automatically by the fork CI monitor from templates drafted with an AI-assisted tool, so it may contain mistakes. An Apache Airflow maintainer, a real person, will review the pull request once its fork CI is green. Why the first pass is automated.

@SameerMesiah97
SameerMesiah97 force-pushed the Add-AthenaSpark-Deferrable-Mode branch from 63bc33d to a8f2954 Compare November 13, 2026 22:06
@SameerMesiah97
SameerMesiah97 force-pushed the Add-AthenaSpark-Deferrable-Mode branch from a8f2954 to b98af76 Compare November 13, 2026 23:53
Sameer Mesiah added 2 commits November 14, 2026 12:35
Monitor calculations asynchronously with AthenaSparkTrigger, releasing
worker slots while waiting. Preserve metadata and failure handling,
with tests, documentation, and a system test example.
…on.Simplify trigger event validation, remove the redundant hook override, and document the remaining AWS configuration parameters.
@SameerMesiah97
SameerMesiah97 force-pushed the Add-AthenaSpark-Deferrable-Mode branch from b98af76 to f9222da Compare November 14, 2026 12:35
@github-actions github-actions Bot added the ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. label Nov 14, 2026
@shahar1
shahar1 merged commit 53ca160 into apache:main Nov 14, 2026
90 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers fork ci required Gated by AIP-120: CI must pass in the contributor's fork. Managed by the fork CI reconciler. kind:documentation provider:amazon AWS/Amazon - related issues ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants