Skip to content

Allow mounting the Airflow config as a directory to avoid the subPath mount race - #70180

Merged
Miretpl merged 7 commits into
apache:mainfrom
clarityai-eng:chart-config-mount-as-dir
Nov 16, 2026
Merged

Miretpl merged 7 commits into
apache:mainfrom
clarityai-eng:chart-config-mount-as-dir

Conversation

@diegoalvarez-clarity

@diegoalvarez-clarity diegoalvarez-clarity commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

Why

The chart mounts airflow.cfg (and airflow_local_settings.py) into every component and the KubernetesExecutor pod template via a subPath bind-mount of the config ConfigMap. subPath + ConfigMap hits a known Kubernetes race: on node/volume lifecycle events (kubelet volume reconstruction, node autoscaling) the pre-computed subPath source inode goes stale and runc fails the mount at container init:

error during container init: error mounting
".../volume-subpaths/config/base/1" to rootfs at "/opt/airflow/airflow.cfg"

The task fails with StartError and no application log (the container never starts). It's intermittent, cross-component, multi-node, and runtime-version-independent.

What

Add an opt-in mountConfigAsDir (default false; existing behaviour unchanged). When true, the config ConfigMap is mounted as a directory at {airflowHome}/config (no subPath) and AIRFLOW_CONFIG is set to {airflowHome}/config/airflow.cfg. A directory-projected ConfigMap volume has no pre-computed subPath bind source, so it's immune to the race. airflow_local_settings.py keeps its default path, so local-settings loading is unaffected. The separate pod_template_file.yaml subPath mount is out of scope.

Tests

  • test_mount_config_as_dir — flag on, across all ten templates that include the config mount (deployments, statefulsets, cronjobs, jobs): exactly one directory config mount, no airflow.cfg or airflow_local_settings.py subPath, AIRFLOW_CONFIG set.
  • test_config_mounted_via_subpath_by_default — no values: historical airflow.cfg subPath, no local-settings subPath, no directory mount, no AIRFLOW_CONFIG.
  • test_local_settings_mounted_via_subpath_when_set — airflowLocalSettings set: both subPath mounts.

Notes

Backward compatible (default off).

This PR was created with the assistance of generative AI tools.

@salvamadrid

Copy link
Copy Markdown

+1

@Miretpl

Miretpl commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

The task fails with StartError and no application log (the container never starts). It's intermittent, cross-component, multi-node, and runtime-version-independent. See #33788.

The solution to the issue mentioned here is basically not to mount the /opt/airflow directory as a whole, and the solution is mentioned in the discussion by Jarek. If you have the same case as mentioned in the discussion, you can mount a volume at /opt/airflow/config (or a different location) and set the desired env variables/config options accordingly.

@diegoalvarez-clarity

Copy link
Copy Markdown
Contributor Author

Thanks @Miretpl — I think we're describing two different cases, so let me draw the line clearly.

The whole /opt/airflow scenario from #33788 is a user-supplied mount, and yes, mounting at /opt/airflow/config instead is the right fix there.

The race this PR targets is on the chart's own config mount. The shared airflow_config_mount helper unconditionally emits two subPath mounts — airflow.cfg and airflow_local_settings.py — onto the config volume. Those are what runc fails to bind under node/volume-lifecycle churn (kubelet volume reconstruction / node autoscaling):

error mounting ".../volume-subpaths/config/base/1" to rootfs at "/opt/airflow/airflow.cfg"

The container never starts, so the task hard-fails with no application log.

A values-level extraVolumeMounts at /opt/airflow/config is additive — it can't remove the helper's subPath mounts, so the race persists. There's no values-only way to drop them today, and that's the gap this PR closes.

mountConfigAsDir: true switches the shared helper from the two subPaths to a single directory mount at /opt/airflow/config + AIRFLOW_CONFIG. It's opt-in and default-off, so existing behavior is unchanged. helm template diff: subPath count 2 → 0, everything else byte-identical; airflow_local_settings.py stays at its default path via the directory mount.

@diegoalvarez-clarity

Copy link
Copy Markdown
Contributor Author

Hi @jedcunningham @hussein-awala @jscheffl @bugraoz93 @Miretpl 👋

Gentle nudge on this one when you get a chance. To recap where it stands: the PR adds an opt-in mountConfigAsDir flag (default false, so existing behavior is unchanged) that mounts the config ConfigMap as a directory instead of via subPath binds, which avoids the stale-inode StartError race we hit during node reconstruction / autoscaling.

@Miretpl — I followed up on your question about #33788 above: this targets the chart's own airflow.cfg / airflow_local_settings.py subPath mounts rather than user-supplied ones, since values-only extraVolumeMounts can't remove the helper's subPath mounts. Happy to expand or adjust if that framing doesn't fully address your concern.

Tests cover both the new directory-mount path and the preserved subPath behavior. I'm glad to rebase or make changes to help this move forward — thanks for taking a look! 🙏

@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions Bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Sep 14, 2026

@Miretpl Miretpl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR lacks many things from logic verification to the test coverage. As it is stated that there was AI used, please for the future verify carefully the content of the PR.

Comment thread chart/templates/_helpers.yaml Resolved
Comment thread chart/templates/_helpers.yaml Resolved
Comment thread chart/newsfragments/70180.feature.rst Resolved
Comment thread chart/values.schema.json Resolved
Comment thread chart/values.yaml Resolved
@Miretpl

Miretpl commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

@Miretpl — I followed up on your question about #33788 above: this targets the chart's own airflow.cfg / airflow_local_settings.py subPath mounts rather than user-supplied ones, since values-only extraVolumeMounts can't remove the helper's subPath mounts. Happy to expand or adjust if that framing doesn't fully address your concern.

So, as it is a different issue, the mentioned issue should not be mentioned in the description.

@Miretpl Miretpl added backport-to-chart/v1-2x-test Automatic backport to chart 1.2x maintenance branch and removed stale Stale PRs per the .github/workflows/stale.yml policy file labels Sep 17, 2026
@Miretpl Miretpl added this to the Airflow Helm Chart 1.23.0 milestone Sep 17, 2026
@diegoalvarez-clarity

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review, @Miretpl. All comments are addressed in three commits, one per area: the helper reuse (1342d42), the values and schema wording plus the newsfragment removal (8ef39ae), and the test rework (05572da). The directory-mount test now covers all ten templates that include the config mount, and the default-behaviour test renders with no values. On the airflow_local_settings.py point I replied on the thread with why the path is unchanged and added an assertion for it.

I've also removed the misleading reference to the other issue. Sorry for the confusion.

Ready for another look.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions Bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 3, 2026
@diegoalvarez-clarity

Copy link
Copy Markdown
Contributor Author

Hello @Miretpl. Would you mind taking a look at the changes so the PR is not closed due to staleness? I think I addressed everything, but let me know if that's not the case.

Thank you!

@github-actions github-actions Bot removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 4, 2026
@Miretpl

Miretpl commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

@diegoalvarez-clarity can you rebase the PR?

@diegoalvarez-clarity

diegoalvarez-clarity commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

@diegoalvarez-clarity can you rebase the PR?

Done! @Miretpl I need approval for the CI to run, probably because this is my first contribution to the chart. Feel free to merge it if CI passes after your approval.

Edit: The CI passed so the PR is ready to merge whenever.

Thank you!

diegoalvarez-clarity and others added 3 commits November 16, 2026 08:51
… mount race

The chart mounts airflow.cfg (and airflow_local_settings.py) into every component and
the KubernetesExecutor pod template via a subPath bind-mount of the config ConfigMap.
subPath + ConfigMap hits a known Kubernetes race: on node/volume lifecycle events
(kubelet volume reconstruction, node autoscaling) the pre-computed subPath source inode
goes stale and runc fails the mount at container init with a StartError and no
application log. It is intermittent, cross-component, and runtime-version-independent.

Add an opt-in mountConfigAsDir (default false, existing behaviour unchanged). When true,
the config ConfigMap is mounted as a directory at {airflowHome}/config and AIRFLOW_CONFIG
is set to {airflowHome}/config/airflow.cfg. A directory-projected ConfigMap volume has no
pre-computed subPath bind source, so it is immune to the race. airflow_local_settings.py
keeps its existing path.

Refs: apache#33788

Co-authored-by: diego15ag <11668493+diego15ag@users.noreply.github.com>
Sphinx spellcheck (chart docs) flagged "inode" and "runc". Reword
"stale-inode" to "stale-mount" and backtick ``runc`` so both are accepted,
keeping the description precise.

Co-authored-by: diego15ag <11668493+diego15ag@users.noreply.github.com>
The helper and the env var both derived {airflowHome}/config independently.
Make the helper return the unquoted path and quote at the two call sites, so
the config subdirectory is encoded once. Rendered output is unchanged.
Use the reviewer's wording for the values.yaml comment and the schema
description; the rationale lives in the PR description. Drop the
newsfragment as not needed.
Cover every component that includes the config mount helper, not only the
core five, and branch on kind for the CronJob pod path. Express the mount
and env assertions as jmespath filters and drop the length assertion that
checked nothing. Split the default-mount test so the default case renders
with no values, and move the airflowLocalSettings case to its own test.
@Miretpl
Miretpl merged commit 9e6126a into apache:main Nov 16, 2026
102 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport failed to create: chart/v1-2x-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
❌ chart/v1-2x-test Commit Link

You can attempt to backport this manually by running:

cherry_picker 9e6126a chart/v1-2x-test

This should apply the commit to the chart/v1-2x-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

@Miretpl

Miretpl commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

@diegoalvarez-clarity could you do manual backport?

@diegoalvarez-clarity

Copy link
Copy Markdown
Contributor Author

@diegoalvarez-clarity could you do manual backport?

@Miretpl done: #77194

The conflict was only in the tests. I also had to adapt one assertion, because the 1.x chart ships a default airflowLocalSettings, so it is mounted by default there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:helm-chart Airflow Helm Chart backport-to-chart/v1-2x-test Automatic backport to chart 1.2x maintenance branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants