Repository navigation
Allow mounting the Airflow config as a directory to avoid the subPath mount race - #70180
Conversation
|
+1 |
The solution to the issue mentioned here is basically not to mount the |
|
Thanks @Miretpl — I think we're describing two different cases, so let me draw the line clearly. The whole The race this PR targets is on the chart's own config mount. The shared
The container never starts, so the task hard-fails with no application log. A values-level extraVolumeMounts at
|
|
Hi @jedcunningham @hussein-awala @jscheffl @bugraoz93 @Miretpl 👋 Gentle nudge on this one when you get a chance. To recap where it stands: the PR adds an opt-in mountConfigAsDir flag (default false, so existing behavior is unchanged) that mounts the config ConfigMap as a directory instead of via subPath binds, which avoids the stale-inode StartError race we hit during node reconstruction / autoscaling. @Miretpl — I followed up on your question about #33788 above: this targets the chart's own airflow.cfg / airflow_local_settings.py subPath mounts rather than user-supplied ones, since values-only extraVolumeMounts can't remove the helper's subPath mounts. Happy to expand or adjust if that framing doesn't fully address your concern. Tests cover both the new directory-mount path and the preserved subPath behavior. I'm glad to rebase or make changes to help this move forward — thanks for taking a look! 🙏 |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
Miretpl
left a comment
There was a problem hiding this comment.
This PR lacks many things from logic verification to the test coverage. As it is stated that there was AI used, please for the future verify carefully the content of the PR.
So, as it is a different issue, the mentioned issue should not be mentioned in the description. |
|
Thanks for the thorough review, @Miretpl. All comments are addressed in three commits, one per area: the helper reuse (1342d42), the values and schema wording plus the newsfragment removal (8ef39ae), and the test rework (05572da). The directory-mount test now covers all ten templates that include the config mount, and the default-behaviour test renders with no values. On the airflow_local_settings.py point I replied on the thread with why the path is unchanged and added an assertion for it. I've also removed the misleading reference to the other issue. Sorry for the confusion. Ready for another look. |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
|
Hello @Miretpl. Would you mind taking a look at the changes so the PR is not closed due to staleness? I think I addressed everything, but let me know if that's not the case. Thank you! |
|
@diegoalvarez-clarity can you rebase the PR? |
05572da to
54c61b8
Compare
Done! @Miretpl I need approval for the CI to run, probably because this is my first contribution to the chart. Feel free to merge it if CI passes after your approval. Edit: The CI passed so the PR is ready to merge whenever. Thank you! |
… mount race
The chart mounts airflow.cfg (and airflow_local_settings.py) into every component and
the KubernetesExecutor pod template via a subPath bind-mount of the config ConfigMap.
subPath + ConfigMap hits a known Kubernetes race: on node/volume lifecycle events
(kubelet volume reconstruction, node autoscaling) the pre-computed subPath source inode
goes stale and runc fails the mount at container init with a StartError and no
application log. It is intermittent, cross-component, and runtime-version-independent.
Add an opt-in mountConfigAsDir (default false, existing behaviour unchanged). When true,
the config ConfigMap is mounted as a directory at {airflowHome}/config and AIRFLOW_CONFIG
is set to {airflowHome}/config/airflow.cfg. A directory-projected ConfigMap volume has no
pre-computed subPath bind source, so it is immune to the race. airflow_local_settings.py
keeps its existing path.
Refs: apache#33788
Co-authored-by: diego15ag <11668493+diego15ag@users.noreply.github.com>
Sphinx spellcheck (chart docs) flagged "inode" and "runc". Reword "stale-inode" to "stale-mount" and backtick ``runc`` so both are accepted, keeping the description precise. Co-authored-by: diego15ag <11668493+diego15ag@users.noreply.github.com>
The helper and the env var both derived {airflowHome}/config independently.
Make the helper return the unquoted path and quote at the two call sites, so
the config subdirectory is encoded once. Rendered output is unchanged.
Use the reviewer's wording for the values.yaml comment and the schema description; the rationale lives in the PR description. Drop the newsfragment as not needed.
Cover every component that includes the config mount helper, not only the core five, and branch on kind for the CronJob pod path. Express the mount and env assertions as jmespath filters and drop the length assertion that checked nothing. Split the default-mount test so the default case renders with no values, and move the airflowLocalSettings case to its own test.
54c61b8 to
7fb355b
Compare
Backport failed to create: chart/v1-2x-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 9e6126a chart/v1-2x-testThis should apply the commit to the chart/v1-2x-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
|
@diegoalvarez-clarity could you do manual backport? |
The conflict was only in the tests. I also had to adapt one assertion, because the 1.x chart ships a default |
Why
The chart mounts
airflow.cfg(andairflow_local_settings.py) into every component and the KubernetesExecutor pod template via asubPathbind-mount of theconfigConfigMap.subPath+ ConfigMap hits a known Kubernetes race: on node/volume lifecycle events (kubelet volume reconstruction, node autoscaling) the pre-computed subPath source inode goes stale and runc fails the mount at container init:The task fails with
StartErrorand no application log (the container never starts). It's intermittent, cross-component, multi-node, and runtime-version-independent.What
Add an opt-in
mountConfigAsDir(defaultfalse; existing behaviour unchanged). Whentrue, theconfigConfigMap is mounted as a directory at{airflowHome}/config(no subPath) andAIRFLOW_CONFIGis set to{airflowHome}/config/airflow.cfg. A directory-projected ConfigMap volume has no pre-computed subPath bind source, so it's immune to the race.airflow_local_settings.pykeeps its default path, so local-settings loading is unaffected. The separatepod_template_file.yamlsubPath mount is out of scope.Tests
test_mount_config_as_dir— flag on, across all ten templates that include the config mount (deployments, statefulsets, cronjobs, jobs): exactly one directoryconfigmount, noairflow.cfgorairflow_local_settings.pysubPath,AIRFLOW_CONFIGset.test_config_mounted_via_subpath_by_default— no values: historicalairflow.cfgsubPath, no local-settings subPath, no directory mount, noAIRFLOW_CONFIG.test_local_settings_mounted_via_subpath_when_set—airflowLocalSettingsset: both subPath mounts.Notes
Backward compatible (default off).
This PR was created with the assistance of generative AI tools.