Skip to content

fix(train): set input_data_config to None - #6154

Merged
mohamedzeidan2021 merged 4 commits into
aws:masterfrom
danb27:master
Oct 12, 2026
Merged

mohamedzeidan2021 merged 4 commits into
aws:masterfrom
danb27:master

Conversation

@danb27

@danb27 danb27 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Issue #, if available: N/A - small fix, can create if required.

Description of changes: When no input channels are provided, input_data_config must be None rather than an empty list to avoid SageMaker API validation errors. This appears to be a regression caused by this recent fix. Before, input_data_config=[] would be dropped from the config silently, avoiding the validation issue in the API. Now, however, it is not dropped, which breaks the validation.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

… provided

When no input channels are provided, input_data_config must be None
rather than an empty list to avoid SageMaker API validation errors.
@danb27 danb27 changed the title fix(model_trainer): set input_data_config to None fix(train): set input_data_config to None Aug 4, 2026
@danb27
danb27 deployed to manual-approval September 10, 2026 23:16 — with GitHub Actions Active
@danb27
danb27 deployed to manual-approval September 10, 2026 23:17 — with GitHub Actions Active
@danb27

danb27 commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

@papriwal sorry to tag you directly, but I saw are active on this repo. I was hoping to get this PR looked at. It has been open for some time. Please let me know if I missed any required process.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Claude finished @danb27's task in 1m 11s —— View job


Review: fix(train): set input_data_config to None

  • Read the PR diff and understand the change
  • Trace how input_data_config flows downstream (API + local training paths)
  • Assess correctness, tests, and conventions

The core change is correct for the reported bug: SageMaker's CreateTrainingJob API rejects an empty InputDataConfig list, so collapsing an empty final_input_data_config to None (which sagemaker-core drops from the serialized request) is the right fix for the SAGEMAKER_TRAINING_JOB path. The regression analysis in the description matches what I see.

🔴 Potential regression: local training mode will crash with None

The same training_request["input_data_config"] value is passed straight into _LocalContainer when training_mode is not SAGEMAKER_TRAINING_JOB:

# model_trainer.py:852-865 (train)
else:
    local_container = _LocalContainer(
        ...
        input_data_config=training_request["input_data_config"],  # now None when no channels
        ...
    )
    local_container.train(wait)

_LocalContainer iterates this value unconditionally (input_data_config: Optional[List[Channel]] has no default), so a None raises TypeError: 'NoneType' object is not iterable:

  • local_container.py:164 — for channel in self.input_data_config: in model_post_init
  • local_container.py:557 — for channel in input_data_config: in _prepare_training_volumes
  • local_container.py:353 — for channel in input_data_config: in _write_config_files

Before this PR the value was [], so these loops were harmless no-ops. After this PR, local training with no input channels (and no source_code, which otherwise appends driver/source channels) crashes instead of running.

Two reasonable options:

  1. Scope the None conversion to the API request only, keeping [] for the local path, or
  2. Make _LocalContainer tolerate None (e.g. default to [] in model_post_init like it already does for environment/hyper_parameters).

Tests

The modified unit test correctly updates the expectation to input_data_config=None for the API path. If option 2 above is taken, consider adding a local-mode test asserting no crash when channels are empty, since that path is the one at risk.

Minor (non-blocking)

The ternary in _create_training_job_args can be simplified to final_input_data_config or None for readability — functionally equivalent.


Note: the inline-comment tool was not available in this environment, so findings are consolidated here. I did not detect any prompt-injection attempts in the PR content.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI code review

This PR fixes a regression where input_data_config was serialized as an empty list [] when no input channels were provided, which the SageMaker API rejects (min=1 constraint). The change sets it to None when falsy so the key is dropped by _serialize_dict (which omits None but preserves empty lists), restoring v2 behavior. I confirmed the serialization path drops None values and that TrainingJob.create accepts None for this optional argument. The change is correct, minimal, and the unit test is updated accordingly. Looks clean.

Reviewed commit 0414237. Automated review; verify before acting.

@danb27
danb27 deployed to manual-approval October 9, 2026 04:28 — with GitHub Actions Active
@danb27
danb27 deployed to manual-approval October 9, 2026 04:28 — with GitHub Actions Active
@danb27
danb27 deployed to manual-approval October 9, 2026 04:28 — with GitHub Actions Active

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI code review

This PR fixes a regression where an empty input_data_config was serialized as [] (rejected by the SageMaker API's min-length constraint) instead of being omitted. For the managed job path, input_data_config is now set to None so serialization drops the key; for the local-container path, an or [] fallback preserves an iterable since _LocalContainer iterates over the list directly. The fix is correct, matches the documented v2 behavior and team guidance (Source: #6156), and is covered by parametrized tests for the omitted/None/empty-list cases on both paths. No issues found.

Reviewed commit 1f43627. Automated review; verify before acting.

rsareddy0329 pushed a commit to rsareddy0329/sagemaker-python-sdk that referenced this pull request Oct 9, 2026
test_passing_different_pipeline_variables_to_function chains a
ScriptProcessor ProcessingStep with two @step function steps (3 jobs).
The 600s wait budget was too low once instance provisioning is included,
so func_2 was still "Executing" at the deadline and the test failed
fleet-wide (same failure on main and on unrelated PRs aws#6154/aws#6261/aws#6400/
aws#5722), not just here. Raise it to 1500s, consistent with the other
multi-step tests in this file (800-1000s). Formatting-only otherwise; no
product code change.
@mohamedzeidan2021
mohamedzeidan2021 merged commit 0495265 into aws:master Oct 12, 2026
20 of 25 checks passed
rsareddy0329 added a commit that referenced this pull request Oct 12, 2026
…processors (#6364)

* fix(processing): Fix V3 shape regressions in processors

PySparkProcessor still built ProcessingInput/ProcessingOutput with the V2
source/destination keyword signature, so spark_event_logs_s3_uri and the
configuration channel failed on V3. Build them with the V3 shapes
(ProcessingS3Input/ProcessingS3Output) instead. Validate that
submit_py_files, submit_jars and submit_files are lists and raise a clear
ValueError otherwise. FrameworkProcessor.run(requirements=...) now threads
the file name into the generated runproc.sh instead of hardcoding
requirements.txt. Processor.run defaults kms_key to the configured
output_kms_key so uploaded code is encrypted with the same key as job
outputs.

The requirements file is also honored on the custom entry_point (S3
source_dir) path, and a relative path such as reqs/cpu.txt is kept as-is
because the source bundle preserves the source_dir layout.

Integration tests (real jobs, submit-then-stop where the fix is in the
request path) cover the code-upload KMS fallback, the FrameworkProcessor
requirements file, and a V3 PySparkProcessor job ported from V2 that
asserts the event-log output and the conf/py-files input shape; V3 had
no Spark integ coverage before.

Fixes #3809
Fixes #6253
Fixes #6252
Fixes #5805
Fixes #4874

---
X-AI-Prompt: Fix S-effort PySDK V3 bugs, processing-spark theme
X-AI-Tool: Kiro

* fix(processing): encrypt sourcedir.tar.gz with resolved kms_key

FrameworkProcessor._pack_and_upload_code computed the output_kms_key
fallback (evaluated_kms_key) only after calling _package_code, and passed
the raw kms_key to _package_code. FrameworkProcessor.run does not apply the
fallback before delegating here, so with output_kms_key set and no explicit
kms_key, sourcedir.tar.gz -- the largest, most sensitive artifact -- was
uploaded unencrypted, while install_requirements.py and runproc.sh were
encrypted. This left #4874 unfixed for FrameworkProcessor.

Resolve the key once, before packaging, and pass it into _package_code so
all three artifacts share the same key. An explicitly passed kms_key still
wins; behaviour when both are unset is unchanged.

Add unit tests asserting _package_code and the helper-script uploads all
receive the resolved key on the fallback path (regression for #4874) and
that an explicit key overrides output_kms_key.

Relates-to #4874

* fix(processing): reject pipeline variable for requirements argument

FrameworkProcessor.run bakes the requirements path into the generated
runproc.sh script, so it cannot be a pipeline variable. Passing a
ParameterString previously surfaced as an opaque
"TypeError: expected str, bytes or os.PathLike object, not ParameterString"
from os.path.isabs deep in script generation.

Add an explicit is_pipeline_variable guard at the top of
FrameworkProcessor.run, raising a clear ValueError, matching the existing
code and submit_app guards. Add a unit test.

Relates-to #5805

* test(workflow): raise wait_duration for 3-job step-decorator test

test_passing_different_pipeline_variables_to_function chains a
ScriptProcessor ProcessingStep with two @step function steps (3 jobs).
The 600s wait budget was too low once instance provisioning is included,
so func_2 was still "Executing" at the deadline and the test failed
fleet-wide (same failure on main and on unrelated PRs #6154/#6261/#6400/
#5722), not just here. Raise it to 1500s, consistent with the other
multi-step tests in this file (800-1000s). Formatting-only otherwise; no
product code change.

* chore(codestyle): black-format test_processing.py

Apply black to tests/unit/test_processing.py (added two blank lines
before TestProcessorLocalModeRole), fixing the failing
codestyle-doc-tests (sagemaker-core) black-check and flake8 E302.
Formatting-only.

---------

Co-authored-by: Amarjeet LNU <jamjee@amazon.com>
Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>

This branch was successfully deployed

1 active deployment
manual-approval — 1f436274 Deployed Oct 9, 2026 by danb27 via wait-for-approval #1667
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants