Skip to content

Read and write bundle and datalist files as UTF-8 - #9282

Open
CodeAvk wants to merge 2 commits into
Project-MONAI:devfrom
CodeAvk:fix/9277-utf8-file-encoding
Open

CodeAvk wants to merge 2 commits into
Project-MONAI:devfrom
CodeAvk:fix/9277-utf8-file-encoding

Conversation

@CodeAvk

@CodeAvk CodeAvk commented Nov 15, 2026 •

Copy link
Copy Markdown

Fixes #9277.

Description

On Windows, Python often uses cp1252 as the default file encoding instead of UTF-8. Some parts of the code, including ConfigParser.load_config_file / export_config_file, the bundle scripts (_check_monai_version, init_bundle, _add_model_card_metadata), BundleWorkflow (properties_path), and the Decathlon datalist helpers, were opening files without specifying an encoding.

Because of this, UTF-8 config files containing special characters could either raise a UnicodeDecodeError or be read incorrectly. For example, "Größe" was displayed as "Größe". Writing files with ensure_ascii=False or allow_unicode=True could also raise a UnicodeEncodeError.

This PR fixes the issue by explicitly setting encoding="utf-8" wherever these files are opened.

I've also added tests that simulate cp1252 as the default encoding, so the issue can be caught on any OS, including Linux CI.

I reproduced the issue and verified the fix on Windows 11 with Python 3.13.

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • New tests added to cover the changes.

Note: files that are not valid UTF-8 (for example saved as cp1252 or GBK) now raise UnicodeDecodeError instead of being read with the system's default encoding.

Fixes Project-MONAI#9277.

ConfigParser, the bundle scripts, BundleWorkflow and the Decathlon
datalist helpers opened their JSON, YAML and text files without an
encoding, so Python used the system default. On Windows that is often
cp1252, where a UTF-8 file with non-ASCII text either raises
UnicodeDecodeError or is decoded into wrong characters without an
error (e.g. "Größe" is read as "Größe"), and writing with
ensure_ascii=False or allow_unicode=True raises UnicodeEncodeError.

Open these files with encoding="utf-8". The new tests patch open() to
use cp1252 as the default, so they fail without the fix on any OS.

Signed-off-by: CodeAvk <avksmlavk@gmail.com>
@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review.

Walkthrough

Bundle configuration, metadata, model cards, workflow properties, and Decathlon datalist file operations now specify UTF-8. Regression tests simulate a cp1252 default and check non-ASCII content.

Estimated code review effort: 2 (Simple) | ~10 minutes

Severity of issue fixed: Medium

Merge Risk: Minimal · up to 18107

This change makes bundle and Decathlon datalist file reads and writes use UTF-8 consistently, so non-ASCII content works on systems with a legacy default encoding. Files saved in a non-UTF-8 encoding such as cp1252 will now fail to load with a decode error, which is intended. No merge-blocking risk was identified.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: bundle and datalist files are now read and written as UTF-8. This matches the raw summary, which shows explicit encoding="utf-8" added to file opens in config_parser, …
Linked Issues check Passed [#9277] Objective: read and write bundle config, bundle script, BundleWorkflow properties, and Decathlon datalist JSON as UTF-8 regardless of the system default encoding. Verified at head by direct so…
Out of Scope Changes check Passed Per the whole-PR summary, all changed source files are those named in #9277: config_parser.py, scripts.py, workflows.py, and decathlon_datalist.py. Each source change adds an explicit encoding argumen…
Description check Passed The description covers the required template elements. It names the linked issue (Fixes #9277), gives a clear description of the encoding fix and the affected files, and checks the "New tests added" b…
Full details: Title check

Explanation

The title names the main change: bundle and datalist files are now read and written as UTF-8. This matches the raw summary, which shows explicit encoding="utf-8" added to file opens in config_parser, scripts, workflows, and decathlon_datalist. It is specific and concise.

Full details: Linked Issues check

Explanation

[#9277] Objective: read and write bundle config, bundle script, BundleWorkflow properties, and Decathlon datalist JSON as UTF-8 regardless of the system default encoding. Verified at head by direct source read: monai/data/decathlon_datalist.py. load_decathlon_datalist and load_decathlon_properties open files with encoding="utf-8". create_cross_validation_datalist writes with encoding="utf-8". Per the whole-PR summary (merge base 1dba735 to head 1810703): ConfigParser.load_config_file and export_config_file (config_parser.py), _check_monai_version, init_bundle, _add_model_card_metadata (scripts.py), and BundleWorkflow properties_path (workflows.py) now specify UTF-8. Diffs for config_parser.py, scripts.py, and workflows.py were unavailable (changed blobs missing). A head read of config_parser.py was truncated before the load and export functions. These four files rest on the summary only. Tests: the summary reports a cp1252-default open simulation in tests/bundle/test_config_parser.py and tests/data/test_load_decathlon_datalist.py. Those test files were not read directly. Incremental changes since previous head 844d9af are not separately inspectable. The prior conclusion remains supported by the summary and the verified Decathlon code.

Full details: Out of Scope Changes check

Explanation

Per the whole-PR summary, all changed source files are those named in #9277: config_parser.py, scripts.py, workflows.py, and decathlon_datalist.py. Each source change adds an explicit encoding argument, and the summary reports no behavior or public API changes. The test changes support the encoding objective. Decathlon source at head matches this scope. Diffs for three source files were unavailable, so their scope rests on the summary. No unrelated change is demonstrated.

Full details: Description check

Explanation

The description covers the required template elements. It names the linked issue (Fixes #9277), gives a clear description of the encoding fix and the affected files, and checks the "New tests added" box. It also explains the behavior change for non-UTF-8 files. Remaining template items are unchecked, but they are optional, and the template's Types of changes boxes are answered.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 844d9af and 1810703.

Files selected for processing: 1

  • tests/bundle/test_config_parser.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

@CodeAvk
CodeAvk marked this pull request as ready for review November 15, 2026 15:27
@CodeAvk

CodeAvk commented Nov 15, 2026

Copy link
Copy Markdown
Author

Thanks @ericspod! It's ready for review now. All checks pass, and I tested it locally on Windows 11.

@kesonglab kesonglab left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — this addresses a real problem on Windows (cp1252 as the default locale encoding). The direction and the set of touched call sites look right.

Some suggestions:

  1. Consider a repo-wide sweep rather than point fixes. This class of bug recurs; grep -rn 'open(' monai/ | grep -v encoding= or enabling ruff's PLW1514 (unspecified-encoding) would catch remaining sites and prevent regressions. Worth confirming nothing is left in monai/bundle/ and monai/data/.

  2. This is a behaviour change for non-UTF-8 inputs. Files that previously loaded under cp1252/GBK will now raise UnicodeDecodeError. That is the correct contract for JSON/YAML, but it is user-visible and probably deserves a note in the PR description.

  3. Newline handling on write. Without newline="" / newline="\n", Windows writes \r\n. If any bundle content is hashed or compared textually, that produces different bytes. newline="\n" may be worth setting on the write sites.

  4. Test coverage: a negative case (a file with non-UTF-8 bytes, now raising) would pin down the new contract.

LGTM modulo the above.

A cp1252-encoded config used to load when cp1252 was the default encoding. It now raises UnicodeDecodeError, which this test pins down.

Signed-off-by: CodeAvk <avksmlavk@gmail.com>
@CodeAvk

CodeAvk commented Nov 20, 2026

Copy link
Copy Markdown
Author

Thanks for the review!

  1. I kept this PR to the bundle/datalist files from Bundle configs and datalists fail to handle non-UTF-8 default encodings on Windows #9277. In monai/bundle and monai/data, the remaining text-mode opens are csv_saver.py, an empty hash file and os.devnull. csv_saver.py could be a follow-up PR.
  2. Good point, I've added a note to the description: non-UTF-8 files now raise UnicodeDecodeError instead of being read with the locale encoding.
  3. This PR doesn't change newline handling (text mode wrote \r\n on Windows before as well), so I'd leave that for a separate discussion.
  4. Added a test that a cp1252-encoded config now raises UnicodeDecodeError.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bundle configs and datalists fail to handle non-UTF-8 default encodings on Windows

2 participants