Repository navigation
Conversation
Fixes Project-MONAI#9277. ConfigParser, the bundle scripts, BundleWorkflow and the Decathlon datalist helpers opened their JSON, YAML and text files without an encoding, so Python used the system default. On Windows that is often cp1252, where a UTF-8 file with non-ASCII text either raises UnicodeDecodeError or is decoded into wrong characters without an error (e.g. "Größe" is read as "Größe"), and writing with ensure_ascii=False or allow_unicode=True raises UnicodeEncodeError. Open these files with encoding="utf-8". The new tests patch open() to use cp1252 as the default, so they fail without the fix on any OS. Signed-off-by: CodeAvk <avksmlavk@gmail.com>
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
|
Thanks @ericspod! It's ready for review now. All checks pass, and I tested it locally on Windows 11. |
kesonglab
left a comment
There was a problem hiding this comment.
Thanks — this addresses a real problem on Windows (cp1252 as the default locale encoding). The direction and the set of touched call sites look right.
Some suggestions:
-
Consider a repo-wide sweep rather than point fixes. This class of bug recurs;
grep -rn 'open(' monai/ | grep -v encoding=or enabling ruff'sPLW1514(unspecified-encoding) would catch remaining sites and prevent regressions. Worth confirming nothing is left inmonai/bundle/andmonai/data/. -
This is a behaviour change for non-UTF-8 inputs. Files that previously loaded under cp1252/GBK will now raise
UnicodeDecodeError. That is the correct contract for JSON/YAML, but it is user-visible and probably deserves a note in the PR description. -
Newline handling on write. Without
newline=""/newline="\n", Windows writes\r\n. If any bundle content is hashed or compared textually, that produces different bytes.newline="\n"may be worth setting on the write sites. -
Test coverage: a negative case (a file with non-UTF-8 bytes, now raising) would pin down the new contract.
LGTM modulo the above.
A cp1252-encoded config used to load when cp1252 was the default encoding. It now raises UnicodeDecodeError, which this test pins down. Signed-off-by: CodeAvk <avksmlavk@gmail.com>
|
Thanks for the review!
|
Fixes #9277.
Description
On Windows, Python often uses cp1252 as the default file encoding instead of UTF-8. Some parts of the code, including
ConfigParser.load_config_file/export_config_file, the bundle scripts (_check_monai_version,init_bundle,_add_model_card_metadata),BundleWorkflow(properties_path), and the Decathlon datalist helpers, were opening files without specifying an encoding.Because of this, UTF-8 config files containing special characters could either raise a
UnicodeDecodeErroror be read incorrectly. For example, "Größe" was displayed as "Größe". Writing files withensure_ascii=Falseorallow_unicode=Truecould also raise aUnicodeEncodeError.This PR fixes the issue by explicitly setting
encoding="utf-8"wherever these files are opened.I've also added tests that simulate cp1252 as the default encoding, so the issue can be caught on any OS, including Linux CI.
I reproduced the issue and verified the fix on Windows 11 with Python 3.13.
Types of changes
Note: files that are not valid UTF-8 (for example saved as cp1252 or GBK) now raise
UnicodeDecodeErrorinstead of being read with the system's default encoding.