Repository navigation
Tracking issue: standard library findings from a property-testing run (replaces #158695 to #158785) #158786
Description
As @ZeroIntensity asked in #158748 (comment), this is the single issue for the reports I filed in bulk on 4 October (UTC), #158695 to #158785. I'm sorry for the flood: my tool filed them one after another and I should not have let it. It now files nothing here while this issue is open, and never more than one report per project at a time.
The findings are compiled in one place, with a table of every result, its verdict and a link to its reproducer and output: https://github.com/augusto-rehfeldt/bugforge-results (I used that repository rather than a gist; it is the same list.)
What they are: an automated property-testing tool I run (bugforge) reads a module's documentation, searches for an input that breaks a documented property, and writes a standalone reproducer. Every reproducer in the list was executed on Python 3.14.6 on Windows, and its output is in the linked folder and in the closed issue.
What they are not: reviewed. The reports were written by a language model and nobody, including me, has checked each one by hand. I expect a good share to be duplicates, intended behaviour, or inputs too contrived to matter, and several findings in the same module are probably one defect.
I won't open separate issues or pull requests for any of these unless a maintainer of the module asks for one here. If the list is not wanted at all, please close this and nothing more will follow.
(This text was drafted with an AI assistant.)
Activity
picnixz commentedon Oct 4, 2026
You are meant to. Maintainers do not have the time to make that so please do this work first.
hugovk commentedon Oct 4, 2026
See https://devguide.python.org/getting-started/ai-tools/
terryjreedy commentedon Oct 4, 2026
It is good that bugforge provides standalone reproducers.
However, it is buggy or even a bit fraudulent in its overall design. The Python docs are written in standard, sometime ambiguous English, to be read by humans with some minimum intelligence, knowledge, and good faith. In https://docs.python.org/3/reference/introduction.html, Guido wrote
The English in the language reference is backed by a humanized version of the formal specifications from which the Python parser is generated. The only backups in the library reference are short examples. Different module docs assume different backgrounds.
LLMs are today the hypothesized reader from Mars. Worse, bugforge processes and interprets single sentences out of context, not whole paragraphs and sections, in light of assumed backgrounds.
Another bug is that bugforge issues 'verdicts' as to whether a finding is a bug. Golly gee, it appears to be programmed to call everything a bug unless it can see otherwise, which it is generally incapable of doing. By constrast, #154760 was produced by a human who reviewed and revised the 'ai' findings, and specifies that findings are not bugs until a human maintainer (me in the case of IDLE) decides that it is. This report also tried to avoid reporting duplicates of existing issues.
I looked at the linked prior single issue about random.choices. The sentence is “If a weights sequence is specified, selections are made according to the relative weights.” The bugforge interpretaton is "Equal relative weights imply equal selection probabilities." Both are true in the assumed context of real numbers. Both are practically true for any numbers entered by the target audience, sane humans trying to do some calculation. I verified that the example code find no problem if the weights are 1,1 instead of 2**-1074, 2**-1074.
The bugforge test: enter subnormal floats, which are known not to obey normal arithmetic rules. Example:
This might explain how a probability that should have been 1/2 became 1/4 and might therefore be equivalent to the issue. Until a human who is competent to read the IEEE floating point RFC and its section on subnormals says otherwise, I presume that the above and the issues are not bugs.
Here is another IEEE floating point oddity.
NaN and OverflowError are other possible responses. I believe this example is equivalent to another of the findings. Not a bug, but a distraction from us working on our backlog of real issues.