Repository navigation
Generate opcode metadata from bytecodes.c instead of opcode.py #105481
Description
We would ideally have bytecodes.c as the single source of truth about opcodes. So opcode.py and the code generated from it should be replaced by alternatives from the cases_generator, if we can.
Linked PRs
- gh-105481: add flags to each instr in the opcode metadata table, to replace opcode.hasarg/hasname/hasconst #105482
- gh-105481: add pseudo-instructions to the cases DSL #105506
- gh-105481: Fix types and bug for pseudos #105788
- gh-105481: add HAS_JUMP flag to opcode metadata #105791
- gh-105481: remove HAS_ARG, HAS_CONST, IS_JUMP_OPCODE, IS_PSEUDO_OPCODE and replace by their new versions #105865
- gh-105481: generate _specializations and _specialized_instructions from bytecodes.c #105913
- gh-105481: refactor instr flag related code into a new InstructionFlags class #105950
- gh-105481: move Python/opcode_metadata.h to Include/internal/pycore_opcode_metadata.h #106673
- gh-105481: expose opcode metadata via the _opcode module #106688
- gh-105481: Generate the opcode lists in dis from data extracted from bytecodes.c #106758
- gh-105481: do not auto-generate pycore_intrinsics.h #106913
- gh-105481: add OPCODE_IS_INSTRUMENTED (generated from bytecodes.c) to replace MIN_INSTRUMENTED_OPCODE (defined in opcode.py) #107276
- gh-105481: remove dependency of _inline_cache_entries on opname #107339
- gh-105481: the ENABLE_SPECIALIZATION flag does not need to be generated by the build script, or exposed in opcode.py #107534
- gh-105481: combine regen-opcode-targets with regen-opcode to avoid calculating the specialized opcodes in two places #107540
- gh-105481: simplify definition of pseudo ops in Lib/opcode.py #107561
- GH-105481: Mark more files as generated #107598
- gh-105481: split opcode_ids.h out of opcode.h so that it can be generated separately #107866
- gh-105481: reduce repetition in opcode metadata generation code #107942
- gh-105481: generate op IDs from bytecode.c instead of hard coding them in opcode.py #107971
- gh-105481: generate _PyOpcode_Deopt and _PyOpcode_OpName in generate_cases instead of generate_opcode_h #108025
- gh-105481: fix out of date comment #108079
- gh-105481: opcode.h is no longer generated during the build #108080
- gh-105481: remove regen-opcode. Generated _PyOpcode_Caches in regen-cases. #108367
Activity
pythongh-105481: add flags to each instr in the opcode metadata table…
pythongh-105481: add pseudo-instructions to the cases DSL
gh-105481: add pseudo-instructions to the bytecodes DSL (#105506)
gh-105481: add flags to each instr in the opcode metadata table, to r…
pythongh-105481: add HAS_JUMP flag to opcode metadata
pythongh-105481: add HAS_JUMP flag to opcode metadata
gh-105481: Fix types and a bug for pseudos (#105788)
gh-105481: add HAS_JUMP flag to opcode metadata (#105791)
iritkatriel commentedon Jun 15, 2023
@gvanrossum The old macros HAS_ARG and HAS_CONST are exposed in Include/opcode.h (and have been for a long time). Does this mean that before we can remove them, we need to expose the opcode_metadata.h file there as well? We should also mark all of this as unstable API (how do we do that?)
Merge branch 'main' into modulelocals
gvanrossum commentedon Jun 16, 2023
Neither
HAS_ARGnorHAS_CONSTis mentioned in the docs at all. I am of the opinion that in this case this implies they are not public, and anyone using them should not be surprised if they disappear. (It would be more of an issue if they turned into lies.)opcode.h is mentioned only once in the docs, as the source of truth for dis.py. (It is also mentioned twice in Misc/NEWS.d/, but that's really just a changelog.)
I do think we ought to provide similar functionality to debuggers.
Info about declaring unstable APIs comes from PEP 649, which links to https://devguide.python.org/developer-workflow/c-api/#c-api. It looks like you must use the
PyUnstable_prefix (e.g.PyUnstable_BytecodeHasArg), and it ought to be in a .h file under Include/cpython/.iritkatriel commentedon Jun 16, 2023
Do you mean to expose the stuff in opcode_metadata.h file through some unstable c api?
gvanrossum commentedon Jun 16, 2023
Yeah, to the extent that it’s useful, in function form (so the data structure is not public).
Or we can wait until someone asks.
32 remaining items
pythongh-105481: allocate op IDs in cases generator. Generate _PyOpco…
pythongh-105481: generate _PyOpcode_Deopt and _PyOpcode_OpName in gen…
gh-105481: generate op IDs from bytecode.c instead of hard coding the…
pythongh-105481: fix out of date comment
pythongh-105481: opcode.h is no longer generated during the build
gh-105481: fix out of date comment (#108079)
gh-105481: opcode.h is no longer generated during the build (#108080)
pythongh-105481: remove regen-opcode. Generated _PyOpcode_Caches in r…
gh-105481: remove regen-opcode. Generated _PyOpcode_Caches in regen-c…
vstinner commentedon Nov 8, 2023
@encukou wrote "Unstable API needs documentation and tests."
I reopen this issue and close gh-107149.
iritkatriel commentedon Nov 13, 2023
I'm not sure it was the right move to rename
_PyUnstable_GetUnaryIntrinsicNametoPyUnstable_GetUnaryIntrinsicName, etc.I created them as private because I did not intend to document them as a public API (we need a reason to do that). I put Unstable in the name because it's one of those things that can change all the time. If we don't want the
_PyUnstable_prefix then let's remove theUnstableand leave the_until we have a reason/decision to make these part of the public C API.iritkatriel commentedon Nov 13, 2023
I'm moving this discussion back to #107149, and closing this, because this PR was about a large change, and #107149 is about the naming.