Simplify lpython.py #2274

Activity

1. rebcabin commented on Aug 13, 2023

   

   rebcabin

   on Aug 13, 2023

   Contributor

   More actions

   I think the only way to robustly support pointers is a "heavy" API interface, and it's probably not worth the effort. I think, design-wise, we should make beautiful array types and then discourage people from writing C-flavored Python.

   👍React with 👍1Reacted by Ondřej Čertík
2. certik commented on Aug 13, 2023

   

   certik

   on Aug 13, 2023

   Last edited by certik

   ContributorAuthor

   More actions

   I think that lpython.py should only contain:

   • new type annotations, such as these:

     lpython/src/runtime/lpython/lpython.py

     Lines 80 to 98 in 8df052e

     	i1 = Type("i1")
     	i8 = Type("i8")
     	i16 = Type("i16")
     	i32 = Type("i32")
     	i64 = Type("i64")
     	u8 = Type("u8")
     	u16 = Type("u16")
     	u32 = Type("u32")
     	u64 = Type("u64")
     	f32 = Type("f32")
     	f64 = Type("f64")
     	c32 = Type("c32")
     	c64 = Type("c64")
     	CPtr = Type("c_ptr")
     	Const = ConstType("Const")
     	Callable = Type("Callable")
     	Allocatable = Type("Allocatable")
     	Pointer = PointerType("Pointer")
     	S = Type("S")

   • functions that implement some operation that is robustly and cleanly implemented using CPython and its own intrinsic types, an example is:

     lpython/src/runtime/lpython/lpython.py

     Lines 760 to 761 in 8df052e

     	def bitnot(x, bitsize):
     	return (~x) % (2 ** bitsize)

   • decorators that are implemented in a clean and robust way, some examples are: @overload, @pythoncall, @ccall, @lpython.

   If done right, the decorators would be used in low level code where you need to interface with CPython or C or need to use overloads (not very common). The custom functions like bitnot are very rare, most code would not use it. That leaves the type annotations --- we can later support some more default CPython annotations as well, perhaps the user can create them. Then people would not need import lpython at all. We would just compile regular CPython code that runs in CPython without any non-standard lpython.py library. Perhaps we can support numpy types like int32. For interfacing C, we could just support ctypes directly (not as nice syntax, but we could do it). And pythoncall is just a small decorator, and perhaps there is an even simpler way to do it, for example we can annotate a function (with an empty decorator) and interpret the insides as CPython. This would be optional, the lpython module provides a convenience way, but if you don't want to introduce this dependency into a larger project, we should support a "non-lpython module mode" as well.

   What we do not want to do:

   • depend on internals of ctypes and hoping they can "emulate" pointers, an example is

     lpython/src/runtime/lpython/lpython.py

     Lines 594 to 616 in 8df052e

     	def c_p_pointer(cptr, targettype, targetshape=None):
     	targettype_ptr = convert_type_to_ctype(targettype)
     	if isinstance(targettype, Array):
     	if targetshape is None:
     	raise ValueError("target shape must be "
     	"provided if target type is an array.")
     	# TODO: Add support for multi-dimensional shape of target variable
     	if py_is_dataclass(targettype._type):
     	return ctypes.cast(cptr.value, ctypes.py_object).value
     	newa = ctypes.cast(cptr, targettype_ptr)
     	return newa
     	else:
     	if py_is_dataclass(targettype):
     	if cptr.value is None:
     	return None
     	return ctypes.cast(cptr, ctypes.py_object).value
     	targettype_ptr = ctypes.POINTER(targettype_ptr)
     	newa = ctypes.cast(cptr, targettype_ptr)
     	if is_ctypes_Structure(targettype):
     	# return after wrapping newa inside PointerToStruct
     	return PointerToStruct(newa)
     	return newa

   • Return wrapped entities, such as

     lpython/src/runtime/lpython/lpython.py

     Line 615 in 8df052e

     return PointerToStruct(newa)

     .

   Instead, we need to design the subset of CPython in such a way that we can just use native CPython constructs. In particular, if it is an array, we should just provide some function that converts a CPtr to a numpy array, but we should NOT emulate pointer access or pointer arithmetic directly.

   Probably we do not want to even support pointers directly like in here:

   lpython/src/runtime/lpython/lpython.py

   Line 509 in 8df052e

   def pointer(x, type_=None):

   , because it is returning a ctype "emulation", and that's fragile. Internally in ASR we can keep pointers, I think that's fine.

   All we need to do is to provide some way to interact with C, but our current level of API seems "heavy". For example, take this:

   lpython/integration_tests/structs_02.py

   Lines 9 to 19 in 8df052e

   	def f(a: CPtr) -> None:
   	x: i32
   	y: f32
   	a1: A = A(3, f32(3.25))
   	a2: Pointer[A]
   	a2 = pointer(a1)
   	print(a2, pointer(a1))
   	x = a2.x
   	y = a2.y
   	assert x == 3
   	assert f64(y) == 3.25

   , currently we emulate this via ctypes. Rather, the implementation of pointer should be just:

   def pointer(x):
       return x
   

   Which in CPython b = pointer(a) increments the refcount of a and both a and b point to the same structure, and so it behaves identically to a pointer. LPython will ensure that the semantic subset that we do is 100% identical. Consequently, you can only take "pointers" of structs, lists, arrays, but not ints, floats.

   However perhaps we don't even need a notion of pointers at all, we just need some light API to interact with C, and when C returns a pointer to int, we just need some way to call it, perhaps via a numpy array of ints, of size 1, or something like that, and LPython ensures that there is no overhead.