It isn't possible to temporarily resurrect objects safely in the Py3.15t limited API

[da-woods]

Bug report

Bug description:

In Cython we do something like

Py_SET_REFCNT(o, Py_REFCNT(o) + 1);
// Run user implemented deallocation code
Py_SET_REFCNT(o, Py_REFCNT(o) - 1);

The reason for this is that we don't want any refcounting (e.g. of temps) within the user code to accidentally trigger a recursive call to dealloc (the user code really shouldn't resurrect the object and it's their own problem if they do, but it is reasonable for it to generate temporaries, which may well be out of their control anyway).

This turns out not to be thread-safe in the freethreaded build because it makes TryIncRef succeed. Outside the limited API it's possible to fix by manual access to the reference count, but obviously not inside the limited API.

It would be nice to have a way of to avoid this.

• Most convenient would be to properly access _PyObject_ResurrectStart and _PyObject_ResurrectEnd.
• The other thing that could work is some way to defer the deallocation until the next GC cycle (when I think any TryIncRefs would have fall out of scope anyway). I suspect I could manually arrange to do that, although it feels fragile, possibly relies on the GC being single-threaded, and which I haven't tested (so may not work...)

CPython versions tested on:

3.15

Operating systems tested on:

No response

[scoder]

@ngoldbaum @encukou

[ngoldbaum]

@kumaraditya303 @nascheme in case one of you two have some insight

[kumaraditya303]

Python 3.15 added PyObject_CallFinalizerFromDealloc to limited API which does the same "call finalizer by temporary resurrecting object", can't cython use that for this?

[da-woods]

Cython has split finalization into __del__ and __dealloc__ user-defined functions.

__del__ is indeed called from PyObject_CallFinalizerFromDealloc and follows Python __del__ semantics (i.e. may resurrect, finalizer will be triggered by Python only once, but base class finalizers rely on super(), can be called from Python via inst.__del__() manually at any point)

This is wrapping the user __dealloc__ function. That has the advantage that you can relying on it being called (because Cython implicitly generates the call in tp_dealloc). Because of this we've always recommended that people put their "genuinely must happen C-level resource handling cleanup in it".

There's potentially a bit of a mess around resurrection in the user-defined __dealloc__ function. We've assumed that it really shouldn't happen and made no attempt to check, but it's up to the user not to do anything stupid. However, even the Cython generated code may bump ref-counts up and down temporarily which is what the guard is for.

I think this split partly reflects how Cython has adapted to changes in the finalization of Python over the years so might not be exactly what you'd do if designing from scratch.

I have tried to think a bit about how we could use PyObject_CallFinalizerFromDealloc for __dealloc__ to but the "can be overridden from subclasses" bit of it makes it quite hard.

[ilhameijayadi265-ux]

Bug report

Bug description:

In Cython we do something like

Py_SET_REFCNT(o, Py_REFCNT(o) + 1);
// Run user implemented deallocation code
Py_SET_REFCNT(o, Py_REFCNT(o) - 1);
The reason for this is that we don't want any refcounting (e.g. of temps) within the user code to accidentally trigger a recursive call to dealloc (the user code really shouldn't resurrect the object and it's their own problem if they do, but it is reasonable for it to generate temporaries, which may well be out of their control anyway).

This turns out not to be thread-safe in the freethreaded build because it makes TryIncRef succeed. Outside the limited API it's possible to fix by manual access to the reference count, but obviously not inside the limited API.

It would be nice to have a way of to avoid this.

• Most convenient would be to properly access _PyObject_ResurrectStart and _PyObject_ResurrectEnd.
• The other thing that could work is some way to defer the deallocation until the next GC cycle (when I think any TryIncRefs would have fall out of scope anyway). I suspect I could manually arrange to do that, although it feels fragile, possibly relies on the GC being single-threaded, and which I haven't tested (so may not work...)

CPython versions tested on:

3.15

Operating systems tested on:

No response