plugins: defer spenderp awaiting-channel recovery

[bittylicious]

Summary

spenderp scans CHANNELD_AWAITING_LOCKIN / DUALOPEND_AWAITING_LOCKIN channels for saved funding PSBTs and queues signpsbt recovery attempts after an unclean shutdown.

On a large mainnet node, one such signpsbt took about 130 seconds before failing with:

Failed signpsbt for waiting channel ...: {"code":-32602,"message":"Could not add keypaths to PSBT?"}

Previously, that recovery work ran in the plugin init path. Unrelated important builtin plugins were still waiting to finish startup and hit their init timeout, causing lightningd to shut down/restart. The plugin named in the fatal log varied depending on which builtins were enabled (autoclean, chanbackup, commando, funder, cln-bwatch, topology).

This PR registers the recovery scan on a zero-delay plugin timer. The timer is already due, but it is dispatched only after the init response returns and control enters the event loop. Recovery behaviour is preserved while slow or failing recovery signing no longer blocks spenderp init or the global plugin startup handshake.

Reproduction context

Observed on CLN v26.06 and v26.06.2 with a SQLite wallet DB initially around 9.1 GiB, later vacuumed to around 4.7 GiB. Vacuum/integrity checks and upgrading to v26.06.2 did not change the symptom.

The triggering channel was in CHANNELD_AWAITING_LOCKIN, with:

• funding.withheld=false
• a top-level funding_txid
• a saved funding.psbt

Disabling spenderp stopped the startup crash loop. Re-enabling only the funding-related builtin group reproduced the slow signpsbt and later spenderp failure log.

Validation

The earlier live validation used a delayed recovery timer: a patched Docker build was deployed on the affected mainnet node, startup reached Server started, and the later recovery attempt still produced the original wallet/PSBT error without causing an exit or restart. RPC remained responsive and getinfo returned normally. That validates the recovery-after-init intent, not the zero-delay scheduling correction in this revision.

For this revision, global_timer(..., time_from_sec(0), ...) uses the existing plugin timer semantics: the callback runs after init has responded and the event loop begins. The existing CI regression suite is the remaining automated coverage for that correction and is awaiting CI approval/run.

CI note

A prior Valgrind run encountered the known 180-second timeout collision; it is not treated as validation of this change. Fresh CI remains required.

Notes

This does not decide whether a saved PSBT should be retried at all. It only prevents slow recovery signing from being startup-fatal. A follow-up may still be useful to narrow retry criteria for saved funding PSBTs that already correspond to a broadcast funding transaction.

Changelog

Changelog-None

[cdecker]

Excellent work @bittylicious, do you have any idea why the signpsbt call might be this slow? I like the workaround, but I'm curious as to the root cause.

[bittylicious]

I don't know the definitive root cause yet, but my strongest suspicion is that the time is being spent in wallet/PSBT keypath matching before signpsbt eventually fails with:

Could not add keypaths to PSBT?

The affected channel already had a funding_txid, but because it was still in CHANNELD_AWAITING_LOCKIN, spenderp retried the saved funding.psbt during startup. My guess is that this saved PSBT no longer maps cleanly to the wallet's derivation data/current wallet state, so CLN does a lot of wallet matching work before concluding it cannot add the keypaths.

The large wallet DB may be a contributor, but probably not the whole explanation: on this node the SQLite wallet DB was about 9.1 GiB, VACUUM reduced it to about 4.7 GiB with clean integrity checks, and the slow signpsbt behaviour remained.

So I think the workaround is still useful because it prevents this recovery path from being startup-fatal, but it does not explain or fix the underlying slow signpsbt. The next things I would look at are:

• compare the saved funding.psbt for the affected channel with the broadcast funding transaction;
• instrument/trace signpsbt, especially the wallet PSBT keypath-matching path;
• check whether this retry should be skipped or guarded once a channel already has a known funding_txid and is merely awaiting lock-in.

Unfortunately I don't really have time to dig deeper into the root cause, but having this guard seems sensible.

[bittylicious]

I have updated this branch to current master and fixed the Valgrind regression from the earlier 180-second delay. Recovery is now scheduled with an already-due (time_from_sec(0)) plugin timer: it runs on event-loop entry after the init response, rather than inline during the init handshake.

The new CI and macOS workflow runs are currently waiting for fork-workflow approval. @cdecker, when convenient, could you approve those runs and review the updated change?