Revdep check GHA

[VisruthSK]

Add a revdepcheck GHA to do reverse dependency checks remotely. Using 4 workers resulted in resource exhaustion--3 workers lead to a total runtime of about 11 hours. The main advantage to this approach, even though it is slow, is to not block someone's compute.

The GHA works by gracefully stopping the task before 6 hours (the current GHA time limit), uploading progress, and recursively calling itself until it finishes. We rely on the revdepcheck package's own resume functionality to continue work from the previous runs.

[jgabry]

This is cool, thanks @VisruthSK.

The main advantage to this approach, even though it is slow, is to not block someone's compute.

That said, we'll be blocking some GHA compute for 11 hours that other parts of the stan-dev organization might need.

[jgabry]

When I download the results, in the cran.md file I see:

### Failed to check

* bayesdfa         (NA)
* dcvar            (NA)
* pcFactorStan     (NA)
* rmsb             (NA)
* rstanarm         (NA)
* TriDimRegression (NA)

Guessing these ones timed out?

But then there's a separate file failures.md that lists a bunch of installation errors. Since those errors occurred with both the CRAN and dev versions of loo they're not marked as problems and not listed in failed to check. But it's a decent number of packages.

Then at the bottom of README.md it lists 44 packages that it failed to check, which maybe is the combination of the lists in the cran.md and the failures.md file? When I run it locally I also get this list but I only get 18 failures instead of 44.

[VisruthSK]

That said, we'll be blocking some GHA compute for 11 hours that other parts of the stan-dev organization might need.

Yeah... since its dispatched manually we could run on "off hours"/when the largest chunk of Stan people are usually not working, or let people know?

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.70%. Comparing base (1a9afd3) to head (4d7d462).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #382   +/-   ##
=======================================
  Coverage   92.70%   92.70%           
=======================================
  Files          31       31           
  Lines        3031     3031           
=======================================
  Hits         2810     2810           
  Misses        221      221           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[VisruthSK]

Then at the bottom of README.md it lists 44 packages that it failed to check, which maybe is the combination of the lists in the cran.md and the failures.md file? When I run it locally I also get this list but I only get 18 failures instead of 44.

Ran into some system dependency issues which I've tried to fix on this next run, which was skipping checking some packages.

Guessing these ones timed out?

So some were install errors and others were timing out. dcvar, pcFactorStan, and rstanarm all timed out.

[jgabry]

Yeah... since its dispatched manually we could run on "off hours"/when the largest chunk of Stan people are usually not working, or let people know?

Yeah, good idea. We can try to run it during "off hours".

[jgabry]

Do we need to merge this in order to test it after you added more dependencies?

The GHA works by gracefully stopping the task before 6 hours (the current GHA time limit), uploading progress, and recursively calling itself until it finishes. We rely on the revdepcheck package's own resume functionality to continue work from the previous runs.

Is this what it's doing in your other repo too? That is, the one that you ran successfully (except for the dependency issues) used this approach too and it worked?

[VisruthSK]

Do we need to merge this in order to test it after you added more dependencies?

No, that's why I'm running it on a fork actually. GH only allows workflow_run GHAs to be run when you have the workflow on master.

I've started another run here: https://github.com/VisruthSK/loo/actions/runs/29283758907/job/86931106720 which should spawn a sub-job or two too.

Is this what it's doing in your other repo too? That is, the one that you ran successfully (except for the dependency issues) used this approach too and it worked?

Yeah this is what the two runs whose results you looked at are doing.

[jgabry]

Ok cool, let's see how the run you just started goes before merging this

[jgabry]

Looks like it failed with:

The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

I'm not sure if this is a general issue with the approach we're taking here or just a one off issue.

If we want to try a different approach, I just saw that @andrjohns made an RStan PR for checking revdeps: stan-dev/rstan#1190