Skip to content

[ExecuTorch][WebGPU] Convert remaining native tests to GTest#20706

Open
JulianCloudNTH wants to merge 1 commit into
gh/JulianCloudNTH/85/basefrom
gh/JulianCloudNTH/85/head
Open

[ExecuTorch][WebGPU] Convert remaining native tests to GTest#20706
JulianCloudNTH wants to merge 1 commit into
gh/JulianCloudNTH/85/basefrom
gh/JulianCloudNTH/85/head

Conversation

@JulianCloudNTH

@JulianCloudNTH JulianCloudNTH commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Stack from ghstack (oldest at bottom):

Convert the remaining hand-rolled int main() + printf/bool ok native tests to GTest so the whole backends/webgpu/test/ suite is uniform, filterable via --gtest_filter, and self-reporting (extends the GTest conversion already applied to test_dynamic_shape). Harness-only change: every test case, tensor shape, tolerance, artifact filename, and skip condition is preserved 1:1 — only the pass/fail reporting mechanism changes.

Key changes:

  • test/test_webgpu_native.cpp, test/native/test_dispatch_order.cpp, test/native/test_index.cpp, test/native/test_scratch_buffer.cpp, test/native/test_update_cache.cppmain+printf/bool ok accumulator → TEST() cases using EXPECT_*/ASSERT_*; each keeps a custom main() that brings up the WebGPU device once then RUN_ALL_TESTS() (device-absent still SKIPs by returning 0).
  • CMakeLists.txt — move every native-test target into the if(TARGET GTest::gtest) block, linking GTest::gtest.
  • scripts/test_webgpu_native_ci.sh — add -DEXECUTORCH_BUILD_TESTS=ON to the native-test configure so the now-gtest-gated targets are defined; the existing probe/build/run logic is unchanged.

Differential Revision: D110536636

[ghstack-poisoned]
@pytorch-bot

pytorch-bot Bot commented Jul 2, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20706

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 102e7c6 with merge base 73c259e (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 2, 2026
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@JulianCloudNTH

Copy link
Copy Markdown
Contributor Author

@claude Review thoroughly. Check for stale code / bugs.

@claude

claude Bot commented Jul 3, 2026

Copy link
Copy Markdown

Claude finished @JulianCloudNTH's task in 8m 7s —— View job


Review: WebGPU native tests → GTest

Reviewed all 7 files against origin/gh/JulianCloudNTH/85/base. The conversion is careful and faithful — cases, shapes, tolerances, artifact names, skip conditions, and the device-absent→return 0 SKIP behavior are preserved, and every deleted sweep wrapper maps cleanly onto a TEST(). ASSERT_* inside the void helpers is the correct pattern here (it returns from the helper, matching the old return false). One real bug and a couple of minor notes below.


🔴 Sibling build script will break: test_build_webgpu.sh still lacks -DEXECUTORCH_BUILD_TESTS=ON

This PR moves webgpu_native_test, webgpu_dispatch_order_test, webgpu_scratch_buffer_test, webgpu_update_cache_test, webgpu_index_test inside the if(TARGET GTest::gtest) block (CMakeLists.txt:190-209). GTest::gtest only exists when googletest is added as a subdirectory, which happens only under BUILD_TESTING/EXECUTORCH_BUILD_TESTS:

third-party/CMakeLists.txt:18   if(BUILD_TESTING) add_subdirectory(googletest) endif()
CMakeLists.txt:253              if(EXECUTORCH_BUILD_TESTS) include(CTest) else() set(BUILD_TESTING OFF)

You correctly added -DEXECUTORCH_BUILD_TESTS=ON to scripts/test_webgpu_native_ci.sh. But the sibling backends/webgpu/test/test_build_webgpu.sh — the documented local end-to-end script — configures without it (test_build_webgpu.sh:84-95) and then builds these targets unconditionally:

cmake --build "${NATIVE_BUILD_DIR}" --target webgpu_native_test -j${NPROC}          # :97
cmake --build "${NATIVE_BUILD_DIR}" --target webgpu_dispatch_order_test -j${NPROC}  # :98
cmake --build "${NATIVE_BUILD_DIR}" --target webgpu_scratch_buffer_test -j${NPROC}  # :99

Unlike the CI script, this one does not probe --target help first, so if no system GTest is picked up by the find_package(GTest QUIET) fallback, these targets won't be defined and the build fails with "unknown target". Before this PR the targets were defined regardless of EXECUTORCH_BUILD_TESTS, so the script worked. Add -DEXECUTORCH_BUILD_TESTS=ON to the configure at test_build_webgpu.sh:84. Fix this →


🟡 Minor: pass/fail boundary changed from inclusive to exclusive in a few tests

Several gates moved from if (err > tol) FAIL (pass iff <= tol) to EXPECT_LT(err, tol) (pass iff < tol), e.g. test_index.cpp (EXPECT_LT(max_abs_err, 1e-3f) / EXPECT_LT(max_rel_err, 1e-3f)), test_update_cache.cpp / test_webgpu_native.cpp update_cache (EXPECT_LT(max_abs_err, 1e-3f)). The AND/OR logic is preserved; only the exact-boundary case (err == 1e-3f) flips from pass to fail. This is essentially never hit in practice and is stricter, not looser — but it's a deviation from the "tolerance preserved 1:1" claim. dispatch_order avoided this by keeping the exact expression via EXPECT_FALSE(a && b); the others could do the same with EXPECT_LE if strict 1:1 is desired. Non-blocking.


🟢 Things I checked that are correct

  • test_query_pool_roundtrip(*get_default_webgpu_context()) — safe: RUN_ALL_TESTS() only runs after set_default_webgpu_context(&ctx), and the device-absent path returns before it.
  • SDPA expect_reject path (ASSERT_NE(err, Error::Ok) + return) preserves the original PASS/FAIL semantics.
  • SdpaAllFamiliesRanWhenDirSet recomputes .pte presence independently but is equivalent to the old sdpa_ran && replay_ran && dyn_ran && incache_ran guard.
  • required/heavy gating in the q4gsw & SDPA sweeps correctly maps FAILADD_FAILURE() (non-fatal, so the loop continues, matching the old ok = ... && ok) and SKIPprintf/GTEST_SKIP.
  • quant_within_tol moved to a full definition (forward-decl removed) with identical body; no other refs dangling.
  • CI script --target help probe still guards the CI path, so the webgpu_index_test move into the gated block is covered there.

Nice, mechanical, well-commented conversion overall — the one thing to fix before landing is the test_build_webgpu.sh configure flag.
gh/JulianCloudNTH/85/head

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant