fix: prevent sender control future race#357
Conversation
There was a problem hiding this comment.
Pull request overview
This PR addresses a control-message future race in the computer-core message-sending pipeline (specifically QueuedMessageSender) that could surface as The origin future must be null, and also tightens computer-test integration behavior to fail faster instead of stalling for long BSP timeouts.
Changes:
- Refactors control-message handling in
QueuedMessageSenderso each START/FINISH message carries its ownCompletableFuture, and the in-flight control future is cleared before completion. - Adds unit regressions for consecutive control messages and for transport failures completing the control future exceptionally.
- Updates sender integration tests to apply shorter BSP wait timeouts and to use a bounded
waitForServices()helper.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| computer/computer-test/src/main/java/org/apache/hugegraph/computer/suite/integrate/SenderIntegrateTest.java | Adds CI-oriented BSP/service timeouts and a fail-fast wait helper for master/worker futures |
| computer/computer-test/src/main/java/org/apache/hugegraph/computer/core/sender/QueuedMessageSenderTest.java | Adds regressions for control-future sequencing and transport exception completion |
| computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/sender/QueuedMessageSender.java | Refactors control-message future lifecycle and transport-exception handling |
| computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/sender/QueuedMessage.java | Extends queued message model to optionally carry a control future |
Comments suppressed due to low confidence (2)
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/sender/QueuedMessageSender.java:273
- Same issue as
sendStartMessage(): ifsetControlFuture()throwsComputerException, it will currently bubble out ofsendFinishMessage()and can terminate the send-executor thread. Since the future is already completed exceptionally insetControlFuture(), catchComputerExceptionhere and return to avoid killing the sender thread.
public void sendFinishMessage(CompletableFuture<Void> future)
throws TransportException {
this.setControlFuture(future);
try {
this.client.finishSessionAsync().whenComplete((r, e) -> {
computer/computer-test/src/main/java/org/apache/hugegraph/computer/suite/integrate/SenderIntegrateTest.java:122
- The master options set
withRpcServerPort()twice (8611then0). The first value is immediately overridden and can be misleading when debugging port binding issues; it’s clearer to keep only the effective port setting.
.withRpcServerHost("127.0.0.1")
.withRpcServerPort(8611)
.withRpcServerPort(0)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| public void sendStartMessage(CompletableFuture<Void> future) | ||
| throws TransportException { | ||
| this.setControlFuture(future); | ||
| try { | ||
| this.client.startSessionAsync().whenComplete((r, e) -> { |
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: An existing review thread covers the sender-executor termination risk, and the new fail-fast service wait can race past lifecycle cleanup. Evidence: exact-head static inspection by six independent lanes; git diff --check passed; visible exact-head checks are green.
| for (CompletableFuture<Void> future : futures) { | ||
| future.whenComplete((r, e) -> { | ||
| if (e != null) { | ||
| result.completeExceptionally(e); |
There was a problem hiding this comment.
result on the first failure lets the caller enter cleanup before the other service threads have initialized and published their service references. A thread can add its service after closeWorker()/closeMaster() has already observed null or finished iterating workerServices; the multi-worker threads also do not own their services with try-with-resources. That can leave Netty/etcd resources running into later tests. Please cancel and join every spawned thread and ensure each thread closes any service it initializes in a finally, including services published after the first failure.
Purpose of the PR
Main Changes
Fixes a race condition in
QueuedMessageSenderwhere consecutive control messages could fail withThe origin future must be null.The sender now clears the in-flight control future before completing it, allowing the next control message to be sent safely. Transport failures also complete the affected future exceptionally.
Tests
QueuedMessageSenderTestandIntegrateTestSuite.Verifying these changes
Does this PR potentially affect the following parts?
Documentation Status
Doc - TODODoc - DoneDoc - No Need