HIVE-29681: Introduce contextual table search#6595
Draft
dengzhhu653 wants to merge 20 commits into
Draft
Conversation
dengzhhu653
marked this pull request as draft
July 10, 2026 01:05
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a new metastore-search module to the standalone metastore, adding an embedded Lucene-based indexing/search subsystem with hybrid (keyword + semantic) retrieval, embedding-model integration, and backup/restore mechanics, and wires it into distribution packaging.
Changes:
- Adds a new
standalone-metastore/metastore-searchMaven module and includes it in standalone-metastore build + assembly packaging. - Implements Lucene-backed indexing/search components (mapping, query contract, hybrid fusion, embedding inference + caching, index state/backup utilities).
- Adds extensive unit/integration tests and supporting test utilities for indexing/search behavior and recovery scenarios.
Reviewed changes
Copilot reviewed 89 out of 89 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| standalone-metastore/pom.xml | Adds metastore-search module to standalone metastore reactor build. |
| standalone-metastore/packaging/src/assembly/src.xml | Includes metastore-search artifacts in standalone metastore assembly. |
| standalone-metastore/packaging/pom.xml | Adds packaging dependency on hive-standalone-metastore-search. |
| standalone-metastore/metastore-search/src/test/resources/log4j2.properties | Test logging config for the new module. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/testutil/TestLeaderElection.java | Test leader-election stub used by indexer tests. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/testutil/StubEmbedModel.java | Deterministic embedding model for tests (no ONNX runtime). |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/testutil/RealMetastoreServer.java | In-process HMS server fixture for integration tests. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/testutil/MetastoreBootstrapMocks.java | Mockito helpers for bootstrap/indexer initialization tests. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/testutil/InMemoryIndexStateClient.java | In-memory IndexStateClient used across backup/restore tests. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/testutil/IndexMutationApplier.java | Applies metastore event tasks to the Indexer in tests. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/search/TestSearchQuery.java | Tests SearchQuery defaults and validation. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/search/TestHybridSearch.java | Tests hybrid query parsing and fusion-request generation. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/search/InMemorySearchFixture.java | In-memory Lucene+stub-embedding search fixture for tests. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/metastore/TestRealMetastoreSearchIntegration.java | Integration tests for bootstrap + incremental updates against real HMS. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/metastore/TestRealMetastoreSearchHybrid.java | Integration tests for keyword/hybrid behavior and ranking. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/metastore/TestRealMetastoreSearchFailureRecovery.java | Integration tests for restore and poison-event recovery behavior. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/metastore/TestMetastoreIndexer.java | Tests leader/follower bootstrap and backup restore flows. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/metastore/TestMetastoreEventHandlerIntegration.java | Tests event polling and mutation coalescing integration. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/metastore/TestBootstrapIndexer.java | Tests bulk bootstrap indexing behavior and failures. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/mapping/TestIndexMapping.java | Tests default schema/hybrid-field exposure and analyzers. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/inference/TestEmbedModelRegistry.java | Tests model registry behavior and deterministic stub vectors. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/inference/TestEmbeddingPrompt.java | Tests embedding prompt prefix logic. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/inference/TestEmbeddingCache.java | Tests embedding cache scoping, copying, and hit/miss tracking. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/TestIndexManagerFailureRecovery.java | Tests unhealthy index state gating. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/TestIndexerSearchIntegration.java | Tests end-to-end indexing/search over in-memory fixture. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/TestIndexerFlushCommit.java | Tests commit/flush thresholds and forced commits. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/TestIndexerEmbeddingCache.java | Tests embedding cache dedupe behavior in indexing pipeline. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/store/TestIndexBackupUtils.java | Tests backup sync/restore correctness and staging handling. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/store/TestIndexBackupFailureRecovery.java | Tests interrupted/partial restore and stale-file cleanup behavior. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/manifest/TestIndexManifest.java | Tests manifest JSON, diffing, and equality semantics. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/config/TestSearchConfig.java | Tests SearchConfig defaults and overrides. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/config/TestIndexStateConfig.java | Tests IndexStateConfig remote/memory flags and URI validation. |
| standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/config/TestIndexConfig.java | Tests IndexConfig defaults and overrides. |
| standalone-metastore/metastore-search/src/main/java/org/apache/lucene/search/package-info.java | Documents copied Lucene-like classes placed under Lucene package namespace. |
| standalone-metastore/metastore-search/src/main/java/org/apache/lucene/search/BayesianScoreEstimator.java | Adds Bayesian score parameter estimation utility used by fusion scoring. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/search/TableSearchResult.java | Public-ish result wrapper for table search hits and totals. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/search/TableSearchHit.java | Search hit type with table identity, score, and returned fields. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/search/SearchReqResp.java | Internal request/response records and search mode enum. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/search/SearchQuery.java | Narrow public query contract and validation helpers. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/search/SearchBackend.java | Pluggable backend interface for search implementations. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/search/LuceneSearchBackend.java | Embedded Lucene backend implementation + async init readiness. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/search/HybridSearch.java | Hybrid query parsing and fusion request builder. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/metastore/MetastoreTableMapper.java | Maps HMS Table metadata into a Lucene document schema. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/metastore/MetastoreSchemas.java | Defines the default index mapping for Hive tables. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/metastore/MetastoreEventListener.java | Listener contract for coalesced indexing tasks and health signals. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/metastore/MetastoreCluster.java | Leader-election wiring for distributed index coordination. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/mapping/TableDocument.java | Builds Lucene Documents from mapped fields (lexical/filter/semantic). |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/mapping/SearchParams.java | Field search parameters (lexical/semantic/hybrid + distance). |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/mapping/IndexMapping.java | Index schema container with analyzers and hybrid-field helpers. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/mapping/FieldSchema.java | Schema types describing field storage/filter/search behavior. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/mapping/field/TextField.java | Field value + optional embedding. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/mapping/field/IdField.java | Identifier field wrapper. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/mapping/field/Field.java | Field interface and Javadoc. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/inference/LocalOnnxEmbeddingModel.java | ONNX embedding model adapter with single-thread inference executor. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/inference/EmbedModelRegistry.java | Model registry creation/warmup + optional embedding cache plumbing. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/inference/EmbedModel.java | Embedding model interface and task types. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/inference/EmbeddingPrompt.java | Task-based prefixing for model input strings. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/inference/EmbeddingCache.java | In-memory embedding cache keyed by model/task/text. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/store/RemoteStateClient.java | Hadoop FS-backed remote index state storage for backup/restore. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/store/LocalStateClient.java | Lucene Directory-backed local index state reader/writer + validation. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/store/IndexStateClient.java | Storage abstraction for manifests and index file operations. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/store/IndexBackupUtils.java | Backup sync/restore logic and staging manifest flow. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/manifest/IndexManifest.java | JSON manifest model + file diffing ops. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/IndexSession.java | Co-locates indexer + searcher manager + periodic refresh. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/IndexManager.java | Opens local/remote index state and manages health + backup flows. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/index/FlushTrackingWriter.java | Tracks Lucene flushes to gate commit frequency. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/exception/SearchException.java | Search-layer checked exception type. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/exception/InitializeException.java | Initialization exception type + wrapper helper. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/exception/IndexNotReadyException.java | Not-ready exception used by readiness checks. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/exception/IndexNotHealthyException.java | Health-gating exception surfaced when index is unhealthy. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/exception/IndexIOException.java | IO exception type for index state operations. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/exception/IndexException.java | Indexing exception type + wrapper helper. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/config/SearchConfig.java | Search tuning knobs (refresh interval, hybrid weights, init timeout, etc.). |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/config/InferenceConfig.java | Embedding model configuration and local/remote model file staging. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/config/IndexStateConfig.java | Local/remote index location configuration. |
| standalone-metastore/metastore-search/src/main/java/org/apache/hive/search/config/IndexConfig.java | Indexing configuration knobs (flush/commit, bootstrap, poll, etc.). |
| packaging/pom.xml | Adds top-level packaging dependency on standalone-metastore-search artifact. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
asf-ci-hive
added
tests pending
tests failed
and removed
tests failed
tests pending
labels
asf-ci-hive
added
tests pending
tests passed
and removed
tests passed
tests pending
labels
asf-ci-hive
added
tests pending
tests failed
and removed
tests passed
tests pending
labels
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 90 out of 90 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (2)
standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/testutil/InMemoryIndexStateClient.java:103
- crc32C() currently computes CRC32, not CRC32C. Either rename the field/method to crc32, or compute CRC32C to match the manifest field name and avoid cross-implementation inconsistencies.
standalone-metastore/metastore-search/src/test/java/org/apache/hive/search/index/store/TestIndexBackupUtils.java:139 - crc32C() computes CRC32 (different polynomial) even though the method name implies CRC32C. This can invalidate the intent of the checksum-based sync tests.
| import java.util.List; | ||
| import java.util.Map; | ||
| import java.util.Optional; | ||
| import java.util.zip.CRC32; |
|
|
||
| import java.io.ByteArrayInputStream; | ||
| import java.util.List; | ||
| import java.util.zip.CRC32; |
Comment on lines
+57
to
+63
| public static void validateRemoteUri(String uriText) throws IndexIOException { | ||
| try { | ||
| URI.create(uriText); | ||
| } catch (IllegalArgumentException e) { | ||
| throw new IndexIOException("invalid store.remote.uri: " + uriText, e); | ||
| } | ||
| } |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?