feat(kernel): SEA-via-kernel backend (opt-in, behind build tag)#393
Draft
mani-mathur-arch wants to merge 5 commits into
Draft
feat(kernel): SEA-via-kernel backend (opt-in, behind build tag)#393mani-mathur-arch wants to merge 5 commits into
mani-mathur-arch wants to merge 5 commits into
Conversation
Add a second execution backend that runs statements over the Statement Execution API via the Rust databricks-sql-kernel, reached through a cgo C ABI. It is opt-in behind WithUseKernel + the databricks_kernel build tag, so the default build stays pure-Go (CGO_ENABLED=0) and is unaffected. Pure-Go side (always compiled): - WithUseKernel / WithWarehouseID connector options; UseKernel / WarehouseID on config with DSN parsing (useKernel, warehouseId). - connector.Connect selects the backend via newKernelBackend, which in a build without the tag is a stub returning a clear "not compiled in" error rather than silently falling back to Thrift. Kernel side (//go:build cgo && databricks_kernel): - KernelBackend/kernelOp implement backend.Backend/Operation over the C ABI: PAT session open (warehouse id or http path), blocking execute with out-of-band context cancellation (a watcher goroutine drives the kernel's detached statement canceller), and result streaming. - kernelRows imports Arrow batches zero-copy via the Arrow C Data Interface and scans the scalar types (ints, floats, bool, string, binary, date, timestamp, and top-level decimal as an exact string per #274); unsupported types return an explicit error. KernelError maps to the driver's error surface with sqlstate, and to driver.ErrBadConn for session-unusable statuses. Tests: tagged unit tests for error/status mapping and scalar scanning; live e2e (exercised against a staging warehouse) for select, per-type scanning, CloudFetch, and context cancellation, plus a Thrift-parity check that both backends render scalars identically. The cgo link directives point at a locally built kernel; committing a prebuilt per-platform static lib and adding a tagged CI job are a follow-up, so the kernel path is not yet covered by CI. Co-authored-by: Isaac Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
Extend the kernel backend's scanner to the complex data types, so the SEA path covers the full scalar-plus-nested type set. Nested Arrow values — list, map, struct, and VARIANT (which arrives as a nested value) — render to a JSON string that is byte-identical to the Thrift arrow path, so a query's result is the same across backends. GEOMETRY arrives as a WKT string and is read by the existing string arm. The renderer (scan_nested.go) recurses into child arrays and mirrors the Thrift marshal() rules: time.Time as a quoted .String(), and nested decimals as float64 (the exact-string decimal applies only to a top-level decimal column, #274). scanCell delegates nested columns here; genuinely unhandled types (interval/ duration) still return an explicit error. Tests: unit tests for list/map/struct/nested-null rendering; the live e2e data types table gains array/map/struct/variant/geometry cases; the Thrift-parity query gains the same, asserting byte-identical output across backends. Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
Route the driver's existing connection options through to the kernel session, so a kernel-backed connection honors the same knobs as Thrift with no change to the user-facing API — only WithUseKernel selects the backend. The connector reads the same config the Thrift backend reads and translates it to the kernel's flat connection config: - Session confs (STATEMENT_TIMEOUT, QUERY_TAGS, TIMEZONE, …): the same SessionParams map Thrift forwards, applied one key at a time via set_session_conf. This covers Query Tags and server statement timeout. - TLS: TLSConfig.InsecureSkipVerify (WithSkipTLSHostVerify) maps to the kernel's skip-hostname-verification setter — the one TLS knob the driver actually honors today. - HTTP proxy: resolved via http.ProxyFromEnvironment for the connection's endpoint, the same cached HTTP(S)_PROXY / NO_PROXY decision the Thrift transport makes, then passed to set_proxy. No new dependency or option. - SPOG org routing continues to ride in the http path's ?o= (parsed kernel-side). M2M/OAuth is deliberately not included here: its credentials are held inside the authenticator rather than on config, so wiring it needs a small config change that is better done on its own. Tests: a proxy-resolution unit test; live e2e that reads query tags and statement timeout back from the server via SET (proving they were applied, not just accepted) and that a TLS skip-verify connection still succeeds. Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
…leak Address three review findings on the kernel backend. Session timezone: DATE/TIMESTAMP values were rendered in UTC, ignoring the configured location, so a connection with a timezone returned times in a different zone than the Thrift backend. Forward cfg.Location into the kernel config and apply it (.In(loc)) when scanning dates/timestamps, including inside nested/JSON values — matching the Thrift path. nil location keeps UTC. Silently-dropped options: newKernelBackend ignored Catalog/Schema and EnableMetricViewMetadata. Neither is wired for the kernel yet — Catalog/Schema have no kernel C-ABI setter, and metric-view maps to a server session conf we want to route backend-neutrally rather than duplicate the Thrift literal here — so the backend now returns a clear error at connect time when either is set, instead of running with different behavior than Thrift. Both are follow-ups. Handle leak: kernelOp.Results returned without closing the operation when get_result_stream failed — and on the query path nothing else closes it, since the (absent) Rows was to own teardown. Close the operation on that error path. Tests: unit tests for timezone rendering (location applied vs nil=UTC) and the unsupported-option rejections; a live timezone e2e asserting the scanned timestamp carries the configured location. Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
A DECIMAL inside a nested value (struct field, list element, map value/key) was
rendered via ToFloat64, so it emitted a lossy float64 — e.g. a struct decimal
19.99 came out as {"d":19.990000000000002}, diverging from the Thrift path's
{"d":19.99}. The top-level decimal column was already exact (#274); only the
nested path was lossy.
writeJSON now renders a nested Decimal128 with the same exact-string helper the
top-level scan uses, emitted as a raw JSON number literal — mirroring the Thrift
arrow path's marshalScalar → ValueString (databricks-sql-go#253/#274). Map keys
go through scalarForJSON, which now defers to the scalar scan (exact string) as
well, so a decimal key is exact too.
The earlier nested tests missed this because they used float64-exact values
(1.5, 2.5). Add a struct-decimal unit case (19.99) and a nested-decimal column
to the live Thrift-parity query as regression guards.
Signed-off-by: Mani Kaustubh Mathur <mani.mathur@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a second execution backend that runs statements over the Statement Execution
API via the Rust
databricks-sql-kernel, reached through a cgo C ABI. It isopt-in behind
WithUseKernel(true)+ thedatabricks_kernelbuild tag, so thedefault build stays pure-Go (
CGO_ENABLED=0) and is unchanged — the only newuser-facing surface is
WithUseKernel.The user API is otherwise identical to Thrift: the kernel backend reads the same
config.Configthe Thrift backend reads and routes those options to the kernelinternally (the kernel setters are never exposed to the user), mirroring how the
kernel's pyo3 / napi bindings work.
What's implemented
Verified with unit tests + live e2e against a staging warehouse, and a
Thrift-parity check asserting both backends render results identically:
cancellation (watcher goroutine → the kernel's detached statement canceller),
server statement timeout, query tags
and the complex types — array, map, struct, variant — all rendered to JSON
byte-identical to Thrift; geometry (WKT). DECIMAL is rendered exactly, matching
Thrift: a fixed-point string for a top-level column and an exact JSON number
inside a nested value (never a lossy float64).
same decision Thrift makes), SPOG org routing (via the http path
?o=)driver.ErrBadConnfor session-unusable statusesasync HTTP, parallel Cloud Fetch
Deferred (rejected loudly at connect time where applicable)
Options that aren't wired yet return a clear error rather than behaving
differently than Thrift:
(needs a kernel change)
backend-neutrally rather than duplicate a Thrift-specific literal
Not yet offered: OAuth M2M / U2M, metadata commands.
Structure
internal/backend/kernel/(//go:build cgo && databricks_kernel): the cgobinding —
cgo.go(FFI-safe call helper + error mapping),backend.go(session open + config),
operation.go(blocking execute + watcher cancel),rows.go(zero-copy Arrow C Data Interface import + scalar scan),scan_nested.go(nested→JSON renderer),
scan.go(exact-string decimal).WithUseKernel/WithWarehouseIDoptions,UseKernel/WarehouseIDconfig + DSN parsing, a build-tagged stub that returns a clearerror when the kernel backend isn't compiled in, and the factory branch in
connector.Connect.Test plan
CGO_ENABLED=0 go build ./...+ full pure-Go suite pass (default build unchanged)go build -tags databricks_kernel(CGO_ENABLED=1) compiles and linksrendering, proxy resolution, unsupported-option rejection
CloudFetch (1M rows), ctx cancellation, query tags / statement timeout
(read back via
SET), timezone, TLS skip-verifyboth backends renders identical rows
isaac review(findings addressed)This pull request and its description were written by Isaac.