Skip to content

[WIP][core] Fix column masking correctness in query-auth reads#8570

Open
plusplusjiajia wants to merge 1 commit into
apache:masterfrom
plusplusjiajia:core-mask-remap-fail-closed
Open

[WIP][core] Fix column masking correctness in query-auth reads#8570
plusplusjiajia wants to merge 1 commit into
apache:masterfrom
plusplusjiajia:core-mask-remap-fail-closed

Conversation

@plusplusjiajia

@plusplusjiajia plusplusjiajia commented Jul 12, 2026

Copy link
Copy Markdown
Member

Purpose

A cross-column mask (e.g. display := concat_ws('-', first, last)) throws at read time when the query projects the masked target but not the mask's input columns:

Column masking refers to field 'first' which is not present in output row type

Row-filter operands are already added to the read projection (#8447); this PR does the same for column-mask inputs, and hardens the paths that surfaced while testing it:

  • Projection widening: TableQueryAuthResult.requiredAuthFields collects filter operands plus, transitively, the inputs of every mask whose target is readable. The widening appends missing table columns as-is
    (preserving nested pruning), the widened columns are projected back out after masking, and the scan pushes the widened read type before planning so file-level column pruning (e.g. data evolution) keeps the
    files those columns live in.
  • Stale rules fail closed at plan time: rules referencing columns absent from the latest schema (e.g. after a rename/drop) are rejected once in AbstractDataTableScan.authQuery, the common path of every scan,
    re-validated only when the fetched rules change. Projected system fields (e.g. _ROW_ID) stay valid; unprojected ones, nested-pruned columns and blob-view columns are rejected.
  • Read schema is fixed once the first split reader exists (split reads cache their format readers and ignore later read-type changes): every split is projected back to the query's read type, and masks apply
    only to columns readable from the query, so a column merely retained in a previously widened schema cannot activate a mask after a rules change. Previously the first createReader call mutated this.readType, so
    later splits of the same TableRead leaked the auth-added columns — also on the pre-existing row-filter path.
  • Re-configuration fixes on the same path: MergeFileSplitRead.withReadType and DataEvolutionFileStoreScan.withReadType reset stale projections; IncrementalDiffSplitRead projects from the merge read's actual
    output type.

Behavior changes are scoped to query-auth.enabled tables; tables without query auth and the write path are unaffected.

Out of scope: raw-value pushdown on masked columns (filter/TopN/limit statistics pruning, masked partition predicates) is unsafe today on master for the already-merged masking feature. This PR does not change
any pushdown behavior; the fix is #8582 (stacked on this PR), where a predicate on a masked column evaluates on the masked value.

Tests

  • MockRESTCatalogTest: cross-column masking with/without projecting the inputs, nested-pruning interaction, stale rules after rename/drop, system-field rules (projected and not), blob-view rejection, rules
    changing between reads of one TableRead, repeated reads.
  • MergeFileSplitReadTest: repeated withReadType re-configuration; diff read over a projected merge read.
  • TableQueryAuthResultTest: requiredAuthFields / appendMissingFields units.
  • Engines: Flink RESTCatalogITCase and Spark SparkCatalogWithRestTest cross-column ITCases.

@plusplusjiajia plusplusjiajia force-pushed the core-mask-remap-fail-closed branch 4 times, most recently from 6bb3c6c to 117e3e5 Compare July 12, 2026 04:39
@plusplusjiajia plusplusjiajia force-pushed the core-mask-remap-fail-closed branch 14 times, most recently from 62a53ad to 5a49a17 Compare July 12, 2026 12:54
@plusplusjiajia plusplusjiajia force-pushed the core-mask-remap-fail-closed branch 2 times, most recently from f2f677f to 62849c7 Compare July 12, 2026 16:10
A cross-column mask (e.g. display := concat_ws('-', first, last)) threw
at read time when the query projected the masked target but not the
mask's input columns: "Column masking refers to field 'first' which is
not present in output row type".

Row-filter operands are already added to the read projection and
projected back out (apache#8447); this does the same for column-mask inputs.
TableQueryAuthResult.requiredAuthFields collects filter operands plus,
transitively, the inputs of every mask whose target is readable; the
widening appends the missing table columns as-is (preserving nested
pruning), and the scan pushes the widened read type before planning so
file-level column pruning (e.g. data evolution) keeps the files those
columns live in.

Stale rules (columns absent from the latest schema, e.g. after a rename
or drop) fail closed once at plan time in AbstractDataTableScan.authQuery,
the common path of every scan; re-validated only when the fetched rules
change. Projected system fields (e.g. _ROW_ID) stay valid, unprojected
ones are rejected; rules touching a nested-pruned or unprojected
blob-view column are rejected too.

The read schema is fixed once the first split reader exists (split reads
cache their format readers and ignore later read-type changes); every
split is projected back to the query's read type, and masks apply only
to columns readable from the query, so a column merely retained in a
previously widened schema does not activate a mask after a rules change.
Previously the first createReader call mutated this.readType, so later
splits of the same TableRead leaked the auth-added columns -- also on
the pre-existing row-filter path.

Related re-configuration fixes: MergeFileSplitRead.withReadType and
DataEvolutionFileStoreScan.withReadType reset stale projections, and
IncrementalDiffSplitRead projects from the merge read's actual output
type instead of the full table row type.

Raw-value pushdown safety on masked columns (filter/TopN/limit stats
pruning) is deliberately left to a follow-up PR.

Behavior changes are scoped to query-auth.enabled tables. Tables without
query auth and the write path are unaffected. System tables and
vector/full-text search reads do not consult auth rules (pre-existing;
restrict at the permission layer).
@plusplusjiajia plusplusjiajia force-pushed the core-mask-remap-fail-closed branch from 62849c7 to 17823a0 Compare July 12, 2026 16:52
@plusplusjiajia plusplusjiajia changed the title [WIP][core] Augment scan projection with column-mask input columns [core] Augment scan projection with column-mask input columns Jul 13, 2026
@JingsongLi

Copy link
Copy Markdown
Contributor

@plusplusjiajia plusplusjiajia changed the title [core] Augment scan projection with column-mask input columns [WIP][core] Augment scan projection with column-mask input columns Jul 13, 2026
@plusplusjiajia

Copy link
Copy Markdown
Member Author

@JingsongLi Agreed. I'll fold the masked-column pushdown safety into this PR so masking is correct end to end in one place, and close #8582. Updating this PR shortly.

@plusplusjiajia plusplusjiajia changed the title [WIP][core] Augment scan projection with column-mask input columns [WIP][core] Fix column masking correctness in query-auth reads Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants