Use Calcite's validation system for type checking & coercion

[yuancu]

Description

Please review the sub-PRs instead: #4892 (comment)

This PR migrates from our custom type checking mechanism to Apache Calcite's native validation system by introducing a SqlNode validation layer. This addresses the lack of a proper SQL validation phase and resolves compatibility issues with user-defined types (UDTs).

This implements Approach 1 described in #3865.

How It Works

The PR introduces a validation layer in the query execution pipeline:

RelNode → SqlNode → [Validation & Type Coercion] → SqlNode → RelNode

This approach leverages Calcite's robust type system while preserving OpenSearch's custom type semantics through careful type mapping and restoration.

Benefits

• Comprehensive type checking: Access to Calcite's full type checker suite (variable-length operands, same-operand constraints, OR-relationship type checkers) vs. our limited family-based checking
• Function optimization: Ability to rewrite functions at the SqlNode level (e.g., sqrt(x) → pow(x, 0.5)) before generating physical plans
• Automatic type coercion: Implicit casting where type-safe and appropriate, improving query compatibility
• Reduced maintenance: Leverage Calcite's battle-tested validation logic instead of maintaining custom implementations

Work Items / Implementation Details

Core Infrastructure

• Introduced SqlNode validation layer with round-trip conversion (RelNode ↔ SqlNode)
• Extended Calcite validator to support OpenSearch UDTs via type substitution
• Implemented custom deriveType, coerceOperandType, and type inference methods
• Made validator thread-safe with proper synchronization

Type System Enhancements

• UDT Support: To support OpenSearch-specific UDTs (which Calcite doesn't natively support), we bridge Calcite's SqlTypeName enum with OpenSearch UDTs through dynamic type mapping:
  1. Convert UDTs to standard SQL types (e.g., EXPR_TIME → SqlTypeName.TIME) before validation
  2. Apply Calcite's validation and type coercion
  3. Restore UDTs after validation to maintain plan correctness
• Cross-type Comparisons: Override commonTypeForBinaryComparison to enable datetime cross-type comparisons (DATE vs TIME, etc.)
• IP Type Handling: Emulated IP type using standard SQL types for validation, with special handling in operators
• Composite Types: Extended SQL-UDT conversion to handle nested/composite types
• Type Coercion: Use SAFE_CAST for string-to-number conversions to tolerate malformatted data; use CAST for literal numbers

Function & Operator Handling

• Defined operand type checkers for all built-in functions:
  • Array functions (array_slice, reduce, mvappend, etc.) with proper type inference
  • JSON functions (json_extract, json_set, etc.)
  • Mathematical functions (corrected atan overloading, percentile approximations)
  • Aggregation functions (DISTINCT_COUNT_APPROX, COUNT(*) rewriting, etc.)
• Function overloading support:
  • ADD operator: String concatenation vs numeric addition
  • ATAN: Single-operand vs two-operand versions
  • GEOIP: String overrides due to UDT erasure
• Arithmetic operations between strings and numerics (implicit coercion)
• Define SqlKind for DIVIDE and MOD UDFs

Query Construct Support

• Preserve null ordering and collation through SqlNode round-trips
• Support SEMI and ANTI joins in SQL conversion
• Preserve sort orders in subqueries
• Pass RelHint through conversions (added SqlHint for LogicalAggregate)
• Handle windowed aggregates and bucket_nullable flags
• Support IN/NOT IN with tuple inputs via row constructor rewriting

Dialect & Compatibility

• Extended OpenSearch Spark SQL dialect for custom syntax
• Fixed interval semantics mismatch between SQL and PPL
• Fixed quarter interval bug in Calcite
• Handle identifier expansion in aggregate functions
• Properly handle LogicalValues for empty row generation

Edge Cases & Fixes

• Skip validation for unsupported patterns:
  • Bin-on-timestamp operations (not yet implemented)
  • Group-by window functions (return original plan gracefully)
  • Specific aggregation patterns with LogicalValues
• Fixed nullability attribute preservation through SAFE_CAST
• Trim unused fields after SQL→Rel conversion
• Ensure RelToSqlConverter is instantiated per-use (stateful component)
• Fixed float literal handling with explicit casts
• Remove JSON_TYPE operator insertion where inappropriate

Test Fixes

• Fixed integration tests across multiple suites:
  • CalcitePPLDateTimeBuiltinFunctionIT: Interval semantics
  • CalcitePPLBuiltinFunctionIT: LOG function, sarg deserialization
  • CalciteArrayFunctionIT: Type checkers, reduce function inference
  • CalciteMathematicalFunctionIT, CalcitePPLAggregationIT
  • CalciteBinCommandIT: Timestamp operations, windowed aggregates in GROUP BY
  • CalciteStreamstatsCommandIT: Sort columns, bucket_nullable
  • CalcitePPLJsonBuiltinFunctionIT: String conversion
• Updated all explain integration tests for plan changes
• Fixed YAML tests, doctests, and unit tests
• Updated ClickBench query plans

Code Cleanup

• Removed legacy type checking from PPLFuncImpTable
• Deprecated UDFOperandMetadata.wrapUDT interface
• Removed unused classes: CalciteFuncSignature, PPLTypeChecker
• Consolidated EnhancedCoalesce into built-in Coalesce
• Removed type checkers from operator registration (now handled by Calcite)

Optimizations

• Eliminated SAFE_CAST on non-string literal numbers (use CAST for better performance)
• Investigated and addressed dedup optimization issues

Performance Impact

Dataset	Scenario	Analyze (ms)	Optimize (ms)	Execute (ms)	Total Avg (ms)
clickbench	Without Validation	7.93	8.29	0.95	17.53
	With Validation	9.59	8.72	1.00	19.69
	Difference	+1.66 (+20.9%)	+0.43 (+5.2%)	+0.05 (+5.3%)	+2.16 (+12.3%)
big5	Without Validation	3.52	4.27	0.80	8.99
	With Validation	3.37	3.80	0.72	8.22
	Difference	-0.15 (-4.3%)	-0.47 (-11.0%)	-0.08 (-10.0%)	-0.77 (-8.6%)
tpch	Without Validation	8.65	692.00	183.15	1028.80
	With Validation	7.91	689.16	175.63	1008.77
	Difference	-0.74 (-8.6%)	-2.84 (-0.4%)	-7.52 (-4.1%)	-20.03 (-1.9%)

Profiled on personal laptop, each test runs twice, then averaged.

Conclusion: No significant performance degradation. ClickBench shows slight overhead (+12.3%) during analyze phase due to validation, but big5 and TPCH show improvements, likely from better query optimization enabled by proper type information.

Related Issues

Resolves #4636, resolves #3865, resolves #5175

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

• New Features
  • Calcite-based validation with implicit type coercion and LEFT SEMI/ANTI JOIN support; expanded IP and datetime casting; stronger UDF operand validation and broader variadic support.
• Bug Fixes
  • More reliable window/ORDER BY behavior, float/interval literal handling, mixed-type IN/BETWEEN predicates, datetime binning, and safer dynamic-field handling in spath.
• Documentation
  • Explain outputs moved to YAML; several examples marked non-executable (ignore) pending fixes.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Walkthrough

Introduces extensive Calcite-based PPL validation and type-coercion plumbing, new PPL-specific validator/convertlets/coercion rules, UDT/type utilities, many UDF operand-metadata updates, Rex/Rel shuttles and Rel↔Sql converters, removal of legacy coercion/type-checker code, and large test/expected-output updates across integ and unit tests.

Changes

Cohort / File(s)	Summary
Calcite validation core & providers core/src/main/java/org/opensearch/sql/calcite/CalcitePlanContext.java, core/src/main/java/org/opensearch/sql/calcite/validate/PplValidator.java, core/src/main/java/org/opensearch/sql/calcite/validate/SqlOperatorTableProvider.java, core/src/main/java/org/opensearch/sql/calcite/validate/PplTypeCoercion.java, core/src/main/java/org/opensearch/sql/calcite/validate/PplTypeCoercionRule.java, core/src/main/java/org/opensearch/sql/calcite/validate/PplConvertletTable.java	Add provider interface, new PplValidator, PPL-specific TypeCoercion/Rule and convertlet table; expose validator creation and operator-table injection.
Rel/Sql conversion & shuttles core/src/main/java/org/opensearch/sql/calcite/validate/converters/OpenSearchRelToSqlConverter.java, .../OpenSearchSqlToRelConverter.java, core/src/main/java/org/opensearch/sql/calcite/validate/shuttles/PplRelToSqlRelShuttle.java, .../SqlRewriteShuttle.java, .../SkipRelValidationShuttle.java	New Rel↔Sql converters and shuttles to translate RelNode→SqlNode→validate→RelNode, handle joins/hints, literal fixes, and selective validation skipping.
QueryService validation flow core/src/main/java/org/opensearch/sql/executor/QueryService.java	Integrates new validation step that converts RelNode→SqlNode, validates via Calcite, converts back; supports tolerant fallback and configurable skip.
Type utilities & factory changes core/src/main/java/org/opensearch/sql/calcite/utils/OpenSearchTypeUtil.java, .../OpenSearchTypeFactory.java, .../OpenSearchTypeFactory.leastRestrictive, core/src/main/java/org/opensearch/sql/calcite/validate/ValidationUtils.java	Add OpenSearchTypeUtil helpers, move UDT handling there, implement leastRestrictive override and utilities for syncing attributes/UDT creation.
Operand/type coercion & operator table refactor core/src/main/java/org/opensearch/sql/calcite/utils/PPLOperandTypes.java, core/src/main/java/org/opensearch/sql/expression/function/PPLBuiltinOperators.java, core/src/main/java/org/opensearch/sql/expression/function/PPLFuncImpTable.java	Replace composite/family wrappers with direct OperandTypes usage, introduce SCALAR checkers, migrate many operators from SqlOperator to SqlFunction, and simplify function/agg registries to single-implementation maps.
Rex/Rel visitors & builders core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java, .../CalciteRexNodeVisitor.java, .../ExtendedRexBuilder.java, core/src/main/java/org/opensearch/sql/calcite/utils/PlanUtils.java, core/src/main/java/org/opensearch/sql/calcite/DynamicFieldsHelper.java	Embed plan collations into RexOver windows, simplify window construction, adjust casting/type-derivation for mixed numeric/char, and small plan utility replacements to use OpenSearchTypeUtil.
UDF operand metadata & nullability annotations many files under core/src/main/java/org/opensearch/sql/expression/function/... (e.g., CollectionUDF/*, jsonUDF/*, udf/*, udf/datetime/*, udf/math/*, udf/ip/*, UserDefinedFunctionBuilder.java, UDFOperandMetadata.java, UserDefinedFunctionBuilder.java)	Make numerous getOperandMetadata() returns non-null, replace nulls with UDFOperandMetadata.wrap(...), migrate many operand checks to Calcite SqlOperandTypeChecker/OperandTypes, add @NonNull annotations, and introduce new custom checkers (e.g., IP, MAP repeats, transform/variadic semantics).
Removed legacy type-coercion framework core/src/main/java/org/opensearch/sql/expression/function/CoercionUtils.java, core/src/main/java/org/opensearch/sql/expression/function/PPLTypeChecker.java, core/src/main/java/org/opensearch/sql/expression/function/CalciteFuncSignature.java	Remove legacy CoercionUtils and PPLTypeChecker/CalciteFuncSignature artifacts; their responsibilities moved into Calcite validation/coercion paths.
New/changed validation helpers & convertlets core/src/main/java/org/opensearch/sql/calcite/validate/PplRelToSqlRelShuttle.java, core/src/main/java/org/opensearch/sql/calcite/validate/PplConvertletTable.java	Add convertlet table and rel→sql shuttle for literal normalization and operator remapping (e.g., IP operators).
Tests & expected outputs test additions/changes under core/src/test/..., integ-test/src/test/..., integ-test/src/test/resources/expectedOutput/calcite/*, api/src/test/...	Large test surface updates: many new/updated unit tests for coercion/validation/shuttles/type utils; numerous integration expected-plan YAML changes (many JSON→YAML resources replaced/added); update transpiler test import path.
Docs & examples docs/user/ppl/...	Minor docs/example updates (make code blocks ignore execution or formatting tweaks).

Sequence Diagram(s)

sequenceDiagram
    participant Rel as RelNode (PPL plan)
    participant Shuttle as PplRelToSqlRelShuttle
    participant RelToSql as OpenSearchRelToSqlConverter
    participant Validator as SqlValidator / PplValidator
    participant SqlToRel as OpenSearchSqlToRelConverter
    participant Planner as QueryService (convertToCalcitePlan)

    Rel->>Shuttle: traverse & normalize RexLiterals
    Shuttle->>RelToSql: convert RelNode -> SqlNode
    RelToSql->>Validator: validate SqlNode (type coercion, implicit casts)
    Validator-->>SqlToRel: produce validated SqlNode
    SqlToRel->>Planner: convert validated SqlNode -> RelNode (validated RelNode)
    Planner->>Planner: continue planning / optimize using validated RelNode

Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Possibly related PRs

• Introduce logical dedup operators for PPL #5014 — Overlaps changes to CalcitePlanContext and Calcite validation plumbing.
• Support spath with dynamic fields #5058 — Related dynamic-fields / SPATH feature work referenced by changes to DynamicFieldsHelper and SPATH tests.
• Implement spath command with field resolution #5028 — Related SPATH/field-resolution modifications and function registrations.

Suggested labels

backport 2.19-dev

Suggested reviewers

• ps48
• kavithacm
• derek-ho
• joshuali925
• dai-chen

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 19.72% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	Title concisely and accurately describes the primary change: adding Calcite-based validation for type checking and coercion.
Description check	✅ Passed	Description clearly documents the SqlNode validation round-trip, UDT mapping, and type-coercion work and matches the changeset.
Linked Issues check	✅ Passed	Implements SqlNode round-trip, PplValidator, PplTypeCoercion/Rule, UDT mapping and implicit coercion per linked objectives [#4636][#3865].
Out of Scope Changes check	✅ Passed	Changes align with the migration to Calcite validation/coercion; refactors, removals, and test updates appear directly related and in-scope.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 5ec52c9.

'Diff too large, requires skip by maintainers after manual review'

---

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

---

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

[github-actions]

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 3fd9714.

'Diff too large, requires skip by maintainers after manual review'

---

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

---

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

[github-actions]

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 5723ced.

'Diff too large, requires skip by maintainers after manual review'

---

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

---

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

[github-actions]

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 2c831f0.

'Diff too large, requires skip by maintainers after manual review'

---

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

---

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

[github-actions]

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 84f53df.

'Diff too large, requires skip by maintainers after manual review'

---

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

---

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

[yuancu]

PR Split Plan

As suggested by @LantaoJin, this PR has been split into 4 sub-PRs targeting the feature/validation branch:

Sub-PR	Title	Files	Dependency
#5213	Add validation infrastructure and type system	32	Independent
#5214	Define operand type checkers for all PPL built-in functions	86	Independent
#5215	Enable validation pipeline and update all test expected outputs	583	After #5213 + #5214
#5216	Remove legacy type checking code and update documentation	13	After #5215

Merge order: #5213 and #5214 can be reviewed/merged in parallel → #5215 → #5216 → merge feature/validation → main

Reviewer guide:

• Add validation infrastructure and type system #5213 (32 files): Core validation pipeline design — focus on round-trip correctness, graceful fallback, type coercion rules
• Define operand type checkers for all PPL built-in functions #5214 (86 files): Function operand types — highly repetitive pattern, batch-reviewable
• Enable validation pipeline and update all test expected outputs #5215 (583 files, ~546 YAML): If CI passes, the YAML updates are correct — focus on the ~37 source files
• Remove legacy type checking code and update documentation #5216 (13 files): Pure deletion + docs — quick review

[yuancu]

Note: This PR is superseded by 4 smaller sub-PRs targeting the feature/validation branch, as suggested in #4892 (comment).

Please review the sub-PRs instead:

Order	PR	Title	Files	Status
1	#5213	Add validation infrastructure and type system	32	Ready for review
2	#5214	Define operand type checkers for all PPL built-in functions	86	Ready for review
3	#5215	Enable validation pipeline and update all test expected outputs	583	Merge after #5213 + #5214
4	#5216	Remove legacy type checking code and update documentation	13	Merge after #5215

This PR will be closed once all sub-PRs are merged and feature/validation is merged back to main.

[opensearch-trigger-bot]

This PR is stalled because it has been open for 2 weeks with no activity.