Skip to content

[Feature] Add table function relation syntax to SQL grammar #5318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mengweieric merged 3 commits into from
Apr 7, 2026
+151 −0
Merged

[Feature] Add table function relation syntax to SQL grammar #5318

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
023d31f
[Feature] Add table function relation to SQL grammar for vectorSearch()
mengweieric Apr 6, 2026
7b30898
Address review feedback on table function relation grammar
mengweieric Apr 6, 2026
d73e73f
Apply spotless formatting
mengweieric Apr 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions sql/src/main/antlr/OpenSearchSQLParser.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,15 @@ fromClause
relation
: tableName (AS? alias)? # tableAsRelation
| LR_BRACKET subquery = querySpecification RR_BRACKET AS? alias # subqueryAsRelation
| qualifiedName LR_BRACKET tableFunctionArgs RR_BRACKET AS? alias # tableFunctionRelation
;

tableFunctionArgs
: tableFunctionArg (COMMA tableFunctionArg)*
;

tableFunctionArg
: ident EQUAL_SYMBOL functionArg
;

whereClause
Expand Down
22 changes: 22 additions & 0 deletions sql/src/main/java/org/opensearch/sql/sql/parser/AstBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,19 +13,22 @@
import static org.opensearch.sql.sql.antlr.parser.OpenSearchSQLParser.SelectElementContext;
import static org.opensearch.sql.sql.antlr.parser.OpenSearchSQLParser.SubqueryAsRelationContext;
import static org.opensearch.sql.sql.antlr.parser.OpenSearchSQLParser.TableAsRelationContext;
import static org.opensearch.sql.sql.antlr.parser.OpenSearchSQLParser.TableFunctionRelationContext;
import static org.opensearch.sql.sql.antlr.parser.OpenSearchSQLParser.WhereClauseContext;
import static org.opensearch.sql.sql.parser.ParserUtils.getTextInQuery;
import static org.opensearch.sql.utils.SystemIndexUtils.TABLE_INFO;
import static org.opensearch.sql.utils.SystemIndexUtils.mappingTable;

import com.google.common.collect.ImmutableList;
import java.util.Collections;
import java.util.Locale;
import java.util.Optional;
import lombok.RequiredArgsConstructor;
import org.antlr.v4.runtime.tree.ParseTree;
import org.opensearch.sql.ast.expression.Alias;
import org.opensearch.sql.ast.expression.AllFields;
import org.opensearch.sql.ast.expression.Function;
import org.opensearch.sql.ast.expression.UnresolvedArgument;
import org.opensearch.sql.ast.expression.UnresolvedExpression;
import org.opensearch.sql.ast.tree.DescribeRelation;
import org.opensearch.sql.ast.tree.Filter;
Expand All @@ -34,6 +37,7 @@
import org.opensearch.sql.ast.tree.Relation;
import org.opensearch.sql.ast.tree.RelationSubquery;
import org.opensearch.sql.ast.tree.SubqueryAlias;
import org.opensearch.sql.ast.tree.TableFunction;
import org.opensearch.sql.ast.tree.UnresolvedPlan;
import org.opensearch.sql.ast.tree.Values;
import org.opensearch.sql.common.antlr.SyntaxCheckException;
Expand Down Expand Up @@ -189,6 +193,24 @@ public UnresolvedPlan visitSubqueryAsRelation(SubqueryAsRelationContext ctx) {
return new RelationSubquery(visit(ctx.subquery), subqueryAlias);
}

@Override
public UnresolvedPlan visitTableFunctionRelation(TableFunctionRelationContext ctx) {
ImmutableList.Builder<UnresolvedExpression> args = ImmutableList.builder();
ctx.tableFunctionArgs()
.tableFunctionArg()
.forEach(
arg -> {
String argName =
StringUtils.unquoteIdentifier(arg.ident().getText()).toLowerCase(Locale.ROOT);
Copy link
Copy Markdown
Collaborator

@dai-chen dai-chen Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does visitAstExpression (visitIdent) handle this already?

Copy link
Copy Markdown
Collaborator Author

@mengweieric mengweieric Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. visitIdent() only unquotes identifiers; it does not lowercase them. In this path, the arg name is stored in UnresolvedArgument, and ExpressionAnalyzer.visitUnresolvedArgument() preserves that name when creating NamedArgumentExpression. The explicit toLowerCase(Locale.ROOT) in visitTableFunctionRelation() is therefore what canonicalizes named arg keys for table-function relations.

Copy link
Copy Markdown
Collaborator

@RyanL1997 RyanL1997 Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mengweieric — that confirms the lowercase is the only thing canonicalizing arg keys on this path. My concern isn't whether it's needed, but where it lives.

Since UnresolvedArgument → NamedArgumentExpression is a shared pipeline that PPL also feeds (PPL's visitTableFunction doesn't lowercase, it just uses arg.ident().getText()), we now have two front-ends with different casing policies converging on the same TableFunctionResolver. The existing Prometheus resolver compares names with a case-sensitive HashSet.removeAll (TableFunctionUtils.java#L73-L89), so a future vectorSearch resolver that declares args in camelCase (topK, filterType) will silently work in PPL and fail in SQL with Missing arguments:[topK,filterType].

Two ways to keep them in sync — what's your preference?

  1. Move the toLowerCase into ExpressionAnalyzer.visitUnresolvedArgument (or Analyzer.visitTableFunction) so both parsers get the same canonicalization for free.
  2. Keep it here, but also apply it in PPL's visitTableFunction and add a javadoc on TableFunction.java documenting the contract so future resolver authors know to declare arg names in lowercase.

Not a blocker tbh — I was trying to avoid the "works in PPL, fails in SQL" footgun before vectorSearch lands on top of this.

Copy link
Copy Markdown
Collaborator Author

@mengweieric mengweieric Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My preference would be to centralize table-function arg canonicalization in Analyzer.visitTableFunction rather than ExpressionAnalyzer, so both front-ends share one table-function-specific policy without changing all named-arg handling globally. For this PR I kept the fix local to the SQL parser to minimize scope.

UnresolvedExpression argValue = visitAstExpression(arg.functionArg());
args.add(new UnresolvedArgument(argName, argValue));
});
TableFunction tableFunction =
new TableFunction(visitAstExpression(ctx.qualifiedName()), args.build());
String alias = StringUtils.unquoteIdentifier(ctx.alias().getText());
return new SubqueryAlias(alias, tableFunction);
}

@Override
public UnresolvedPlan visitWhereClause(WhereClauseContext ctx) {
return new Filter(visitAstExpression(ctx.expression()));
Expand Down
120 changes: 120 additions & 0 deletions sql/src/test/java/org/opensearch/sql/sql/parser/AstBuilderTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@
import org.opensearch.sql.ast.expression.DataType;
import org.opensearch.sql.ast.expression.Literal;
import org.opensearch.sql.ast.expression.NestedAllTupleFields;
import org.opensearch.sql.ast.expression.UnresolvedArgument;
import org.opensearch.sql.ast.tree.SubqueryAlias;
import org.opensearch.sql.ast.tree.TableFunction;
import org.opensearch.sql.common.antlr.SyntaxCheckException;

class AstBuilderTest extends AstBuilderTestBase {
Expand Down Expand Up @@ -131,6 +134,123 @@ public void can_build_from_index_with_alias_quoted() {
buildAST("SELECT `t`.name FROM test `t` WHERE `t`.age = 30"));
}

@Test
public void can_build_from_table_function() {
assertEquals(
project(
new SubqueryAlias(
"v",
new TableFunction(
qualifiedName("vectorSearch"),
ImmutableList.of(
new UnresolvedArgument("table", stringLiteral("products")),
new UnresolvedArgument("field", stringLiteral("embedding")),
new UnresolvedArgument("vector", stringLiteral("[0.1,0.2]")),
new UnresolvedArgument("option", stringLiteral("k=10"))))),
AllFields.of()),
buildAST(
"SELECT * FROM vectorSearch("
+ "table='products', field='embedding', "
+ "vector='[0.1,0.2]', option='k=10') AS v"));
}

@Test
public void can_build_from_table_function_with_where_order_limit() {
assertEquals(
project(
limit(
sort(
filter(
new SubqueryAlias(
"s",
new TableFunction(
qualifiedName("vectorSearch"),
ImmutableList.of(
new UnresolvedArgument("table", stringLiteral("products")),
new UnresolvedArgument("field", stringLiteral("embedding")),
new UnresolvedArgument("vector", stringLiteral("[0.1,0.2]")),
new UnresolvedArgument("option", stringLiteral("k=10"))))),
function("=", qualifiedName("s", "category"), stringLiteral("shoes"))),
field(qualifiedName("s", "_score"), argument("asc", booleanLiteral(false)))),
5,
0),
alias("s.title", qualifiedName("s", "title")),
alias("s._score", qualifiedName("s", "_score"))),
buildAST(
"SELECT s.title, s._score FROM vectorSearch("
+ "table='products', field='embedding', "
+ "vector='[0.1,0.2]', option='k=10') AS s "
+ "WHERE s.category = 'shoes' "
+ "ORDER BY s._score DESC "
+ "LIMIT 5"));
}

@Test
public void table_function_args_are_resolved_by_name_not_position() {
assertEquals(
project(
new SubqueryAlias(
"v",
new TableFunction(
qualifiedName("vectorSearch"),
ImmutableList.of(
new UnresolvedArgument("option", stringLiteral("k=10")),
new UnresolvedArgument("field", stringLiteral("embedding")),
new UnresolvedArgument("table", stringLiteral("products")),
new UnresolvedArgument("vector", stringLiteral("[0.1,0.2]"))))),
AllFields.of()),
buildAST(
"SELECT * FROM vectorSearch("
+ "option='k=10', field='embedding', "
+ "table='products', vector='[0.1,0.2]') AS v"));
}

@Test
public void table_function_arg_names_are_canonicalized() {
assertEquals(
project(
new SubqueryAlias(
"v",
new TableFunction(
qualifiedName("vectorSearch"),
ImmutableList.of(
new UnresolvedArgument("table", stringLiteral("products")),
new UnresolvedArgument("field", stringLiteral("embedding")),
new UnresolvedArgument("vector", stringLiteral("[0.1,0.2]")),
new UnresolvedArgument("option", stringLiteral("k=10"))))),
AllFields.of()),
buildAST(
"SELECT * FROM vectorSearch("
+ "TABLE='products', FIELD='embedding', "
+ "VECTOR='[0.1,0.2]', OPTION='k=10') AS v"));
}

@Test
public void table_function_allows_alias_without_as_keyword() {
assertEquals(
project(
new SubqueryAlias(
"v",
new TableFunction(
qualifiedName("vectorSearch"),
ImmutableList.of(
new UnresolvedArgument("table", stringLiteral("products")),
new UnresolvedArgument("vector", stringLiteral("[0.1]"))))),
AllFields.of()),
buildAST("SELECT * FROM vectorSearch(table='products', vector='[0.1]') v"));
}

@Test
public void table_function_relation_requires_alias() {
assertThrows(
SyntaxCheckException.class,
() ->
buildAST(
"SELECT * FROM vectorSearch("
+ "table='products', field='embedding', "
+ "vector='[0.1,0.2]', option='k=10')"));
}

@Test
public void can_build_where_clause() {
assertEquals(
Expand Down
Loading