Skip to content

Commit c8e7cd1

Browse files
committed
SQL: Show/desc commands now support table ids
Extend SHOW TABLES, DESCRIBE and SHOW COLUMNS to support table identifiers not just SQL LIKE pattern. This allows both Elasticsearch-style multi-index patterns and SQL LIKE. To disambiguate between the two (as the " vs ' can be easy to miss), the grammar now requires LIKE keyword as a prefix for all LIKE-like patterns. Also added some docs comparing the two types of patterns. Fix elastic#33294
1 parent 84eaac7 commit c8e7cd1

25 files changed

+1400
-946
lines changed

docs/reference/sql/concepts.asciidoc

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ NOTE: This documentation while trying to be complete, does assume the reader has
99

1010
As a general rule, {es-sql} as the name indicates provides a SQL interface to {es}. As such, it follows the SQL terminology and conventions first, whenever possible. However the backing engine itself is {es} for which {es-sql} was purposely created hence why features or concepts that are not available, or cannot be mapped correctly, in SQL appear
1111
in {es-sql}.
12-
Last but not least, {es-sql} tries to obey the https://en.wikipedia.org/wiki/Principle_of_least_astonishment[principle of least suprise], though as all things in the world, everything is relative.
12+
Last but not least, {es-sql} tries to obey the https://en.wikipedia.org/wiki/Principle_of_least_astonishment[principle of least surprise], though as all things in the world, everything is relative.
1313

1414
=== Mapping concepts across SQL and {es}
1515

@@ -26,7 +26,7 @@ So let's start from the bottom; these roughly are:
2626
|`column`
2727
|`field`
2828
|In both cases, at the lowest level, data is stored in _named_ entries, of a variety of <<sql-data-types, data types>>, containing _one_ value. SQL calls such an entry a _column_ while {es} a _field_.
29-
Notice that in {es} a field can contain _multiple_ values of the same type (esentially a list) while in SQL, a _column_ can contain _exactly_ one value of said type.
29+
Notice that in {es} a field can contain _multiple_ values of the same type (essentially a list) while in SQL, a _column_ can contain _exactly_ one value of said type.
3030
{es-sql} will do its best to preserve the SQL semantic and, depending on the query, reject those that return fields with more than one value.
3131

3232
|`row`
@@ -43,7 +43,7 @@ Notice that in {es} a field can contain _multiple_ values of the same type (esen
4343

4444
|`catalog` or `database`
4545
|`cluster` instance
46-
|In SQL, `catalog` or `database` are used interchangebly and represent a set of schemas that is, a number of tables.
46+
|In SQL, `catalog` or `database` are used interchangeably and represent a set of schemas that is, a number of tables.
4747
In {es} the set of indices available are grouped in a `cluster`. The semantics also differ a bit; a `database` is essentially yet another namespace (which can have some implications on the way data is stored) while an {es} `cluster` is a runtime instance, or rather a set of at least one {es} instance (typically running distributed).
4848
In practice this means that while in SQL one can potentially have multiple catalogs inside an instance, in {es} one is restricted to only _one_.
4949

@@ -62,4 +62,4 @@ Multiple clusters, each with its own namespace, connected to each other in a fed
6262

6363
|===
6464

65-
As one can see while the mapping between the concepts are not exactly one to one and the semantics somewhat different, there are more things in common than differences. In fact, thanks to SQL declarative nature, many concepts can move across {es} transparently and the terminology of the two likely to be used interchangebly through-out the rest of the material.
65+
As one can see while the mapping between the concepts are not exactly one to one and the semantics somewhat different, there are more things in common than differences. In fact, thanks to SQL declarative nature, many concepts can move across {es} transparently and the terminology of the two likely to be used interchangeably through-out the rest of the material.
Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
[role="xpack"]
2+
[testenv="basic"]
3+
[[sql-index-pattern]]
4+
== Index patterns
5+
6+
{es-sql} supports two types of patterns for matching multiple indices or tables:
7+
8+
* {es} multi-index
9+
10+
The {es} notation for enumerating, including or excluding <<multi-index,multi index syntax>>
11+
is supported _as long_ as it is quoted or escaped as a table identifier.
12+
13+
For example:
14+
15+
["source","sql",subs="attributes,callouts,macros"]
16+
----
17+
include-tagged::{sql-specs}/docs.csv-spec[showTablesEsMultiIndex]
18+
----
19+
20+
Notice the pattern is surrounded by double quotes `"`. It enumerated `*` meaning all indices however
21+
it excludes (due to `-`) all indices that start with `l`.
22+
This notation is very convenient and powerful as it allows both inclusion and exclusion, depending on
23+
the target naming convention.
24+
25+
* SQL `LIKE` notation
26+
27+
The common `LIKE` statement (including escaping if needed) to match a wildcard pattern, based on one `_`
28+
or multiple `%` characters.
29+
30+
Using `SHOW TABLES` command again:
31+
32+
["source","sql",subs="attributes,callouts,macros"]
33+
----
34+
include-tagged::{sql-specs}/docs.csv-spec[showTablesLikeWildcard]
35+
----
36+
37+
The pattern matches all tables that start with `emp`.
38+
39+
This command supports _escaping_ as well, for example:
40+
41+
["source","sql",subs="attributes,callouts,macros"]
42+
----
43+
include-tagged::{sql-specs}/docs.csv-spec[showTablesLikeEscape]
44+
----
45+
46+
Notice how now `emp%` does not match any tables because `%`, which means match zero or more characters,
47+
has been escaped by `!` and thus becomes an regular char. And since there is no table named `emp%`,
48+
an empty table is returned.
49+
50+
In a nutshell, the differences between the two type of patterns are:
51+
52+
[cols="^h,^,^",options="header"]
53+
|===
54+
| Feature | Multi index | SQL `LIKE`
55+
56+
| Type of quoting | `"` | `'`
57+
| Inclusion | Yes | Yes
58+
| Exclusion | Yes | No
59+
| Enumeration | Yes | No
60+
| One char pattern | No | `_`
61+
| Multi char pattern | `*` | `%`
62+
| Escaping | No | `ESCAPE`
63+
64+
|===
65+
66+
Which one to use, is up to you however try to stick to the same one across your queries for consistency.
67+
68+
NOTE: As the query type of quoting between the two patterns is fairly similar (`"` vs `'`), {es-sql} _always_
69+
requires the keyword `LIKE` for SQL `LIKE` pattern.
70+

docs/reference/sql/language/index.asciidoc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ This chapter describes the SQL semantics supported in X-Pack namely:
77

88
<<sql-data-types>>:: Data types
99
<<sql-commands>>:: Commands
10+
<<sql-index-patterns>>:: Index patterns
1011

1112
include::data-types.asciidoc[]
1213
include::syntax/index.asciidoc[]
14+
include::index-patterns.asciidoc[]

docs/reference/sql/language/syntax/describe-table.asciidoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,14 +6,14 @@
66
.Synopsis
77
[source, sql]
88
----
9-
DESCRIBE table
9+
DESCRIBE [table identifier<1>|[LIKE pattern<2>]]
1010
----
1111

1212
or
1313

1414
[source, sql]
1515
----
16-
DESC table
16+
DESC [table identifier<1>|[LIKE pattern<2>]]
1717
----
1818

1919

docs/reference/sql/language/syntax/show-columns.asciidoc

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,15 @@
66
.Synopsis
77
[source, sql]
88
----
9-
SHOW COLUMNS [ FROM | IN ] ? table
9+
SHOW COLUMNS [ FROM | IN ]? [ table identifier<1> | [ LIKE pattern<2> ] ]
1010
----
1111

12+
<1> single table identifier or double quoted es multi index
13+
<2> SQL LIKE pattern
14+
15+
See <<sql-index-pattern, index patterns>> for more information about
16+
patterns.
17+
1218
.Description
1319

1420
List the columns in table and their data type (and other attributes).
@@ -17,3 +23,4 @@ List the columns in table and their data type (and other attributes).
1723
----
1824
include-tagged::{sql-specs}/docs.csv-spec[showColumns]
1925
----
26+

docs/reference/sql/language/syntax/show-functions.asciidoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
.Synopsis
77
[source, sql]
88
----
9-
SHOW FUNCTIONS [ LIKE? pattern<1>? ]?
9+
SHOW FUNCTIONS [ LIKE pattern<1>? ]?
1010
----
1111

1212
<1> SQL match pattern

docs/reference/sql/language/syntax/show-tables.asciidoc

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,15 @@
66
.Synopsis
77
[source, sql]
88
----
9-
SHOW TABLES [ LIKE? pattern<1>? ]?
9+
SHOW TABLES [ table identifier<1> | [ LIKE pattern<2> ] ]?
1010
----
1111

12-
<1> SQL match pattern
12+
<1> single table identifier or double quoted es multi index
13+
<2> SQL LIKE pattern
14+
15+
See <<sql-index-pattern, index patterns>> for more information about
16+
patterns.
17+
1318

1419
.Description
1520

@@ -20,7 +25,15 @@ List the tables available to the current user and their type.
2025
include-tagged::{sql-specs}/docs.csv-spec[showTables]
2126
----
2227

23-
The `LIKE` clause can be used to restrict the list of names to the given pattern.
28+
Match multiple indices by using {es} <<multi-index,multi index syntax>>
29+
notation:
30+
31+
["source","sql",subs="attributes,callouts,macros"]
32+
----
33+
include-tagged::{sql-specs}/docs.csv-spec[showTablesEsMultiIndex]
34+
----
35+
36+
One can also use the `LIKE` clause to restrict the list of names to the given pattern.
2437

2538
The pattern can be an exact match:
2639
["source","sql",subs="attributes,callouts,macros"]

x-pack/plugin/sql/src/main/antlr/SqlBase.g4

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -50,18 +50,18 @@ statement
5050
)*
5151
')')?
5252
statement #debug
53-
| SHOW TABLES (LIKE? pattern)? #showTables
54-
| SHOW COLUMNS (FROM | IN) tableIdentifier #showColumns
55-
| (DESCRIBE | DESC) tableIdentifier #showColumns
56-
| SHOW FUNCTIONS (LIKE? pattern)? #showFunctions
53+
| SHOW TABLES (tableLike=likePattern | tableIdent=tableIdentifier)? #showTables
54+
| SHOW COLUMNS (FROM | IN) (tableLike=likePattern | tableIdent=tableIdentifier) #showColumns
55+
| (DESCRIBE | DESC) (tableLike=likePattern | tableIdent=tableIdentifier) #showColumns
56+
| SHOW FUNCTIONS (likePattern)? #showFunctions
5757
| SHOW SCHEMAS #showSchemas
5858
| SYS CATALOGS #sysCatalogs
59-
| SYS TABLES (CATALOG LIKE? clusterPattern=pattern)?
60-
(LIKE? tablePattern=pattern)?
59+
| SYS TABLES (CATALOG clusterLike=likePattern)?
60+
(tableLike=likePattern | tableIdent=tableIdentifier)?
6161
(TYPE string (',' string)* )? #sysTables
6262
| SYS COLUMNS (CATALOG cluster=string)?
63-
(TABLE LIKE? indexPattern=pattern)?
64-
(LIKE? columnPattern=pattern)? #sysColumns
63+
(TABLE tableLike=likePattern | tableIdent=tableIdentifier)?
64+
(columnPattern=likePattern)? #sysColumns
6565
| SYS TYPES #sysTypes
6666
| SYS TABLE TYPES #sysTableTypes
6767
;
@@ -189,6 +189,10 @@ predicate
189189
| IS NOT? kind=NULL
190190
;
191191

192+
likePattern
193+
: LIKE pattern
194+
;
195+
192196
pattern
193197
: value=string patternEscape?
194198
;

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/analysis/index/IndexResolver.java

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -140,49 +140,53 @@ public void resolveNames(String indexWildcard, String javaRegex, EnumSet<IndexTy
140140
boolean retrieveAliases = CollectionUtils.isEmpty(types) || types.contains(IndexType.ALIAS);
141141
boolean retrieveIndices = CollectionUtils.isEmpty(types) || types.contains(IndexType.INDEX);
142142

143+
String[] indices = Strings.commaDelimitedListToStringArray(indexWildcard);
143144
if (retrieveAliases) {
144145
GetAliasesRequest aliasRequest = new GetAliasesRequest()
145146
.local(true)
146-
.aliases(indexWildcard)
147+
.indices(indices)
148+
.aliases(indices)
147149
.indicesOptions(IndicesOptions.lenientExpandOpen());
148150

149151
client.admin().indices().getAliases(aliasRequest, ActionListener.wrap(aliases ->
150-
resolveIndices(indexWildcard, javaRegex, aliases, retrieveIndices, listener),
152+
resolveIndices(indices, javaRegex, aliases, retrieveIndices, listener),
151153
ex -> {
152154
// with security, two exception can be thrown:
153155
// INFE - if no alias matches
154156
// security exception is the user cannot access aliases
155157

156158
// in both cases, that is allowed and we continue with the indices request
157159
if (ex instanceof IndexNotFoundException || ex instanceof ElasticsearchSecurityException) {
158-
resolveIndices(indexWildcard, javaRegex, null, retrieveIndices, listener);
160+
resolveIndices(indices, javaRegex, null, retrieveIndices, listener);
159161
} else {
160162
listener.onFailure(ex);
161163
}
162164
}));
163165
} else {
164-
resolveIndices(indexWildcard, javaRegex, null, retrieveIndices, listener);
166+
resolveIndices(indices, javaRegex, null, retrieveIndices, listener);
165167
}
166168
}
167169

168-
private void resolveIndices(String indexWildcard, String javaRegex, GetAliasesResponse aliases,
170+
private void resolveIndices(String[] indices, String javaRegex, GetAliasesResponse aliases,
169171
boolean retrieveIndices, ActionListener<Set<IndexInfo>> listener) {
170172

171173
if (retrieveIndices) {
172174
GetIndexRequest indexRequest = new GetIndexRequest()
173175
.local(true)
174-
.indices(indexWildcard)
176+
.indices(indices)
177+
.features(Feature.SETTINGS)
178+
.includeDefaults(false)
175179
.indicesOptions(IndicesOptions.lenientExpandOpen());
176180

177181
client.admin().indices().getIndex(indexRequest,
178-
ActionListener.wrap(indices -> filterResults(indexWildcard, javaRegex, aliases, indices, listener),
182+
ActionListener.wrap(response -> filterResults(javaRegex, aliases, response, listener),
179183
listener::onFailure));
180184
} else {
181-
filterResults(indexWildcard, javaRegex, aliases, null, listener);
185+
filterResults(javaRegex, aliases, null, listener);
182186
}
183187
}
184188

185-
private void filterResults(String indexWildcard, String javaRegex, GetAliasesResponse aliases, GetIndexResponse indices,
189+
private void filterResults(String javaRegex, GetAliasesResponse aliases, GetIndexResponse indices,
186190
ActionListener<Set<IndexInfo>> listener) {
187191

188192
// since the index name does not support ?, filter the results manually
@@ -302,7 +306,6 @@ private static GetIndexRequest createGetIndexRequest(String index) {
302306
return new GetIndexRequest()
303307
.local(true)
304308
.indices(Strings.commaDelimitedListToStringArray(index))
305-
.features(Feature.MAPPINGS)
306309
//lenient because we throw our own errors looking at the response e.g. if something was not resolved
307310
//also because this way security doesn't throw authorization exceptions but rather honours ignore_unavailable
308311
.indicesOptions(IndicesOptions.lenientExpandOpen());

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/parser/CommandBuilder.java

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -121,12 +121,14 @@ public Command visitExplain(ExplainContext ctx) {
121121

122122
@Override
123123
public Object visitShowFunctions(ShowFunctionsContext ctx) {
124-
return new ShowFunctions(source(ctx), visitPattern(ctx.pattern()));
124+
return new ShowFunctions(source(ctx), visitLikePattern(ctx.likePattern()));
125125
}
126126

127127
@Override
128128
public Object visitShowTables(ShowTablesContext ctx) {
129-
return new ShowTables(source(ctx), visitPattern(ctx.pattern()));
129+
TableIdentifier ti = visitTableIdentifier(ctx.tableIdent);
130+
String index = ti != null ? ti.qualifiedIndex() : null;
131+
return new ShowTables(source(ctx), index, visitLikePattern(ctx.likePattern()));
130132
}
131133

132134
@Override
@@ -136,8 +138,9 @@ public Object visitShowSchemas(ShowSchemasContext ctx) {
136138

137139
@Override
138140
public Object visitShowColumns(ShowColumnsContext ctx) {
139-
TableIdentifier identifier = visitTableIdentifier(ctx.tableIdentifier());
140-
return new ShowColumns(source(ctx), identifier.index());
141+
TableIdentifier ti = visitTableIdentifier(ctx.tableIdent);
142+
String index = ti != null ? ti.qualifiedIndex() : null;
143+
return new ShowColumns(source(ctx), index, visitLikePattern(ctx.likePattern()));
141144
}
142145

143146
@Override
@@ -172,13 +175,17 @@ else if (value.toUpperCase(Locale.ROOT).equals("TABLE")) {
172175

173176
// if the ODBC enumeration is specified, skip validation
174177
EnumSet<IndexType> set = types.isEmpty() ? null : EnumSet.copyOf(types);
175-
return new SysTables(source(ctx), visitPattern(ctx.clusterPattern), visitPattern(ctx.tablePattern), set, legacyTableType);
178+
TableIdentifier ti = visitTableIdentifier(ctx.tableIdent);
179+
String index = ti != null ? ti.qualifiedIndex() : null;
180+
return new SysTables(source(ctx), visitLikePattern(ctx.clusterLike), index, visitLikePattern(ctx.tableLike), set, legacyTableType);
176181
}
177182

178183
@Override
179184
public Object visitSysColumns(SysColumnsContext ctx) {
180-
Location loc = source(ctx);
181-
return new SysColumns(loc, string(ctx.cluster), visitPattern(ctx.indexPattern), visitPattern(ctx.columnPattern));
185+
TableIdentifier ti = visitTableIdentifier(ctx.tableIdent);
186+
String index = ti != null ? ti.qualifiedIndex() : null;
187+
return new SysColumns(source(ctx), string(ctx.cluster), index, visitLikePattern(ctx.tableLike),
188+
visitLikePattern(ctx.columnPattern));
182189
}
183190

184191
@Override
@@ -190,4 +197,4 @@ public SysTypes visitSysTypes(SysTypesContext ctx) {
190197
public Object visitSysTableTypes(SysTableTypesContext ctx) {
191198
return new SysTableTypes(source(ctx));
192199
}
193-
}
200+
}

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/parser/ExpressionBuilder.java

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@
6363
import org.elasticsearch.xpack.sql.parser.SqlBaseParser.FunctionTemplateContext;
6464
import org.elasticsearch.xpack.sql.parser.SqlBaseParser.GuidEscapedLiteralContext;
6565
import org.elasticsearch.xpack.sql.parser.SqlBaseParser.IntegerLiteralContext;
66+
import org.elasticsearch.xpack.sql.parser.SqlBaseParser.LikePatternContext;
6667
import org.elasticsearch.xpack.sql.parser.SqlBaseParser.LogicalBinaryContext;
6768
import org.elasticsearch.xpack.sql.parser.SqlBaseParser.LogicalNotContext;
6869
import org.elasticsearch.xpack.sql.parser.SqlBaseParser.MatchQueryContext;
@@ -220,6 +221,11 @@ public Expression visitPredicated(PredicatedContext ctx) {
220221
return pCtx.NOT() != null ? new Not(loc, e) : e;
221222
}
222223

224+
@Override
225+
public LikePattern visitLikePattern(LikePatternContext ctx) {
226+
return ctx == null ? null : visitPattern(ctx.pattern());
227+
}
228+
223229
@Override
224230
public LikePattern visitPattern(PatternContext ctx) {
225231
if (ctx == null) {

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/parser/IdentifierBuilder.java

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,10 @@ abstract class IdentifierBuilder extends AbstractBuilder {
1717

1818
@Override
1919
public TableIdentifier visitTableIdentifier(TableIdentifierContext ctx) {
20+
if (ctx == null) {
21+
return null;
22+
}
23+
2024
Location source = source(ctx);
2125
ParseTree tree = ctx.name != null ? ctx.name : ctx.TABLE_IDENTIFIER();
2226
String index = tree.getText();

0 commit comments

Comments
 (0)