Skip to content

Commit 70ea3cf

Browse files
imotovimotov
authored
SQL: Add initial geo support (#42031) (#42135)
Adds an initial limited implementations of geo features to SQL. This implementation is based on the [OpenGIS® Implementation Standard for Geographic information - Simple feature access](http://www.opengeospatial.org/standards/sfs), which is the current standard for GIS system implementation. This effort is concentrate on SQL option AKA ISO 19125-2. Queries that are supported as a result of this initial implementation Metadata commands - `DESCRIBE table` - returns the correct column types `GEOMETRY` for geo shapes and geo points. - `SHOW FUNCTIONS` - returns a list that includes supported `ST_` functions - `SYS TYPES` and `SYS COLUMNS` display correct types `GEO_SHAPE` and `GEO_POINT` for geo shapes and geo points accordingly. Returning geoshapes and geopoints from elasticsearch - `SELECT geom FROM table` - returns the geoshapes and geo_points as libs/geo objects in JDBC or as WKT strings in console. - `SELECT ST_AsWKT(geom) FROM table;` and `SELECT ST_AsText(geom) FROM table;`- returns the geoshapes ang geopoints in their WKT representation; Using geopoints to elasticsearch - The following functions will be supported for geopoints in queries, sorting and aggregations: `ST_GeomFromText`, `ST_X`, `ST_Y`, `ST_Z`, `ST_GeometryType`, and `ST_Distance`. In most cases when used in queries, sorting and aggregations, these function are translated into script. These functions can be used in the SELECT clause for both geopoints and geoshapes. - `SELECT * FROM table WHERE ST_Distance(ST_GeomFromText(POINT(1 2), point) < 10;` - returns all records for which `point` is located within 10m from the `POINT(1 2)`. In this case the WHERE clause is translated into a range query. Limitations: Geoshapes cannot be used in queries, sorting and aggregations as part of this initial effort. In order to fully take advantage of geoshapes we would need to have access to geoshape doc values, which is coming in #37206. `ST_Z` cannot be used on geopoints in queries, sorting and aggregations since we don't store altitude in geo_point doc values. Relates to #29872 Backport of #42031
1 parent 327f44e commit 70ea3cf

File tree

84 files changed

+3976
-70
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

84 files changed

+3976
-70
lines changed

‎docs/reference/sql/functions/geo.asciidoc

+192
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
[role="xpack"]
2+
[testenv="basic"]
3+
[[sql-functions-geo]]
4+
=== Geo Functions
5+
6+
The geo functions work with geometries stored in `geo_point` and `geo_shape` fields, or returned by other geo functions.
7+
8+
==== Limitations
9+
10+
Both <<geo-point, `geo_point`>> and <<geo-shape, `geo_shape`>> types are represented in SQL as geometry and can be used
11+
interchangeably with the following exceptions:
12+
13+
* `geo_shape` fields don't have doc values, therefore these fields cannot be used for filtering, grouping or sorting.
14+
15+
* `geo_points` fields are indexed and have doc values by default, however only latitude and longitude are stored and
16+
indexed with some loss of precision from the original values (4.190951585769653E-8 for the latitude and
17+
8.381903171539307E-8 for longitude). The altitude component is accepted but not stored in doc values nor indexed.
18+
Therefore calling `ST_Z` function in the filtering, grouping or sorting will return `null`.
19+
20+
==== Geometry Conversion
21+
22+
[[sql-functions-geo-st-as-wkt]]
23+
===== `ST_AsWKT`
24+
25+
.Synopsis:
26+
[source, sql]
27+
--------------------------------------------------
28+
ST_AsWKT(geometry<1>)
29+
--------------------------------------------------
30+
31+
*Input*:
32+
33+
<1> geometry
34+
35+
*Output*: string
36+
37+
.Description:
38+
39+
Returns the WKT representation of the `geometry`.
40+
41+
["source","sql",subs="attributes,macros"]
42+
--------------------------------------------------
43+
include-tagged::{sql-specs}/docs/geo.csv-spec[aswkt]
44+
--------------------------------------------------
45+
46+
47+
[[sql-functions-geo-st-wkt-to-sql]]
48+
===== `ST_WKTToSQL`
49+
50+
.Synopsis:
51+
[source, sql]
52+
--------------------------------------------------
53+
ST_WKTToSQL(string<1>)
54+
--------------------------------------------------
55+
56+
*Input*:
57+
58+
<1> string WKT representation of geometry
59+
60+
*Output*: geometry
61+
62+
.Description:
63+
64+
Returns the geometry from WKT representation.
65+
66+
["source","sql",subs="attributes,macros"]
67+
--------------------------------------------------
68+
include-tagged::{sql-specs}/docs/geo.csv-spec[aswkt]
69+
--------------------------------------------------
70+
71+
==== Geometry Properties
72+
73+
[[sql-functions-geo-st-geometrytype]]
74+
===== `ST_GeometryType`
75+
76+
.Synopsis:
77+
[source, sql]
78+
--------------------------------------------------
79+
ST_GeometryType(geometry<1>)
80+
--------------------------------------------------
81+
82+
*Input*:
83+
84+
<1> geometry
85+
86+
*Output*: string
87+
88+
.Description:
89+
90+
Returns the type of the `geometry` such as POINT, MULTIPOINT, LINESTRING, MULTILINESTRING, POLYGON, MULTIPOLYGON, GEOMETRYCOLLECTION, ENVELOPE or CIRCLE.
91+
92+
["source","sql",subs="attributes,macros"]
93+
--------------------------------------------------
94+
include-tagged::{sql-specs}/docs/geo.csv-spec[geometrytype]
95+
--------------------------------------------------
96+
97+
[[sql-functions-geo-st-x]]
98+
===== `ST_X`
99+
100+
.Synopsis:
101+
[source, sql]
102+
--------------------------------------------------
103+
ST_X(geometry<1>)
104+
--------------------------------------------------
105+
106+
*Input*:
107+
108+
<1> geometry
109+
110+
*Output*: double
111+
112+
.Description:
113+
114+
Returns the longitude of the first point in the geometry.
115+
116+
["source","sql",subs="attributes,macros"]
117+
--------------------------------------------------
118+
include-tagged::{sql-specs}/docs/geo.csv-spec[x]
119+
--------------------------------------------------
120+
121+
[[sql-functions-geo-st-y]]
122+
===== `ST_Y`
123+
124+
.Synopsis:
125+
[source, sql]
126+
--------------------------------------------------
127+
ST_Y(geometry<1>)
128+
--------------------------------------------------
129+
130+
*Input*:
131+
132+
<1> geometry
133+
134+
*Output*: double
135+
136+
.Description:
137+
138+
Returns the the latitude of the first point in the geometry.
139+
140+
["source","sql",subs="attributes,macros"]
141+
--------------------------------------------------
142+
include-tagged::{sql-specs}/docs/geo.csv-spec[y]
143+
--------------------------------------------------
144+
145+
[[sql-functions-geo-st-z]]
146+
===== `ST_Z`
147+
148+
.Synopsis:
149+
[source, sql]
150+
--------------------------------------------------
151+
ST_Z(geometry<1>)
152+
--------------------------------------------------
153+
154+
*Input*:
155+
156+
<1> geometry
157+
158+
*Output*: double
159+
160+
.Description:
161+
162+
Returns the altitude of the first point in the geometry.
163+
164+
["source","sql",subs="attributes,macros"]
165+
--------------------------------------------------
166+
include-tagged::{sql-specs}/docs/geo.csv-spec[z]
167+
--------------------------------------------------
168+
169+
[[sql-functions-geo-st-distance]]
170+
===== `ST_Distance`
171+
172+
.Synopsis:
173+
[source, sql]
174+
--------------------------------------------------
175+
ST_Distance(geometry<1>, geometry<2>)
176+
--------------------------------------------------
177+
178+
*Input*:
179+
180+
<1> source geometry
181+
<2> target geometry
182+
183+
*Output*: Double
184+
185+
.Description:
186+
187+
Returns the distance between geometries in meters. Both geometries have to be points.
188+
189+
["source","sql",subs="attributes,macros"]
190+
--------------------------------------------------
191+
include-tagged::{sql-specs}/docs/geo.csv-spec[distance]
192+
--------------------------------------------------

‎docs/reference/sql/functions/index.asciidoc

+9
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,14 @@
136136
** <<sql-functions-conditional-least>>
137137
** <<sql-functions-conditional-nullif>>
138138
** <<sql-functions-conditional-nvl>>
139+
* <<sql-functions-geo>>
140+
** <<sql-functions-geo-st-as-wkt>>
141+
** <<sql-functions-geo-st-distance>>
142+
** <<sql-functions-geo-st-geometrytype>>
143+
** <<sql-functions-geo-st-wkt-to-sql>>
144+
** <<sql-functions-geo-st-x>>
145+
** <<sql-functions-geo-st-y>>
146+
** <<sql-functions-geo-st-z>>
139147
* <<sql-functions-system>>
140148
** <<sql-functions-system-database>>
141149
** <<sql-functions-system-user>>
@@ -149,5 +157,6 @@ include::search.asciidoc[]
149157
include::math.asciidoc[]
150158
include::string.asciidoc[]
151159
include::type-conversion.asciidoc[]
160+
include::geo.asciidoc[]
152161
include::conditional.asciidoc[]
153162
include::system.asciidoc[]

‎docs/reference/sql/language/data-types.asciidoc

+2
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,8 @@ s|SQL precision
8181
| interval_hour_to_minute | 23
8282
| interval_hour_to_second | 23
8383
| interval_minute_to_second | 23
84+
| geo_point | 52
85+
| geo_shape | 2,147,483,647
8486

8587
|===
8688

‎docs/reference/sql/limitations.asciidoc

+11
Original file line numberDiff line numberDiff line change
@@ -150,3 +150,14 @@ SELECT count(*) FROM test GROUP BY MINUTE((CAST(date_created AS TIME));
150150
-------------------------------------------------------------
151151
SELECT HISTOGRAM(CAST(birth_date AS TIME), INTERVAL '10' MINUTES) as h, COUNT(*) FROM t GROUP BY h
152152
-------------------------------------------------------------
153+
154+
[float]
155+
[[geo-sql-limitations]]
156+
=== Geo-related functions
157+
158+
Since `geo_shape` fields don't have doc values these fields cannot be used for filtering, grouping or sorting.
159+
160+
By default,`geo_points` fields are indexed and have doc values. However only latitude and longitude are stored and
161+
indexed with some loss of precision from the original values (4.190951585769653E-8 for the latitude and
162+
8.381903171539307E-8 for longitude). The altitude component is accepted but not stored in doc values nor indexed.
163+
Therefore calling `ST_Z` function in the filtering, grouping or sorting will return `null`.

‎server/src/main/java/org/elasticsearch/common/geo/parsers/ShapeParser.java

+22
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,18 @@
2020

2121
import org.elasticsearch.ElasticsearchParseException;
2222
import org.elasticsearch.common.ParseField;
23+
import org.elasticsearch.common.bytes.BytesReference;
2324
import org.elasticsearch.common.geo.builders.ShapeBuilder;
25+
import org.elasticsearch.common.xcontent.LoggingDeprecationHandler;
26+
import org.elasticsearch.common.xcontent.NamedXContentRegistry;
2427
import org.elasticsearch.common.xcontent.XContent;
28+
import org.elasticsearch.common.xcontent.XContentBuilder;
2529
import org.elasticsearch.common.xcontent.XContentParser;
30+
import org.elasticsearch.common.xcontent.json.JsonXContent;
2631
import org.elasticsearch.index.mapper.BaseGeoShapeFieldMapper;
2732

2833
import java.io.IOException;
34+
import java.io.InputStream;
2935

3036
/**
3137
* first point of entry for a shape parser
@@ -67,4 +73,20 @@ static ShapeBuilder parse(XContentParser parser, BaseGeoShapeFieldMapper shapeMa
6773
static ShapeBuilder parse(XContentParser parser) throws IOException {
6874
return parse(parser, null);
6975
}
76+
77+
static ShapeBuilder parse(Object value) throws IOException {
78+
XContentBuilder content = JsonXContent.contentBuilder();
79+
content.startObject();
80+
content.field("value", value);
81+
content.endObject();
82+
83+
try (InputStream stream = BytesReference.bytes(content).streamInput();
84+
XContentParser parser = JsonXContent.jsonXContent.createParser(
85+
NamedXContentRegistry.EMPTY, LoggingDeprecationHandler.INSTANCE, stream)) {
86+
parser.nextToken(); // start object
87+
parser.nextToken(); // field name
88+
parser.nextToken(); // field value
89+
return parse(parser);
90+
}
91+
}
7092
}

‎x-pack/plugin/sql/build.gradle

+1
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ ext {
1616
// SQL test dependency versions
1717
csvjdbcVersion="1.0.34"
1818
h2Version="1.4.197"
19+
h2gisVersion="1.5.0"
1920
}
2021

2122
configurations {

‎x-pack/plugin/sql/jdbc/build.gradle

+3
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,9 @@ dependencies {
2121
compile (project(':libs:x-content')) {
2222
transitive = false
2323
}
24+
compile (project(':libs:elasticsearch-geo')) {
25+
transitive = false
26+
}
2427
compile project(':libs:core')
2528
runtime "com.fasterxml.jackson.core:jackson-core:${versions.jackson}"
2629
testCompile "org.elasticsearch.test:framework:${version}"

‎x-pack/plugin/sql/jdbc/src/main/java/org/elasticsearch/xpack/sql/jdbc/EsType.java

+3-1
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,9 @@ public enum EsType implements SQLType {
4444
INTERVAL_DAY_TO_SECOND( ExtraTypes.INTERVAL_DAY_SECOND),
4545
INTERVAL_HOUR_TO_MINUTE( ExtraTypes.INTERVAL_HOUR_MINUTE),
4646
INTERVAL_HOUR_TO_SECOND( ExtraTypes.INTERVAL_HOUR_SECOND),
47-
INTERVAL_MINUTE_TO_SECOND(ExtraTypes.INTERVAL_MINUTE_SECOND);
47+
INTERVAL_MINUTE_TO_SECOND(ExtraTypes.INTERVAL_MINUTE_SECOND),
48+
GEO_POINT( ExtraTypes.GEOMETRY),
49+
GEO_SHAPE( ExtraTypes.GEOMETRY);
4850

4951
private final Integer type;
5052

‎x-pack/plugin/sql/jdbc/src/main/java/org/elasticsearch/xpack/sql/jdbc/ExtraTypes.java

+1
Original file line numberDiff line numberDiff line change
@@ -29,5 +29,6 @@ private ExtraTypes() {}
2929
static final int INTERVAL_HOUR_MINUTE = 111;
3030
static final int INTERVAL_HOUR_SECOND = 112;
3131
static final int INTERVAL_MINUTE_SECOND = 113;
32+
static final int GEOMETRY = 114;
3233

3334
}

‎x-pack/plugin/sql/jdbc/src/main/java/org/elasticsearch/xpack/sql/jdbc/JdbcColumnInfo.java

+2-1
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
* or more contributor license agreements. Licensed under the Elastic License;
44
* you may not use this file except in compliance with the Elastic License.
55
*/
6+
67
package org.elasticsearch.xpack.sql.jdbc;
78

89
import java.util.Objects;
@@ -89,4 +90,4 @@ public boolean equals(Object obj) {
8990
public int hashCode() {
9091
return Objects.hash(name, type, table, catalog, schema, label, displaySize);
9192
}
92-
}
93+
}

‎x-pack/plugin/sql/jdbc/src/main/java/org/elasticsearch/xpack/sql/jdbc/JdbcConfiguration.java

+2-2
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@
3535
/ Additional properties can be specified either through the Properties object or in the URL. In case of duplicates, the URL wins.
3636
*/
3737
//TODO: beef this up for Security/SSL
38-
class JdbcConfiguration extends ConnectionConfiguration {
38+
public class JdbcConfiguration extends ConnectionConfiguration {
3939
static final String URL_PREFIX = "jdbc:es://";
4040
public static URI DEFAULT_URI = URI.create("http://localhost:9200/");
4141

@@ -47,7 +47,7 @@ class JdbcConfiguration extends ConnectionConfiguration {
4747
// can be out/err/url
4848
static final String DEBUG_OUTPUT_DEFAULT = "err";
4949

50-
static final String TIME_ZONE = "timezone";
50+
public static final String TIME_ZONE = "timezone";
5151
// follow the JDBC spec and use the JVM default...
5252
// to avoid inconsistency, the default is picked up once at startup and reused across connections
5353
// to cater to the principle of least surprise

0 commit comments

Comments
 (0)