Skip to content

Move legacy geo_shape implementation to its own module #77856

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Oct 6, 2021

Conversation

iverase
Copy link
Contributor

@iverase iverase commented Sep 16, 2021

This PR moves the legacy geo_shape implementation to its own module and removes the dependency on Spatial4J / JTS from server. In addition moves the dependency with lucine-spatial-extras away from server.

Unfortunately the diff is pretty big due to run spotless in the new module, but the PR is mostly moving files around.

@iverase iverase added :Analytics/Geo Indexing, search aggregations of geo points and shapes :Core/Infra/Plugins Plugin API and infrastructure v8.0.0 labels Sep 16, 2021
@elasticmachine elasticmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Core/Infra Meta label for core/infra team labels Sep 16, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

Copy link
Member

@rjernst rjernst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only looked at the build.gradle files, assuming everything else has not effectively changed.

@@ -11,6 +11,7 @@ esplugin {
}

dependencies {
api project(path: ':modules:legacy-geo')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to be clear, this will cause a copy of the legacy-geo jar (and it's dependencies) to be added to the xpack spatial module. It won't reuse the same. I think that is ok, as there is no communication between them, I believe this is just for reuse of some common stuff, but I want to double check with you of that understanding.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this because when creating a geo_shape field mapper we might need to create the mapper on the legacy jar:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this because when creating a geo_shape field mapper we might need to create the mapper on the legacy jar:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that the only use? If so, consider moving the shared code into a project under lib. By depending on the module the dependencies, including jts, are pulled into (as a copy) the spatial module.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is tricky, the mapper and shape builders companions are tightly dependent on Elasticsearch server and other projects under lib.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I don’t think that is an issue. The mapper is checking the existing field type against itself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is no interaction between the modules except for the spatial module creating Legacy mappers under certain circumstances. Still I am thinking if we can do better without being too invasive.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does spatial create the legacy mappers, through normal mapper apis (ie json) or directly in code? If it is in code, then I don't think that would work.

The "solution" is to do something we recommend against in general, which is declaring the spatial module "extends" legacy geo. Even though it doesn't actually extend it, it will make the classloader of legacy geo a parent of spatial, so the classes would be available. To do this you would add the new module to the extendedPlugins list in build.gradle, and change the dependencies section to use compileOnly instead of api for the legacy module.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I implemented the "solution" above, therefore now spatial module extends legacy module and dependency
is compile only now. It was very insightful and I got a better understanding how it works. This action has some side effects:

  • Vector tile extends the spatial module but now in order to compile I need to add a compile only dependency to JTS
  • Couple of testing modules were adding spatial as a runtime dependency. I have to add legacy-geo as well to make it happy.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rjernst Do you think this is good enough or we should have more iterations?

# Conflicts:
#	modules/legacy-geo/licenses/lucene-spatial-extras-8.10.0-snapshot-bf2fcb53079.jar.sha1
# Conflicts:
#	modules/legacy-geo/licenses/lucene-spatial-extras-9.0.0-snapshot-32a0a16aff0.jar.sha1
#	x-pack/plugin/sql/qa/server/build.gradle
Copy link
Member

@rjernst rjernst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks ok for now, we can sort it out further later.

@iverase
Copy link
Contributor Author

iverase commented Oct 5, 2021

@elasticmachine update branch

@iverase
Copy link
Contributor Author

iverase commented Oct 6, 2021

@elasticmachine update branch

@iverase iverase added the v7.16.0 label Oct 6, 2021
@iverase iverase merged commit b2e4639 into elastic:master Oct 6, 2021
@iverase iverase deleted the legacyGeoModule branch October 6, 2021 06:13
iverase added a commit to iverase/elasticsearch that referenced this pull request Oct 6, 2021
This commit moves the legacy geo_shape implementation to its own module and removes
the dependency on Spatial4J / JTS from server.
iverase added a commit that referenced this pull request Oct 11, 2021
This commit moves the legacy geo_shape implementation to its own module and removes
the dependency on Spatial4J / JTS from server.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/Geo Indexing, search aggregations of geo points and shapes :Core/Infra/Plugins Plugin API and infrastructure >non-issue Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Core/Infra Meta label for core/infra team v7.16.0 v8.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants