-
Notifications
You must be signed in to change notification settings - Fork 307
Update template to use ES 5.x mapping #462
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
fc43eb2
Update template to use ES 5.x mapping
suyograo 3dfad93
Duplicate spec for 5.x for new mapping use
suyograo 9b22019
turn off 5x for old template tests
suyograo d1d9208
rename spec and tags to match versions
suyograo f912a8c
Rename spec and use proper version tags
suyograo 053473d
Push all changes files
suyograo 2646919
use correct tags
suyograo 4dfa67e
Use rspec tag trickery to run tests for all ver
suyograo 9ab35eb
Update docs, bump version to 5.0
suyograo b0df891
Use ES alpha5 in tests
suyograo 670503c
Change name of rspec tag to greater_than_equal_to
suyograo File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| require_relative "../../../spec/es_spec_helper" | ||
|
|
||
| # This file is a copy of template test for 2.x. We can DRY this up later. | ||
| describe "index template expected behavior for 5.x", :integration => true, :version_greater_than_equal_to_5x => true do | ||
| subject! do | ||
| require "logstash/outputs/elasticsearch" | ||
| settings = { | ||
| "manage_template" => true, | ||
| "template_overwrite" => true, | ||
| "hosts" => "#{get_host_port()}" | ||
| } | ||
| next LogStash::Outputs::ElasticSearch.new(settings) | ||
| end | ||
|
|
||
| before :each do | ||
| # Delete all templates first. | ||
| require "elasticsearch" | ||
|
|
||
| # Clean ES of data before we start. | ||
| @es = get_client | ||
| @es.indices.delete_template(:name => "*") | ||
|
|
||
| # This can fail if there are no indexes, ignore failure. | ||
| @es.indices.delete(:index => "*") rescue nil | ||
|
|
||
| subject.register | ||
|
|
||
| subject.multi_receive([ | ||
| LogStash::Event.new("message" => "sample message here"), | ||
| LogStash::Event.new("somevalue" => 100), | ||
| LogStash::Event.new("somevalue" => 10), | ||
| LogStash::Event.new("somevalue" => 1), | ||
| LogStash::Event.new("country" => "us"), | ||
| LogStash::Event.new("country" => "at"), | ||
| LogStash::Event.new("geoip" => { "location" => [ 0.0, 0.0 ] }) | ||
| ]) | ||
|
|
||
| @es.indices.refresh | ||
|
|
||
| # Wait or fail until everything's indexed. | ||
| Stud::try(20.times) do | ||
| r = @es.search | ||
| insist { r["hits"]["total"] } == 7 | ||
| end | ||
| end | ||
|
|
||
| it "permits phrase searching on string fields" do | ||
| results = @es.search(:q => "message:\"sample message\"") | ||
| insist { results["hits"]["total"] } == 1 | ||
| insist { results["hits"]["hits"][0]["_source"]["message"] } == "sample message here" | ||
| end | ||
|
|
||
| it "numbers dynamically map to a numeric type and permit range queries" do | ||
| results = @es.search(:q => "somevalue:[5 TO 105]") | ||
| insist { results["hits"]["total"] } == 2 | ||
|
|
||
| values = results["hits"]["hits"].collect { |r| r["_source"]["somevalue"] } | ||
| insist { values }.include?(10) | ||
| insist { values }.include?(100) | ||
| reject { values }.include?(1) | ||
| end | ||
|
|
||
| it "does not create .keyword field for the message field" do | ||
| results = @es.search(:q => "message.keyword:\"sample message here\"") | ||
| insist { results["hits"]["total"] } == 0 | ||
| end | ||
|
|
||
| it "creates .keyword field from any string field which is not_analyzed" do | ||
| results = @es.search(:q => "country.keyword:\"us\"") | ||
| insist { results["hits"]["total"] } == 1 | ||
| insist { results["hits"]["hits"][0]["_source"]["country"] } == "us" | ||
|
|
||
| # partial or terms should not work. | ||
| results = @es.search(:q => "country.keyword:\"u\"") | ||
| insist { results["hits"]["total"] } == 0 | ||
| end | ||
|
|
||
| it "make [geoip][location] a geo_point" do | ||
| expect(@es.indices.get_template(name: "logstash")["logstash"]["mappings"]["_default_"]["properties"]["geoip"]["properties"]["location"]["type"]).to eq("geo_point") | ||
| end | ||
|
|
||
| it "aggregate .keyword results correctly " do | ||
| results = @es.search(:body => { "aggregations" => { "my_agg" => { "terms" => { "field" => "country.keyword" } } } })["aggregations"]["my_agg"] | ||
| terms = results["buckets"].collect { |b| b["key"] } | ||
|
|
||
| insist { terms }.include?("us") | ||
|
|
||
| # 'at' is a stopword, make sure stopwords are not ignored. | ||
| insist { terms }.include?("at") | ||
| end | ||
| end | ||
|
|
||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought the idea was to let ES run through its defaults in the case that a stringed field is being dynamically mapped
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, but we need
"norms" : false,to be set, which ES does not by default. That's why I had to explicitly add this rule for strings.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See 2 lines above.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah, I see. sounds good