Dynamic (inline) groovy scripts that parse JSON do not work with 2.0

[plebedev]

I'm trying to migrate from 1.7 to 2.0 and I can't get my existing dynamic groovy script to work with ES.
First, I added this line to elasticsearch.yml:
script.inline: on

With just this line I'm getting an error when I try to do bulk update with the following script for each update action:
final Script updateScript = new Script("event = new groovy.json.JsonSlurper().parseText(_event);ctx._source.events += event", ScriptType.INLINE, "groovy", ImmutableMap.of("_event", eventJson));

I'm getting the error when I execute the action:
_GroovyScriptExecutionException[failed to run inline script [event = new groovy.json.JsonSlurper().parseText(_event);ctx.source.events += event] using lang [groovy]]; nested: NotSerializableExceptionWrapper[Could not initialize class groovy.json.internal.JsonParserCharArray];

When I add these lines to elasticsearch.yml:
script.engine.groovy.inline.aggs: on
script.engine.groovy.inline.mapping: on
script.engine.groovy.inline.search: on
script.engine.groovy.inline.update: on
script.engine.groovy.inline.plugin: on

and try to execute an update with the same script, I get this error:
_GroovyScriptExecutionException[failed to run inline script [event = new groovy.json.JsonSlurper().parseText(_event);ctx.source.events += event] using lang [groovy]]; nested: NotSerializableExceptionWrapper; nested: SecurityException[access denied ("java.util.PropertyPermission" "groovy.json.internKeys" "read")];

I tried to add
permission java.util.PropertyPermission "groovy.json.internKeys", "read";
to java.policy file but it did not help.

I see permission java.util.PropertyPermission "*", "read,write";
in org/elasticsearch/bootstrap/security.policy, so it is not clear to me why I'm getting this permission error unless groovy runs with its own security manager that does not

The same script with 1.7 worked.

I have few other scripts for score functions that do not use JSON, and these work fine.

Thanks,
Peter.

[dakrone]

Relates to #14488, @rmuir we added the policy to allow JSON encoding from Groovy, do you think it's safe to add the policy to allow JSON decoding as well?

[rmuir]

If it is just a property, it does not hurt. Of course, i strongly feel scripts should not be doing any of this.

[plebedev]

As a workaround, I found groovy.json.JsonSlurperClassic class that can be used instead of groovy.json.JsonSlurper without causing any of the errors.

[fs-chris]

I've the very same issue with Elastic version 2.2.0.
Yet the JsonSlurperClassic workaround no longer works because the class is no longer available.

My test script is as simple as that:

import groovy.json.JsonSlurper;
def jsonSlurper = new JsonSlurper(); 
def message = jsonSlurper.parseText(_json);

When first trying to run the script, I get the SecurityException[access denied ("java.util.PropertyPermission" "groovy.json.internKeys" "read") .
Every successive attempt to run the script then gives NoClassDefFoundError[Could not initialize class groovy.json.internal.JsonParserCharArray];

I've tried with the following policy but without success:

grant {
    permission java.util.PropertyPermission "groovy.json.internKeys", "read";
    permission org.elasticsearch.script.ClassPermission "java.lang.Class";
    permission org.elasticsearch.script.ClassPermission "groovy.json.JsonSlurper";
    permission org.elasticsearch.script.ClassPermission "groovy.json.internal.*";
    permission org.elasticsearch.script.ClassPermission "groovy.json.internal.JsonParserCharArray";
    permission org.elasticsearch.script.ClassPermission "org.elasticsearch.common.logging.*";
};

Can anyone please provide an example how to use the JsonSlurper in a Groovy script with Elastic 2.2.0?
Thanks,
Chris

[bleskes]

@plebedev correct me if I'm wrong, but it seems like you use a script to update a document and the parameter for the script is JSON encoded as string. If that's the case, you can use a map of maps (i.e. JSON object ) as the value of the parameter which means you don't need to encode the JSON or alternatively you can decode it on your client. Does that make sense?

[fs-chris]

@bleskes thanks for your suggestion.
However, in my case this is not a feasible solution as it would require me to parse the JSON in the client, which is indeed not desirable for my application.

[bleskes]

@fs-chris thanks

which is indeed not desirable for my application

Can you elaborate on this - most notably I'm interested in understanding how parsing in your application is different than parsing in the ES process.

[fs-chris]

@bleskes thanks for your interest.

I'm using Groovy in an update script.
The script decides whether to process the JSON based on the existing document being updated.
The JSON needs to be parsed only in a very small fraction of all such update requests. Most cases update the document with another argument passed to the script at the same time.

So, always parsing in the application would be unecessary overhead in almost all cases.
First fetching the existing document from the index and then making the decision in the client would be even more overhead.
We need to process several thousands of such requests per second. So performance is essential.

The approach with doing it in the script worked well with previous Elastic releases.

[bleskes]

@fs-chris thanks. I see the discussion on #16808 , most notably @clintongormley 's suggestion:

We need to process several thousands of such requests per second. So performance is essential.

... which indicates that this should be implemented in Java as well.

[plebedev]

@bleskes, We've migrated to 2.2.1 and we had to change the code to pass JSONObject, so it is not parsed in elastinsearch. Works so far :)

Thanks!

[bleskes]

@plebedev cool. Thanks for circling back and letting us know.