Switch default scripting language to painless

[rjernst]

Painless has improved considerably in the past 6 months, both in performance and feature parity with groovy. The performance is as much as 4x better than groovy for the same script. There have also been numerous changes in #17992 to allow existing groovy scripts to work out of the box.

Groovy is on its way out. We should change the default scripting language to painless for 5.0, so that we can at least move groovy to a plugin for 5.0 or 6.0. Users would still be able to change back, just as they could with mvel when we made the switch to groovy.

I've marked this issue as breaking, but I expect the vast majority of groovy scripts to just work in painless.

[dakrone]

+1 to make painless the default for 5.0

[nik9000]

I agree. My instinct is that we need a lot of documenting and examples to make the painless transition truly painless. But we'll be much more keen on writing it if it is the default.

[jdconrad]

+1. Guess we will be seeing some incoming bug reports soon :)

[rmuir]

I agree. My instinct is that we need a lot of documenting and examples to make the painless transition truly painless. But we'll be much more keen on writing it if it is the default.

Yes, this needs to be improved, but lets be fair: that problem is unrelated to painless. scripting ES with any other languages was never easy before: the scripting api is seriously not the most intuitive thing in the world, you don't get stacktraces when things go wrong, you are trying to code a small program in one line of json, and so on.

When @jdconrad and I sat down and evaluated a selection of real scripts in use, we found many are just coming directly from our examples (or maybe copied from stackoverflow), maybe with a few tweaks. Lots of repetition across different users. So I think its very important the examples be good, performant ones. For instance, if we create examples that do slow things then people are going to do just that and then rely on it, and it makes it harder to clean things up.

[jdconrad]

100% agree with @rmuir -- even if we support certain features, I would say document them, but leave them out of examples if they aren't performant. I would even support not providing examples for aggregations. These are advanced, and the users that really need them will be able to do them without examples.

[nik9000]

that problem is unrelated to painless

We absolutely had a problem before. Inventing a whole new language compounds the problem. People used to be able to go to the docs for groovy to figure out how to do a thing and it worked sometimes. There are only two asciidoc pages about painless.

The silver lining here is that we can control the docs so, like you say, we only add things that perform well. Going to the groovy docs for Elasticsearch scripting was as likely as not going to give you examples of stuff that we didn't support in Elasticsearch.

We had a problem with docs for scripting. We made it worse by inventing a new language. We can make it a lot better by building good, performant, tested examples.

[rmuir]

We absolutely had a problem before. Inventing a whole new language compounds the problem. People used to be able to go to the docs for groovy to figure out how to do a thing and it worked sometimes. There are only two asciidoc pages about painless.

Eh, i dont think thats really fair to say. Personally I spent a lot of time completely reorganizing the scripting docs as a whole to try to make this stuff easier to use.

There is only one page on groovy, and a whole complicated second page on security considerations. That page is not something to bother with if you use painless/expressions. More documentation is not better.

I don't think the quality of documentation is determined by volume. Quality not quantity. I would rather us start leading the path of deprecating all these scripting-languages-we-cannot-really-support, and improve (but a key part of that is simplify) scripting docs as part of the process.

Sometimes less is more.

[jdconrad]

@nik9000 Do you have anything specifically in mind that you would like to see in the documentation?

I find Elasticsearch's documentation on general scripting to be very limited. I have a lot of trouble following the aggregations and getting those to work specifically. However, I do believe Painless documentation, which of course can be improved, is better than the other languages. Groovy's documentation may seem well ordered at first outside of Elasticsearch, but it's really all over the place for finding useful information about what the language is doing.

[nik9000]

Do you have anything specifically in mind that you would like to see in the documentation?

That is the thing, I'm really mostly going on instinct and memory of working with mvel and groovy for a while. I did a fair amount of scripting with them I remember looking up stuff like "what is the way to do Math.max in groovy?" "how do I do log1p in mvel?"

Its just been too long since I did that sort of thing for me to have proper empathy for it. I'm not sure where to start or I'd start on it.

[clintongormley]

@jdconrad honestly i find i don't know where to start in painless. i don't know what the syntax should look like, and I don't know what is supported. there may be examples in other places in the docs that i could use for inspiration, but when I look at the painless docs, there's not much more there than a for loop. It also links to various Java classes but I'm not a Java person and I'm not sure how to use those either.

I'd love to use more painless, but I get stuck. To echo nik's comment: other languages have reference and getting started guides but the equivalent doesn't yet exist for Painless. I recognise it's a mammoth task to write these, but it'd make adoption a whole lot easier. Maybe it's just the getting started, maybe it's sufficiently similar to other languages that I'll get the idea quickly enough once I've written a few scripts, so maybe an exhaustive reference isn't required.

[rjernst]

i don't know what the syntax should look like

We say this in the docs: The Painless syntax is similar to Groovy. So start as if you are writing groovy? Part of the reason we chose to start with groovy as a base was explicitly so we would not need to write large amounts of documentation, or diverge to far from a known syntax. It is almost a strict subset of groovy (except it adds limited lambda support which are not in groovy, instead of exposing closures).

It also links to various Java classes but I'm not a Java person and I'm not sure how to use those either

How is this different than what we have for groovy? In fact, it is better, because we actually link to what can be used, instead of not documenting anything?

Should we move discussion about how to improve painless docs to a separate issue? Are the documentation comments concerns that we should not change the default because of documentation?