Provide an option to disable SecurityManager and an option to cache directory.

[xiezhenye]

We meets performance problem in refresh. We found many lstat syscall while refresh operation, and dentry cache is exhausted.

The problem is caused mainly by two issue.
1 While SecurityManager check permission of a file, it calls java.io.FileSystem.canonicalize and it will trace from root dir to the end of path and lstat every node.

2 In org.apache.lucene.store.FSDirectory construct, it calls java.nio.Path.toRealPath and it will also do the same thing.

Caching FSDirectory object and disable SecurityManager will save many unnecessary lstat syscall and improve performance.

Providing an option to disable SecurityManager will also make it easy to trace the code by such as btrace.

[jasontedor]

Providing an option to disable SecurityManager will also make it easy to trace the code by such as btrace.

To level-set expectations: this will never happen. The ability to disable security means there is no security.

[xiezhenye]

In some scene, SecurityManager is not necessary. E.g. using OS level security functions, not using third-part plugin.

[jpountz]

I agree with @jasontedor that disabling the security manager is not an option. Furthermore, I suspect this is only the tip of the iceberg: lstat being a bottleneck makes me suspect that you have one of the following problems:

• too many shards per node
• too low refresh interval
• slow hardware/filesystem

[xiezhenye]

1 Is about 200 shards per node too many? I don't think so. The bottleneck is not CPU nor IO.
2 Our refresh interval is 5s, and it's the highest delay we can bear.
3 We are using SSD, and the performance problem is not caused by IO but the syscall cost and dentry usage.

[jpountz]

How many of these shards are actively indexing, ie. receiving one index request or more every 5 seconds?

[s1monw]

how does this manifest, what is the slowdown you are seeing? for things like
org.apache.lucene.store.FSDirectory where is uses toRealPath it happens in the ctor so I am not sure why this should slow you down? again what slowdowns are you seeing?

[xiezhenye]

Many unnecessary lstat syscall take more time to do refreshing and uses gigas bytes of dentry cache and causes system cache fill the memory. When making new memory allocation request, OS has to clear cache and makes a big latency. Fresh operation will block index operation. So this problem will also affect index operation.
Increasing vfs_cache_pressure will ease the pressure, but also has other side effects.

[s1monw]

@xiezhenye do you have any numbers? I mean did you try running without a security manager ie. by building the code yourself?

[xiezhenye]

That will take some days. I'll try it.

[jasontedor]

No additional feedback, closing.

[entrop-tankos]

It seems to me that we face the same/similar issue

Please take a look at the SVG inside zip.

elasticsearch cpu.zip