Skip to content

Fix memory leak in TransportDeleteExpiredDataAction #89935

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

original-brownbear
Copy link
Contributor

We can loose the listener here because the downstream code in deleteExpiredData might throw. This will cause us to leak the listener and as a result leak memory for the unanswered transport request. Fixed by wrapping the runnable to catch uncaught exceptions.

This shows up in real world cloud logging as:

[instance-0000000008] uncaught exception in thread [elasticsearch[instance-0000000008][ml_utility][T#5]]
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:729) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:419) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:761) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:513) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:327) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:263) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:470) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:218) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1031) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:747) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:390) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:478) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:277) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:103) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
	at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:575) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:100) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:573) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:123) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:573) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:86) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:573) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:96) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:573) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:26) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:573) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:573) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:569) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:84) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:26) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:569) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:554) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$10(AuthorizationService.java:448) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:945) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:909) ~[?:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:352) ~[?:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:41) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:993) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:343) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:441) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:378) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$2(AuthorizationService.java:263) ~[?:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:139) ~[?:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:185) ~[?:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:193) ~[?:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:175) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:136) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:265) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:94) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$1(SecurityActionFilter.java:110) ~[?:?]
	at org.elasticsearch.xpack.core.security.SecurityContext.executeAsInternalUser(SecurityContext.java:121) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationUtils.switchUserBasedOnActionOriginAndExecute(AuthorizationUtils.java:142) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:106) ~[?:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:165) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:113) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:91) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.client.internal.FilterClient.doExecute(FilterClient.java:57) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.client.internal.OriginSettingClient.doExecute(OriginSettingClient.java:43) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:366) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.client.internal.support.AbstractClient.search(AbstractClient.java:510) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.xpack.ml.utils.persistence.SearchAfterDocumentsIterator.executeSearchRequest(SearchAfterDocumentsIterator.java:146) ~[?:?]
	at org.elasticsearch.xpack.ml.utils.persistence.SearchAfterDocumentsIterator.doSearch(SearchAfterDocumentsIterator.java:138) ~[?:?]
	at org.elasticsearch.xpack.ml.utils.persistence.SearchAfterDocumentsIterator.next(SearchAfterDocumentsIterator.java:110) ~[?:?]
	at org.elasticsearch.xpack.ml.utils.persistence.WrappedBatchedJobsIterator.next(WrappedBatchedJobsIterator.java:51) ~[?:?]
	at org.elasticsearch.xpack.ml.utils.persistence.WrappedBatchedJobsIterator.next(WrappedBatchedJobsIterator.java:25) ~[?:?]
	at org.elasticsearch.xpack.ml.job.retention.AbstractExpiredJobDataRemover.removeData(AbstractExpiredJobDataRemover.java:55) ~[?:?]
	at org.elasticsearch.xpack.ml.job.retention.AbstractExpiredJobDataRemover.remove(AbstractExpiredJobDataRemover.java:42) ~[?:?]
	at org.elasticsearch.xpack.ml.job.retention.ExpiredResultsRemover.remove(ExpiredResultsRemover.java:67) ~[?:?]
	at org.elasticsearch.xpack.ml.action.TransportDeleteExpiredDataAction.deleteExpiredData(TransportDeleteExpiredDataAction.java:201) ~[?:?]
	at org.elasticsearch.xpack.ml.action.TransportDeleteExpiredDataAction.deleteExpiredData(TransportDeleteExpiredDataAction.java:172) ~[?:?]
	at org.elasticsearch.xpack.ml.action.TransportDeleteExpiredDataAction.lambda$doExecute$1(TransportDeleteExpiredDataAction.java:140) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) ~[elasticsearch-8.3.3.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.elasticsearch.action.NoShardAvailableActionException
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:548) ~[elasticsearch-8.3.3.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:495) ~[elasticsearch-8.3.3.jar:?]
	... 104 more

We can loose the listener here beacuse the downstream code in `deleteExpiredData`
might throw.
@elasticsearchmachine elasticsearchmachine added the Team:ML Meta label for the ML team label Sep 8, 2022
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

@elasticsearchmachine
Copy link
Collaborator

Hi @original-brownbear, I've created a changelog YAML for you.

@droberts195 droberts195 added v8.4.2 and removed v8.4.1 labels Sep 8, 2022
Copy link
Contributor

@droberts195 droberts195 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks for fixing it

@original-brownbear
Copy link
Contributor Author

np + thanks for reviewing David!

@elasticsearchmachine
Copy link
Collaborator

💚 Backport successful

Status Branch Result
8.4

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this pull request Sep 8, 2022
We can loose the listener here because the downstream code in `deleteExpiredData`
might throw.
elasticsearchmachine pushed a commit that referenced this pull request Sep 8, 2022
We can loose the listener here because the downstream code in `deleteExpiredData`
might throw.
weizijun added a commit to weizijun/elasticsearch that referenced this pull request Sep 9, 2022
* main: (34 commits)
  Make sure ivy repo directory exists before downloading artifacts
  Use 'file://' scheme for local repository URL
  Use DRA artifacts for release build CI jobs
  Log unsuccessful attempts to get credentials from web identity tokens (elastic#88241)
  Script: Write Field API path manipulation (elastic#89889)
  Fetch health info action (elastic#89820)
  Fix memory leak in TransportDeleteExpiredDataAction (elastic#89935)
  [ML] Performance improvements for categorization jobs (elastic#89824)
  [DOCS] Revert changes for ES_JAVA_OPTS (elastic#89931)
  Fix deadlock bug exposed by a test (elastic#89934)
  [Downsampling] Remove `FieldValueFetcher` validator (elastic#89497)
  Fix segment stats in tsdb (elastic#89754)
  Synthetic _source: support dense_vector (elastic#89840)
  REST tests fetching fields with synthetic _source (elastic#89888)
  Do not deserialize back BytesTransportRequest to clone a request in MockTransportService (elastic#89926)
  Add SDK request logging to debug failures of S3BlobStoreRepositoryTests#testRequestStats (elastic#89912)
  Fix SnapshotStatusApisIT.testGetSnapshotsWithSnapshotInProgress (elastic#89925)
  Document synthetic source for text and keyword (elastic#89893)
  Fix CloneSnapshotIT.testRemoveFailedCloneFromCSWithQueuedSnapshotInProgress (elastic#89914)
  Add missing index.mapping.total_fields.limit setting to the target index (elastic#89875)
  ...
weizijun added a commit to weizijun/elasticsearch that referenced this pull request Sep 9, 2022
* main: (176 commits)
  Fix RandomSamplerAggregatorTests testAggregationSamplingNestedAggsScaled test failure (elastic#89958)
  [Downsampling] Replace document map with SMILE encoded doc (elastic#89495)
  Remove full cluster state from error logging in MasterService (elastic#89960)
  [ML] Truncate categorization fields (elastic#89827)
  [TSDB] Removed `summary` and `histogram` metric types (elastic#89937)
  Update testNodeSelectorRouting so that it does not depend on iteration order (elastic#89879)
  Make sure listener is resolved when file queue is cleared (elastic#89929)
  [Stable plugin api] Extensible annotation (elastic#89903)
  Fix double sending of response in TransportOpenIdConnectPrepareAuthenticationAction (elastic#89930)
  Make sure ivy repo directory exists before downloading artifacts
  Use 'file://' scheme for local repository URL
  Use DRA artifacts for release build CI jobs
  Log unsuccessful attempts to get credentials from web identity tokens (elastic#88241)
  Script: Write Field API path manipulation (elastic#89889)
  Fetch health info action (elastic#89820)
  Fix memory leak in TransportDeleteExpiredDataAction (elastic#89935)
  [ML] Performance improvements for categorization jobs (elastic#89824)
  [DOCS] Revert changes for ES_JAVA_OPTS (elastic#89931)
  Fix deadlock bug exposed by a test (elastic#89934)
  [Downsampling] Remove `FieldValueFetcher` validator (elastic#89497)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :ml Machine learning Team:ML Meta label for the ML team v8.4.2 v8.5.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants