Remove unneeded weak reference from prefix logger #22460
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jasontedor
added
:Core/Infra/Logging
|
For the reviewer, we have a test for the prefix logger functionality in |
jasontedor
force-pushed
the
simplify-prefix-logger
branch
from
6201032 to
9140a49
Compare
We have a custom logger implementation known as a prefix logger that is used to write every message by the logger with a given prefix. This is useful for node-level, index-level, and shard-level messages where we want to log the node name, index name, and shard ID, respectively, if possible. The mechanism that we employ is that of a marker. Log4j has a built-in facility for managing these markers, but its effectively a memory leak because these markers are held in a map and can never be released. This is problematic for us since indices and shards do not necessarily have infinite life spans and so on a node where there are many indices being creted and destroyed, this infinite lifespan can be a problem indeed. To solve this, we use our own cache of markers. This is necessary to prevent too many instances of the marker for the same prefix from being created (just think of all the shard-level components that exist in the system), and to workaround the effective leak in Log4j. These markers are stored as weak references in a weak hash map. It is these weak references that are unneeded. When a key is removed from a weak hash map, the corresponding entry is placed on a reference queue that is eventually cleared. This commit simplifies prefix logger by removing this unnecessary weak reference wrapper.
jasontedor
force-pushed
the
simplify-prefix-logger
branch
from
9140a49 to
c6cc6a4
Compare
rjernst
approved these changes
Jan 6, 2017
| public void testPrefixLoggerMarkersCanBeCollected() throws IOException, UserException { | ||
| setupLogging("prefix"); | ||
|
|
||
| for (int i = 0; i < 1 << 20; i++) { |
There was a problem hiding this comment.
Maybe move the limit out to a local, with a comment on the reason for the size (I presume to ensure System.gc() actually collects something below).
|
@jasontedor Can this be merged? |
* master: (1889 commits) Test: remove faling test that relies on merge order Log checkout so SHA is known Add link to community Rust Client (elastic#22897) "shard started" should show index and shard ID (elastic#25157) await fix testWithRandomException Change BWC versions on create index response Return the index name on a create index response Remove incorrect bwc branch logic from master Correctly format arrays in output [Test] Extending parsing checks for SearchResponse (elastic#25148) Scripting: Change keys for inline/stored scripts to source/id (elastic#25127) [Test] Add test for custom requests in High Level Rest Client (elastic#25106) nested: In case of a single type the _id field should be added to the nested document instead of _uid field. `type` and `id` are lost upon serialization of `Translog.Delete`. (elastic#24586) fix highlighting docs Fix NPE in token_count datatype with null value (elastic#25046) Remove the postings highlighter and make unified the default highlighter choice (elastic#25028) [Test] Adding test for parsing SearchShardFailure leniently (elastic#25144) Fix typo in shards.asciidoc (elastic#25143) List Hibernate Search (elastic#25145) ...
jasontedor
added a commit
that referenced
this pull request
Jun 10, 2017
We have a custom logger implementation known as a prefix logger that is used to write every message by the logger with a given prefix. This is useful for node-level, index-level, and shard-level messages where we want to log the node name, index name, and shard ID, respectively, if possible. The mechanism that we employ is that of a marker. Log4j has a built-in facility for managing these markers, but its effectively a memory leak because these markers are held in a map and can never be released. This is problematic for us since indices and shards do not necessarily have infinite life spans and so on a node where there are many indices being creted and destroyed, this infinite lifespan can be a problem indeed. To solve this, we use our own cache of markers. This is necessary to prevent too many instances of the marker for the same prefix from being created (just think of all the shard-level components that exist in the system), and to workaround the effective leak in Log4j. These markers are stored as weak references in a weak hash map. It is these weak references that are unneeded. When a key is removed from a weak hash map, the corresponding entry is placed on a reference queue that is eventually cleared. This commit simplifies prefix logger by removing this unnecessary weak reference wrapper. Relates #22460
jasontedor
added a commit
that referenced
this pull request
Jun 10, 2017
We have a custom logger implementation known as a prefix logger that is used to write every message by the logger with a given prefix. This is useful for node-level, index-level, and shard-level messages where we want to log the node name, index name, and shard ID, respectively, if possible. The mechanism that we employ is that of a marker. Log4j has a built-in facility for managing these markers, but its effectively a memory leak because these markers are held in a map and can never be released. This is problematic for us since indices and shards do not necessarily have infinite life spans and so on a node where there are many indices being creted and destroyed, this infinite lifespan can be a problem indeed. To solve this, we use our own cache of markers. This is necessary to prevent too many instances of the marker for the same prefix from being created (just think of all the shard-level components that exist in the system), and to workaround the effective leak in Log4j. These markers are stored as weak references in a weak hash map. It is these weak references that are unneeded. When a key is removed from a weak hash map, the corresponding entry is placed on a reference queue that is eventually cleared. This commit simplifies prefix logger by removing this unnecessary weak reference wrapper. Relates #22460
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have a custom logger implementation known as a prefix logger that is used to write every message by the logger with a given prefix. This is useful for node-level, index-level, and shard-level messages where we want to log the node name, index name, and shard ID, respectively, if possible. The mechanism that we employ is that of a marker. Log4j has a built-in facility for managing these markers, but its effectively a memory leak because these markers are held in a map and can never be released. This is problematic for us since indices and shards do not necessarily have infinite life spans and so on a node where there are many indices being creted and destroyed, this infinite lifespan can be a problem indeed. To solve this, we use our own cache of markers. This is necessary to prevent too many instances of the marker for the same prefix from being created (just think of all the shard-level components that exist in the system), and to workaround the effective leak in Log4j. These markers are stored as weak references in a weak hash map. It is these weak references that are unneeded. When a key is removed from a weak hash map, the corresponding entry is placed on a reference queue that is eventually cleared. This commit simplifies prefix logger by removing this unnecessary weak reference wrapper.
Relates #20429