Skip to content

sincedb_write_interval not respected when in read mode #260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
akwoprosper opened this issue Feb 27, 2020 · 0 comments · Fixed by #276
Closed

sincedb_write_interval not respected when in read mode #260

akwoprosper opened this issue Feb 27, 2020 · 0 comments · Fixed by #276

Comments

@akwoprosper
Copy link

I just noticed that when using the read mode, the sincedb file is not updated.
Apparently, it is updated only when the file has completely been read.
This is problematic when dealing with large files since logstash may crash or be restarted before the file is completely written which will lead to rereading the same file from beginning.
@kares kares mentioned this issue Sep 10, 2020
Merged
kares added a commit that referenced this issue Sep 25, 2020
@kares
Fix: sincedb_clean_after not being respected (#276)
20db56d 
In certain cases, in read mode sincedb is not updated.

There was no periodic check for sincedb updates that would cause the sincedb_clean_after entries to cleanup.
The cleanup relied on new files being discovered or new content being added to existing files -> causing sincedb updates.

The fix here is to periodically flush sincedb (from the watch loop).
Besides, to make the process more deterministic, there's a minor change to make sure the same "updated" timestamp is used to mark the last changed time.

resolves #250
expected to also resolve #260
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix: sincedb_clean_after not being respected kares/logstash-input-file
1 participant
@akwoprosper