File exclude behavior changed in undocumented ways

[sysadmin1139]

After doing an upgrade from logstash 2.4 to logstash 6.7.2 (this was a long time coming) we observed a difference in behavior of how the exclude feature behaves.

input {
  file {
    path => '/var/lib/example/log/*.log'
    exclude => [
      'unfortunate-secrets.log',
      'uselessly-noisy.log'
    ]
    type => 'app_example'
  }
}

This fragment worked to exclude the unfortunate-secrets file in Logstash 2.4, and many earlier versions as well. Not anymore. The new behavior has both of the excluded files included in the pipeline. As a workaround we added a filter {} stage to drop {} anything with path == '/var/lib/example/log/unfortunate-secrets.log' which works.

The documentation surrounding the path attribute hasn't changed. The documentation for exclude includes a mention of Tail mode but is otherwise unchanged between the versions.

This is running on Ubuntu 16.04 with Java8.

[ddaanet]

Hi buddy. I have the same issue, and it's been driving me nuts the whole day.

Documentation says: Exclusions (matched against the filename, not full path). Filename patterns are valid here, too.

But the current (7.5.1) code (in filewatch/discoverer.rb/FileWatch.Discoverer.can_exclude?)matches against the full pathname. I am a complete newbie here, so I have no idea what are the hoops to get a bugfix merged, but I'll open a PR anyway.

[jsvd]

fixed in https://rubygems.org/gems/logstash-input-file/versions/4.1.12