New tail_mode and stat_poll_interval settings to support not keeping file handles permanently open

[david-garcia-garcia]

Fixes #4074

This PR adds two new configuration options:

• tail_mode: default or stat
• stat_poll_interval: how frequently should files be checked for changes when tail_mode=stat

To avoid introducing regresions, I built an abstraction to wrap the actual tail implementation depending on tail_mode. When using tail_mode the previous implementation used. When using tail_mode=stat, a new tail implementation that does not permamently keep the file handle open is used.

[github-actions]

@david-garcia-garcia: There are no 'kind' label on this PR. You need a 'kind' label to generate the release automatically.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind fix
• /kind chore
• /kind dependencies

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@david-garcia-garcia: There are no area labels on this PR. You can add as many areas as you see fit.

• /area agent
• /area local-api
• /area cscli
• /area appsec
• /area security
• /area configuration

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[codecov]

Codecov Report

❌ Patch coverage is 83.23353% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.19%. Comparing base (3e7eba2) to head (b493de1).
⚠️ Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
.../acquisition/modules/file/tailwrapper/stat_tail.go	82.69%	13 Missing and 5 partials ⚠️
...acquisition/modules/file/tailwrapper/nxadm_tail.go	82.85%	4 Missing and 2 partials ⚠️
pkg/acquisition/modules/file/run.go	88.23%	2 Missing ⚠️
...kg/acquisition/modules/file/tailwrapper/factory.go	81.81%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4075      +/-   ##
==========================================
+ Coverage   63.10%   63.19%   +0.09%     
==========================================
  Files         472      475       +3     
  Lines       33458    33610     +152     
==========================================
+ Hits        21113    21240     +127     
- Misses      10224    10243      +19     
- Partials     2121     2127       +6     

Flag	Coverage Δ
bats	46.46% <30.53%> (-0.13%)	⬇️
unit-linux	36.28% <82.63%> (+0.22%)	⬆️
unit-windows	24.56% <59.88%> (+0.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[david-garcia-garcia]

/kind enhancement
/area agent

[mmetc]

Hi @david-garcia-garcia,

Thanks for the PR!

We're close to releasing 1.7.4, so this won't be merged before 1.7.5.

As you can see from the last few commits in the master branch, we are
in the middle of a global refactor of the acquisition:
the current non-blocking StreamingAcquisition() method is being replaced
by a synchronous Stream() method. This lets us drop tombs in favor of
context-based cancellation (more errgroup, waitgroup and fewer channels),
and bubble all retryable errors up to a centralized backoff loop.

For reference, the journalctl datasource is already migrated and merged; syslog
is migrated but not merged yet. Docker and file are likely to be next, shortly
after the 1.7.4 release.

Since your PR is a wrapper around the tail package there shouldn't be a lot of conflicts
if we postpone the review and adapt it to the new interface and concurrency model once they
are in place.

Thanks again for your work and your patience!

[david-garcia-garcia]

@mmetc I see that file acquisition has not been upgraded yet. I did a quick attempt here #4280 but the underlying library used for tail does not support context. I am thiking it could be more straightforward to replace that tail library with a custom made one that optionally supports the behaviour that this PR attempts to support.

[mmetc]

A cross platform tail rewrite would be tricky. I think it's easier to wrap tomb behavior around context. What do you think @blotus

[david-garcia-garcia]

I took a look at the library used and the implementation was not that big, plus it had several features that we are not using so it could be further simplified.

Attempted to fully replace it here:

#4280