chore(host_scanner): improve cleanup algorithm in scans

[Molter73]

Description

In order to remove items from the inode maps for files that may have been deleted and may have missed events, or paths that are no longer monitored due to a configuration change, we used to:

• Iterate over the userspace map.
• Check if the path matches the configured globs.
• Check if the path still exists in the filesystem (via statx).
• If either of the previous checks failed, we remove the inode from the maps

After this cleanup is done, we proceed to glob the configured paths and check all of them on disk, adding any missing inodes and updating any changed paths. The most common situation for these scans is that most inodes should hit (unless the configuration is changed or massive changes to the fs were missed), which means we are essentially doing two statx calls per element in the inode (once during the cleanup iteration and another during the glob expansion).

This change does a few things:

• Add a marker to each tracked inode (a counter that increments every scan iteration).
• Do the glob expansion first, marking each found inode with the current scan number.
• Once we are done exploring the glob expansion, iterate over the map and remove any entries that have an out of date scan marker (i.e: they were not found as part of the glob expansion).

This saves us some statx calls during scans, making them faster.

Of note, this change works because scanning pauses processing of events until the scan is done, otherwise it could be subject to race conditions.

This is a small follow-up to #1221.

Checklist

• Patch has a change log entry OR does not need one.
• Investigated and inspected CI test results
• Updated documentation accordingly

Automated testing

• Added unit tests
• Added integration tests
• Added regression tests

If any of these don't apply, please comment below.

Testing Performed

Reduced number of syscalls

Running fact with the following command: FACT_LOGLEVEL=info RUST_BACKTRACE=1 cargo srun --bin fact --all-features -- -p '/etc/**/*:/etc/' --inodes-max=2097152 --expose-metrics --scan-interval 30, then using perf stat -e 'syscalls:sys_enter_statx,syscalls:sys_enter_bpf' -p "$(pgrep fact)" -- sleep 30 capturing a single scan. No events generated during the run, no modifications to the /etc directory, 25966 inodes tracked for all the scans.

Before changes:

 Performance counter stats for process id '1067511':

            56,597      syscalls:sys_enter_statx
                 0      syscalls:sys_enter_bpf

      30.002870566 seconds time elapsed

After changes:

 Performance counter stats for process id '1954123':

            30,631      syscalls:sys_enter_statx
                 0      syscalls:sys_enter_bpf

      30.004685963 seconds time elapsed

Average scan time

Average scan time before changes: 114.19ms

# HELP stackrox_fact_host_scanner_scan_duration Histogram of scan durations from the host scanner component.
# TYPE stackrox_fact_host_scanner_scan_duration histogram
stackrox_fact_host_scanner_scan_duration_sum 13.245613742999997
stackrox_fact_host_scanner_scan_duration_count 116
stackrox_fact_host_scanner_scan_duration_bucket{le="0.01"} 0
stackrox_fact_host_scanner_scan_duration_bucket{le="0.05"} 0
stackrox_fact_host_scanner_scan_duration_bucket{le="0.1"} 0
stackrox_fact_host_scanner_scan_duration_bucket{le="0.25"} 116
stackrox_fact_host_scanner_scan_duration_bucket{le="0.5"} 116
stackrox_fact_host_scanner_scan_duration_bucket{le="1.0"} 116
stackrox_fact_host_scanner_scan_duration_bucket{le="5.0"} 116
stackrox_fact_host_scanner_scan_duration_bucket{le="10.0"} 116
stackrox_fact_host_scanner_scan_duration_bucket{le="30.0"} 116
stackrox_fact_host_scanner_scan_duration_bucket{le="60.0"} 116
stackrox_fact_host_scanner_scan_duration_bucket{le="120.0"} 116
stackrox_fact_host_scanner_scan_duration_bucket{le="+Inf"} 116

Average scan time after changes: 79.35ms

# HELP stackrox_fact_host_scanner_scan_duration Histogram of scan durations from the host scanner component.
# TYPE stackrox_fact_host_scanner_scan_duration histogram
stackrox_fact_host_scanner_scan_duration_sum 8.8867888
stackrox_fact_host_scanner_scan_duration_count 112
stackrox_fact_host_scanner_scan_duration_bucket{le="0.01"} 0
stackrox_fact_host_scanner_scan_duration_bucket{le="0.05"} 0
stackrox_fact_host_scanner_scan_duration_bucket{le="0.1"} 109
stackrox_fact_host_scanner_scan_duration_bucket{le="0.25"} 112
stackrox_fact_host_scanner_scan_duration_bucket{le="0.5"} 112
stackrox_fact_host_scanner_scan_duration_bucket{le="1.0"} 112
stackrox_fact_host_scanner_scan_duration_bucket{le="5.0"} 112
stackrox_fact_host_scanner_scan_duration_bucket{le="10.0"} 112
stackrox_fact_host_scanner_scan_duration_bucket{le="30.0"} 112
stackrox_fact_host_scanner_scan_duration_bucket{le="60.0"} 112
stackrox_fact_host_scanner_scan_duration_bucket{le="120.0"} 112
stackrox_fact_host_scanner_scan_duration_bucket{le="+Inf"} 112

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 32.80%. Comparing base (0a1d04e) to head (d340324).

Files with missing lines	Patch %	Lines
fact/src/host_scanner.rs	0.00%	39 Missing ⚠️

Additional details and impacted files

@@                                  Coverage Diff                                  @@
##           mauro/fix/miscellaneous-host-scanner-improvements    #1226      +/-   ##
=====================================================================================
- Coverage                                              32.94%   32.80%   -0.15%     
=====================================================================================
  Files                                                     21       21              
  Lines                                                   2996     3009      +13     
  Branches                                                2996     3009      +13     
=====================================================================================
  Hits                                                     987      987              
- Misses                                                  2006     2019      +13     
  Partials                                                   3        3              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.