Problem statement
A small Linux sampler should report useful metrics without an entire monitoring stack, a root requirement, or misleading zeros when a source disappears.
Architecture
Bounded procfs/sysfs reads feed typed parsers, timestamped snapshots, and identity-aware delta calculations. Separate formatters emit JSON Lines or Prometheus text. The sampler retains only the previous snapshot and does not depend on the separate thread-pool project.
What works
- CPU tick accounting, MemTotal/MemAvailable, interface counters, and disk I/O.
- Explicitly selected process state, RSS, CPU counters, and start-time identity.
- First-sample unavailable rates, reset detection, and visible collector errors.
- Configurable interval/count, synthetic fixture sequences, and structured output.
- A 1 MiB per-source read bound, device limits, and at most 64 selected PIDs.
- Separate exit statuses for collector failure and configuration/runtime errors.
Testing
Synthetic procfs/sysfs fixtures exercise parsing, exact rates, counter resets, PID identity, malformed inputs, missing sources, and source-size limits. CLI tests parse the emitted JSON and check error handling and Prometheus snapshots. Fixture reports are labeled explicitly, separate from live collection.
Measurement
The benchmark workflow records fixture reads and JSON serialization, process CPU time, peak working set, and output size. Raw results include their environment and scope; fixture processing measurements are not claims about production exporter overhead.
Collection model
I keep the agent focused on read-only collection and stdout output. Prometheus text snapshots integrate with existing monitoring tools. Namespaces, permissions, source-read timing, and counter identity are part of the metric definitions.
Device names are limited to printable ASCII. Process arguments/environment are not collected. Blocks and partitions must not be blindly summed as independent physical-device traffic.
Source and usage
Browse linux-observability-agent on GitHub for the CLI, metric definitions, fixture examples, and build instructions.
The packet diagnosis article explains why collector observations and conclusions about bottlenecks are different things.