🔍 The Strange Write Load
While investigating unexpectedly high SSD wear on a Proxmox cluster, I found something surprising: several otherwise quiet Debian VMs were writing tens of gigabytes per day.
The pattern was suspiciously regular.
VMs checked every two minutes were writing around 65–67 GiB per day. Systems checked every five minutes were closer to 28 GiB per day.
That suggested the write load was related to the monitoring interval.
🧩 The Culprit: mk_apt
The Checkmk agent plugin mk_apt was installed directly in:
/usr/lib/check_mk_agent/plugins/mk_aptIn that location, the plugin runs every time the Checkmk agent is queried.
mk_apt does more than simply read a file. It invokes APT and performs an upgrade simulation in order to determine which normal and security updates are available.
Running that every few minutes turned out to be surprisingly expensive in terms of disk I/O.
The correlation was almost perfect:
* 2-minute monitoring interval -> about 65 GiB/day
* 5-minute monitoring interval -> about 28 GiB/day
🛠️ The Fix
Checkmk supports asynchronous agent plugins simply by placing them in a directory named after their execution interval.
For example:
mkdir-p /usr/lib/check_mk_agent/plugins/3600
mv /usr/lib/check_mk_agent/plugins/mk_apt /usr/lib/check_mk_agent/plugins/3600/Now mk_apt runs only once per hour, while Checkmk uses the cached result between executions.
An hourly check is also perfectly reasonable for package-update monitoring.
📉 The Result
The difference was dramatic.
Before the change, the VMs together were writing roughly:
~439 GiB/day
After moving mk_apt to asynchronous execution, the write load settled around:
~65 GiB/day
That is a reduction of roughly 85%.
On individual quiet VMs, the effect was even more obvious. Systems that had previously written around 28 GiB/day dropped to roughly 4–5 GiB/day.
No monitoring functionality was lost. Update information is still available, just without asking APT the same question hundreds of times per day.
⚠️ One Small Cache Trap
When deleting the asynchronous plugin cache manually, Checkmk may temporarily report:
Missing monitoring data for plugins, apt
That simply means the old cache is gone while the asynchronous plugin is generating a new result.
Normally it is better to let Checkmk manage the cache lifecycle or invalidate only the specific mk_apt cache when necessary.
🔧 upgrade vs. dist-upgrade
The plugin also contains an UPGRADE setting, for example:
UPGRADE=upgrade
For normal Debian systems this is usually sufficient.
On Proxmox hosts, using:
UPGRADE=dist-upgrade
can make more sense because Proxmox updates may involve dependency changes and package replacements.
In an Ansible-managed environment, this can easily be adjusted based on inventory groups.
💡 The Lesson
Monitoring software is usually considered read-only infrastructure.
That assumption can be wrong.
A small plugin executed every few minutes can indirectly trigger substantial filesystem activity, especially when it invokes package managers, databases, container tooling, or other stateful utilities.
The important lesson was not that mk_apt is bad. It was simply being executed far more often than necessary.
Sometimes SSD wear is not caused by databases, logging, or applications.
Sometimes it is caused by the software that is watching them. 😄