When Monitoring Quietly Wears Out Your SSD – A Checkmk mk_apt Story (Updated 21.09.2026 & 02.10.2026)

by · ⏱ 4 minutes

🔍 The Strange Write Load

While investigating unexpectedly high SSD wear on a Proxmox cluster, I found something surprising: several otherwise quiet Debian VMs were writing tens of gigabytes per day.

The pattern was suspiciously regular.

VMs checked every two minutes were writing around 65–67 GiB per day. Systems checked every five minutes were closer to 28 GiB per day.

That suggested the write load was related to the monitoring interval.

🧩 The Culprit: mk_apt

The Checkmk agent plugin mk_apt was installed directly in:

Bash
/usr/lib/check_mk_agent/plugins/mk_apt

In that location, the plugin runs every time the Checkmk agent is queried.

mk_apt does more than simply read a file. It invokes APT and performs an upgrade simulation in order to determine which normal and security updates are available.

Running that every few minutes turned out to be surprisingly expensive in terms of disk I/O.

The correlation was almost perfect:

* 2-minute monitoring interval  -> about 65 GiB/day
* 5-minute monitoring interval  -> about 28 GiB/day

🛠️ The Fix

Checkmk supports asynchronous agent plugins simply by placing them in a directory named after their execution interval.

For example:

Bash
mkdir-p /usr/lib/check_mk_agent/plugins/3600
mv /usr/lib/check_mk_agent/plugins/mk_apt /usr/lib/check_mk_agent/plugins/3600/

Now mk_apt runs only once per hour, while Checkmk uses the cached result between executions.

An hourly check is also perfectly reasonable for package-update monitoring.

📉 The Result

The difference was dramatic.

Before the change, the VMs together were writing roughly:

~439 GiB/day

After moving mk_apt to asynchronous execution, the write load settled around:

~65 GiB/day

That is a reduction of roughly 85%.

On individual quiet VMs, the effect was even more obvious. Systems that had previously written around 28 GiB/day dropped to roughly 4–5 GiB/day.

No monitoring functionality was lost. Update information is still available, just without asking APT the same question hundreds of times per day.

⚠️ One Small Cache Trap

When deleting the asynchronous plugin cache manually, Checkmk may temporarily report:

Missing monitoring data for plugins, apt

That simply means the old cache is gone while the asynchronous plugin is generating a new result.

Normally it is better to let Checkmk manage the cache lifecycle or invalidate only the specific mk_apt cache when necessary.

🔧 upgrade vs. dist-upgrade

The plugin also contains an UPGRADE setting, for example:

UPGRADE=upgrade

For normal Debian systems this is usually sufficient.

On Proxmox hosts, using:

UPGRADE=dist-upgrade

can make more sense because Proxmox updates may involve dependency changes and package replacements.

In an Ansible-managed environment, this can easily be adjusted based on inventory groups.

💡 The Lesson

Monitoring software is usually considered read-only infrastructure.

That assumption can be wrong.

A small plugin executed every few minutes can indirectly trigger substantial filesystem activity, especially when it invokes package managers, databases, container tooling, or other stateful utilities.

The important lesson was not that mk_apt is bad. It was simply being executed far more often than necessary.

Sometimes SSD wear is not caused by databases, logging, or applications.

Sometimes it is caused by the software that is watching them. 😄

🔄 Update (21.09.2026): Replacing mk_apt with mk-job

After moving mk_apt to a six-hour interval, I eventually removed it completely.

The remaining write load was already small. The more important reason was architectural consistency.

🧭 Monitor the Real Job

Debian already runs the real package maintenance through:

apt-daily.service
apt-daily-upgrade.service

So instead of running an additional APT simulation just for monitoring, I decided to monitor the actual jobs with Checkmk’s mk-job.

The resulting flow is:

systemd timer
      |
      v
apt-daily.service
      |
      v
mk-job
      |
      v
/usr/lib/apt/apt.systemd.daily update

and:

systemd timer
      |
      v
apt-daily-upgrade.service
      |
      v
mk-job
      |
      v
/usr/lib/apt/apt.systemd.daily install

⚙️ Wrap the systemd Services

For apt-daily.service:

Bash
systemctl edit apt-daily.service
Bash
[Service]
ExecStart=
ExecStart=/usr/bin/mk-job job-unattended-update /usr/lib/apt/apt.systemd.daily update

For apt-daily-upgrade.service:

Bash
systemctl edit apt-daily-upgrade.service
Bash
[Service]
ExecStart=
ExecStart=/usr/bin/mk-job job-unattended-upgrade /usr/lib/apt/apt.systemd.daily install

Then reload systemd:

Bash
systemctl daemon-reload

🗑️ Remove mk_apt

Once this was working, mk_apt was no longer needed:

Bash
rm -rf /usr/lib/check_mk_agent/plugins/21600

Only remove the whole directory if it contains no other plugins.

The final setup is now:

mk_apt                     removed
apt-daily                  monitored by mk-job
apt-daily-upgrade   monitored by mk-job

📉 Write Load

The quiet VM write baseline dropped a little further, from roughly 50 GiB/day to around 40–42 GiB/day.

The bigger improvement, however, is conceptual.

🧠 Simpler Architecture

Before:

actual package maintenance
+
separate APT simulation for monitoring

After:

actual package maintenance
+
monitoring of that exact operation

The monitoring now answers the more useful question:

Did automatic package maintenance actually run successfully?

That is simpler, avoids duplicated APT work, and reduces unnecessary SSD writes.

🔄 Update (02.10.2026): Cutting APT Write Load Even Further

After replacing mk_apt with mk-job, one question remained:

Why was an otherwise idle VM still writing about 2 GiB per day?

🔍 Finding the Source

mk-job already exposed the answer.

A normal hourly unattended-update run showed:

Filesystem writes: 204376

which corresponds to roughly 100 MiB of filesystem writes per run.

At the same time, APT maintained two binary cache files:

/var/cache/apt/pkgcache.bin       ~47 MiB
/var/cache/apt/srcpkgcache.bin    ~47 MiB

Together, they matched the observed write volume almost perfectly.

With an hourly update check, that means roughly:

~100 MiB × 24 ≈ 2.3 GiB/day

per VM.

⚙️ Disable the Persistent APT Caches

These binary caches are not required for detecting updates. They mainly avoid reparsing package metadata.

They can be disabled with:

Dir::Cache::pkgcache "";
Dir::Cache::srcpkgcache "";

For example in:

/etc/apt/apt.conf.d/99no-pkgcache

The actual package lists under:

/var/lib/apt/lists/

remain unchanged, so apt update and unattended-upgrades continue to work normally.

My systems intentionally check for new package lists every hour:

APT::Periodic::Update-Package-Lists "always";

That behaviour remains unchanged.

📉 Result

After disabling the two caches, the next real unattended-update run reported:

Filesystem writes: 11960

instead of:

Filesystem writes: 204376

That reduced the write volume from roughly 100 MiB to about 6 MiB per run — around 94% less.

The effect was immediately visible on the otherwise idle test VM:

Before: ~2.0 GiB/day
After:  ~0.12 GiB/day

🧠 Same Update Policy, Less Write Churn

Nothing about the update policy changed.

The VM still checks for new packages every hour and still installs unattended updates normally.

Only the persistent binary APT caches are gone.

For systems that check for updates frequently, this turned out to be another surprisingly large source of avoidable SSD writes.