Checkmk 2.5 Community: Why 2 GB RAM Was Almost Enough

When upgrading from Checkmk 2.3 Raw to Checkmk 2.5 Community, we immediately noticed one thing: memory usage had increased considerably.

Our monitoring VM had only 2 GB of RAM, and initially we wanted to keep it that way.

The installation monitors around 20 hosts and more than 1,000 services. The web interface is normally used by only one administrator, with at most two or three concurrent users.

After quite a bit of testing, the conclusion turned out to be more nuanced than expected:

Checkmk 2.5 Community can run on 2 GB RAM, but whether that is a sensible production configuration depends heavily on how much swapping you are willing to accept.

With 2.5 GB RAM, the situation becomes much more comfortable.

This article documents the measurements, the optimizations we applied, and the three configurations we tested.

🔄 Upgrade Path: 2.3 → 2.4 → 2.5

Checkmk major versions cannot be skipped during an upgrade.

A Checkmk 2.3 site cannot be upgraded directly to 2.5. We therefore:

installed Checkmk 2.4,
upgraded the site from 2.3 to 2.4,
installed Checkmk 2.5,
upgraded the site from 2.4 to 2.5.

Before starting, we shut down the monitoring VM and created a cold snapshot.

After the initial upgrade to Checkmk 2.5, memory usage was already very close to the limit:

Memory: 1.9 GiB total, 1.8 GiB used, 38 MiB available

That was not enough safety margin for a production monitoring system.

🔍 Finding the Memory Consumers

Looking at proportional set size (PSS) quickly showed where most of the memory was going.

Typical values were approximately:

cmk-ui-job-scheduler      ~260–280 MB PSS
automation-helper          ~250–260 MB PSS
Apache workers               ~220–245 MB PSS each

PSS is more useful than RSS for this purpose because shared memory is divided proportionally between the processes using it.

The Apache workers were particularly expensive. With several large workers, Apache could easily consume a substantial part of the VM’s 2 GB RAM.

For a Checkmk installation with very few concurrent web users, that was the obvious place to look for savings.

✂️ Disabling the Automation Helper

The Automation Helper is used by the Checkmk web interface for certain configuration-related operations.

It improves responsiveness and parallelism, but in our installation it consumed roughly 250 MB of additional memory.

Because our site is small and the web interface is used by very few administrators, we disabled it:

Bash

omd stop <checkmk-site>
omd config <checkmk-site> set AUTOMATION_HELPER off
omd start <checkmk-site>

omd stop <checkmk-site>
omd config <checkmk-site> set AUTOMATION_HELPER off
omd start <checkmk-site>

Service discovery, configuration changes and activating changes continued to work normally in our environment.

For a larger installation or a site with many concurrent administrators, disabling the Automation Helper may not be appropriate.

🌐 Reducing Apache Concurrency

Apache was the next major memory consumer.

Checkmk normally configures five Apache worker processes. The configuration allows considerably more concurrency, up to 64 processes, but for a small monitoring site this is far more capacity than we need.

Each fully initialized Checkmk Apache worker consumed approximately 220–245 MB PSS in our tests. The number of simultaneously active workers therefore has a very direct impact on memory consumption.

We did not immediately reduce the limit to two.

First, we configured three workers and tested the web interface extensively. This already reduced memory usage considerably and worked without functional problems.

However, our actual usage pattern is very small:

normally one administrator uses the web interface,
occasionally two people may use it concurrently,
three concurrent active users are already unusual.

After testing three workers, we therefore decided to reduce the limit once more to two.

Two workers proved sufficient for our site, including intensive web UI testing. If both workers are busy at the same time, an additional web request may simply have to wait briefly. The monitoring core itself is independent of this web concurrency limit.

Reducing the limit from three workers to two saved approximately another 230–250 MB of physical memory in our installation.

This is an installation-specific optimization. Sites with more concurrent users should use a higher value.

To configure two workers, first stop the site:

Bash

omd stop <checkmk-site>

omd stop <checkmk-site>

Switch to the site user:

Bash

su - <checkmk-site>

su - <checkmk-site>

Create or edit:

Plaintext

/omd/sites/<checkmk-site>/etc/check_mk/apache.d/zz_local.mk

/omd/sites/<checkmk-site>/etc/check_mk/apache.d/zz_local.mk

with:

Plaintext

apache_process_tuning = {'number_of_processes': 2}

apache_process_tuning = {'number_of_processes': 2}

Leave the site user shell and start Checkmk again:

Bash

exit
omd start <checkmk-site>

exit
omd start <checkmk-site>

Checkmk regenerates the Apache configuration during startup.

Verify:

Bash

cat /omd/sites/<checkmk-site>/etc/apache/conf.d/zzz_check_mk.conf

cat /omd/sites/<checkmk-site>/etc/apache/conf.d/zzz_check_mk.conf

The generated configuration should contain:

# Created by WATO

ServerLimit 2
MaxClients 2

Do not edit zzz_check_mk.conf directly. It is generated by Checkmk.

🧹 Disabling Unused Checkmk Components

Checkmk supports several different ways of communicating with monitored systems and receiving events.

Not every installation needs all of them.

We disabled two components that are unused in our environment.

Agent Receiver

The Agent Receiver is used when monitored systems connect using the Checkmk Agent Controller and cmk-agent-ctl.

Our Linux agents are not connected this way.

Instead, Checkmk retrieves the agent output via SSH.

The Agent Receiver therefore has nothing to do in our setup:

Bash

omd stop <checkmk-site>
omd config <checkmk-site> set AGENT_RECEIVER off
omd start <checkmk-site>

omd stop <checkmk-site>
omd config <checkmk-site> set AGENT_RECEIVER off
omd start <checkmk-site>

Do not disable this component if your installation uses the Checkmk Agent Controller.

Event Console

The Checkmk Event Console processes event-oriented input such as SNMP traps and Syslog messages.

This is separate from normal SNMP polling.

We use SNMP polling for devices such as printers, but do not send SNMP traps or Syslog messages to the Checkmk Event Console.

We therefore disabled it:

Bash

omd stop <checkmk-site>
omd config <checkmk-site> set MKEVENTD off
omd start <checkmk-site>

omd stop <checkmk-site>
omd config <checkmk-site> set MKEVENTD off
omd start <checkmk-site>

Normal SNMP polling continues to work.

Do not disable the Event Console if you use SNMP traps, Syslog input or Event Console rules.

After disabling these components, an existing Checkmk service discovery may still contain services for them. Those services should be removed or disabled intentionally.

💾 The Real Question: What Should `vm.swappiness` Be?

This turned out to be the decisive part of the experiment.

The VM initially had a 4 GB conventional swap volume.

We tested two different swappiness settings with 2 GB RAM.

🧪 Variant 1: 2 GB RAM, `vm.swappiness=0`

At first this looked attractive.

With:

vm.swappiness = 0

Linux strongly preferred reclaiming file-backed cache instead of swapping anonymous process memory.

During moderate use, swap remained completely empty.

Under intensive web UI activity, however, memory pressure became extreme.

In one test the kernel invoked the OOM killer and terminated first a systemd user manager and later the Checkmk UI job scheduler:

Out of memory: Killed process ... (cmk-ui-job-sche)

Checkmk then reported:

ui-job-scheduler: stopped
Overall state: partially running

After additional memory optimizations we repeated the test.

This time no process was killed, but during intensive GUI activity memory reached approximately:

Memory available: ~34 MiB
Swap:                     0 B

The web application temporarily became unreachable and scheduled checks stopped progressing normally.

Changing back to:

vm.swappiness = 1

allowed the kernel to reclaim memory through swap and the system recovered.

Conclusion for this variant

2 GB with vm.swappiness=0 is not a configuration we consider suitable for production.

It may work during normal monitoring operation, but an intensive web UI workload can consume nearly all available memory without giving the kernel enough flexibility to move cold anonymous pages to swap.

swappiness=0 does not disable swap completely, but it makes the kernel much more reluctant to use it. In our workload, that was too late to provide a useful safety margin.

🧪 Variant 2: 2 GB RAM, `vm.swappiness=1`

With:

vm.swappiness = 1

the system remained functional even during heavy web UI activity.

The kernel began swapping cold anonymous pages instead of allowing available memory to collapse completely.

That prevented the failures seen with swappiness=0.

The downside was noticeable swap usage.

Depending on the test, we observed several hundred megabytes of swap being used.

During particularly memory-intensive periods, vmstat showed real swap-out and occasional swap-in activity.

This configuration therefore works.

If some swap usage is acceptable, Checkmk 2.5 Community can be operated on 2 GB RAM under our conditions.

However, this was not what we wanted for our monitoring VM.

We wanted swap to remain primarily a safety mechanism rather than something the system depends on during normal GUI usage.

Conclusion for this variant

2 GB with vm.swappiness=1 is technically viable.

It is considerably safer than swappiness=0, but the price is that a noticeable amount of process memory may be moved to swap during heavy GUI usage.

Whether that is acceptable depends on the environment.

🧪 Variant 3: 2.5 GB RAM, `vm.swappiness=1`

Finally, we increased the maximum VM memory from 2 GB to 2.5 GB.

The Checkmk configuration itself remained unchanged.

The same intensive web UI testing that had caused severe memory pressure with 2 GB now completed without problems.

A representative result directly after heavy GUI activity was:

Memory:         2.3 GiB usable
Used:              ~1.5 GiB
Available:        ~850 MiB
Swap:              ~1 MiB

Later during normal operation we saw:

Memory available: ~1.1 GiB
Swap:                     ~80 MiB

and after several days:

Memory available:   ~1.2 GiB
Swap:                       ~235 MiB

At first glance this might look as if the VM were again running short of memory.

It was not.

The important value here is MemAvailable, not the amount of swap already occupied.

Even with more than 200 MB stored in swap, the VM still had roughly 1.2 GB of immediately available memory.

Linux had simply identified very cold anonymous pages and moved them to swap over time. There is no requirement for the kernel to read those pages back merely because RAM later becomes available again.

Looking at the swapped pages per process confirmed that there was no single runaway consumer. Swap was distributed across long-running processes such as:

cmk-ui-job-scheduler
Apache
Redis
nginx
SSH processes
Nagios
system services

The relevant distinction is therefore between swap occupancy and swap activity.

A few hundred megabytes of cold pages sitting in swap are not in themselves a performance problem.

Continuous swapping in and out under load would be.

For our installation, swappiness=1 therefore remains the preferred setting. It gives the kernel a safety valve if memory pressure occurs, especially when the VM is temporarily ballooned down by the hypervisor.

We deliberately did not return to swappiness=0 merely to keep the swap counter at zero.

An empty swap device is not more valuable than a stable system.

🛟 Reducing Swap from 4 GB to 1 GB

The VM originally had 4 GB of swap.

After observing the real behaviour of the system, this turned out to be far more than necessary.

We therefore reduced the swap volume to:

Swap: 1 GiB

This does not change when Linux decides to swap. That is controlled by memory pressure and vm.swappiness.

It merely limits how much memory can ultimately be pushed out.

For this VM, 1 GB provides ample emergency capacity while avoiding a situation where a badly memory-starved monitoring system could continue limping along with several gigabytes of swapped-out memory.

Our final configuration is therefore:

RAM maximum:        2.5 GiB
RAM minimum:        ~1.75 GiB
Swap:                        1 GiB
vm.swappiness:         1
Apache workers:       2
Ballooning:               enabled
KSM:                         enabled

🎈 Proxmox Ballooning

Because the Proxmox hosts themselves have limited RAM, we do not reserve the full 2.5 GB unconditionally.

The VM uses Proxmox ballooning with approximately:

Minimum memory:    ~1.75 GiB
Maximum memory:   2.5 GiB
Ballooning:                enabled
KSM:                          enabled

During normal operation the VM receives the full 2.5 GB.

The lower minimum exists for an unusual HA scenario in which this Proxmox host temporarily has to run additional VMs from another cluster node during a reboot or failover.

All intensive Checkmk GUI tests described here were performed while the VM had the full 2.5 GB available.

During such an HA situation, additional swap usage or reduced Checkmk web performance would be acceptable because intensive administration of the Checkmk web interface is unlikely to be the priority at that moment.

This is also why we keep vm.swappiness=1.

The small amount of swap used during normal operation is an acceptable trade-off for having considerably more graceful behaviour if the VM is ever forced closer to its minimum memory allocation.

📊 The Three Configurations Compared

Configuration	Behaviour under heavy GUI load	Longer-term behaviour	Production assessment
2 GB, `swappiness=0`	Critically low available memory; GUI/check execution can stall and OOM is possible	Swap stays largely empty, but at the cost of safety margin	Not recommended
2 GB, `swappiness=1`	Stable, but noticeable swap usage	Swap becomes part of normal memory management	Possible if heaven swap usage is acceptable
2.5 GB, `swappiness=1`	Large memory reserve and stable GUI operation	Cold pages gradually accumulate in swap, while plenty of RAM remains available	Recommended for our workload

🐧 Debian Bookworm → Trixie

Our original Checkmk 2.3 installation ran on Debian 12 Bookworm.

After upgrading the Checkmk site to 2.5, we also upgraded the operating system to Debian 13 Trixie and installed the corresponding Checkmk 2.5 package.

The operating system upgrade completed without problems and did not produce any meaningful change in memory consumption.

✅ Conclusion

The original goal was to run Checkmk 2.5 Community comfortably on a 2 GB VM.

Strictly speaking, we did not achieve that.

But 2 GB turned out to be surprisingly close.

After reducing unnecessary memory consumers, Checkmk 2.5 can monitor around 20 hosts and more than 1,000 services with only 2 GB RAM.

The limiting factor is not the normal monitoring workload.

It is the web interface.

With only 2 GB RAM, intensive GUI activity creates temporary memory peaks large enough that we had to choose between two undesirable behaviours:

swappiness=0
→ almost no swap
→ insufficient safety margin under peak load

swappiness=1
→ stable operation
→ noticeable swap usage

Adding only 512 MB changed that significantly.

With 2.5 GB RAM, two Apache workers, the Automation Helper disabled and unused Checkmk components disabled, the same intensive GUI workload runs comfortably with a large amount of memory still available.

swappiness=1 does mean that Linux gradually moves some very cold pages to swap even when there is no current memory shortage. In our longer-term observation, swap usage grew to roughly 235 MB while around 1.2 GB of RAM was still available.

We do not consider this a problem.

The relevant question is not whether swap contains data, but whether the system is actively thrashing between RAM and swap.

For our installation, the practical conclusion is therefore:

2 GB RAM is possible, but 2.5 GB is the point where Checkmk 2.5 Community becomes comfortable.

The final setup keeps 1 GB of swap with vm.swappiness=1 as a safety net rather than trying to achieve a cosmetically perfect 0 B swap counter.

And sometimes the most useful optimization result is finding the point where further optimization stops being worth 512 MB of RAM.

Like this article

🔄 Upgrade Path: 2.3 → 2.4 → 2.5

🔍 Finding the Memory Consumers

✂️ Disabling the Automation Helper

🌐 Reducing Apache Concurrency

🧹 Disabling Unused Checkmk Components

Agent Receiver

Event Console

💾 The Real Question: What Should vm.swappiness Be?

🧪 Variant 1: 2 GB RAM, vm.swappiness=0

Conclusion for this variant

🧪 Variant 2: 2 GB RAM, vm.swappiness=1

Conclusion for this variant

🧪 Variant 3: 2.5 GB RAM, vm.swappiness=1

🛟 Reducing Swap from 4 GB to 1 GB

🎈 Proxmox Ballooning

📊 The Three Configurations Compared

🐧 Debian Bookworm → Trixie

✅ Conclusion

💾 The Real Question: What Should `vm.swappiness` Be?

🧪 Variant 1: 2 GB RAM, `vm.swappiness=0`

🧪 Variant 2: 2 GB RAM, `vm.swappiness=1`

🧪 Variant 3: 2.5 GB RAM, `vm.swappiness=1`