Updated

Monitoring a Linux server means collecting six families of numbers from the kernel and deciding which of them are allowed to wake someone up: CPU time by state, load average relative to core count, memory available, filesystem space per mount, disk I/O per device, and network bytes per interface. The kernel already exposes all of it through /proc and /sys. The work is choosing the right number from each family, sampling it often enough to be useful, and keeping the history so you can answer "when did this start" instead of guessing.

This guide covers what to collect, the commands that read it, and the thresholds I would actually page on.

Key takeaways

  • A single CPU percentage averaged over 5 minutes hides the case that matters. Split it into user, system, iowait, and steal, and read it per core, because one saturated core out of 8 shows up as 12.5% overall.
  • Load average on Linux counts both runnable threads and threads in uninterruptible sleep, so a disk stall inflates it. Divide by nproc before judging it, and treat a sustained 1 minute load above 2.0 per core as a real signal.
  • used memory is the wrong number because Linux fills spare RAM with page cache. Alert on MemAvailable dropping below about 10% of total instead.
  • Filesystem alerts belong per mount point, not per host. A root partition at 90% used with 8 GB free and a 500 GB data volume at 25% are two completely different situations that a host average would blend into one.
  • Sustained iowait above 20% of wall clock almost always means the storage layer, not the application, and it is the cheapest early warning you can collect on a database host.

What to monitor on a Linux server, and why

Every incident I have debugged on a Linux box came back to one of these seven signals. The rest is detail.

Signal The number that matters Common failure it catches
CPU Time per state (user, system, iowait, steal) as % of wall clock, per core Runaway process, noisy neighbour on a VM, storage stall
Load average 1, 5, 15 minute values divided by core count Queueing that a CPU percentage alone will not show
Memory MemAvailable as % of MemTotal OOM killer about to terminate your process
Filesystem Used and free bytes per mount, plus inode usage Full disk taking down writes, logs, and the database
Disk I/O Read and write bytes per block device, and await Saturated volume, throttled cloud disk
Network Bytes in and out per interface Saturated link, unexpected egress, dead interface
Host liveness Time since last metric received The box is gone and nothing else will tell you

That last row is the one teams forget. A monitoring setup that only alerts on thresholds goes silent when the server dies, because a dead host reports nothing at all. Liveness detection has to be inverted: alert on the absence of data.

CPU: why a single percentage lies

CPU utilization is a time share. When a dashboard says 40%, it means the CPUs spent 40% of the sampling window doing something other than idling. That is three different questions collapsed into one answer.

Split it by state. user is your application code. system is kernel time, which climbs with syscall-heavy work like small network writes. iowait is idle time where at least one task was blocked on I/O, so it is a storage signal wearing a CPU costume. steal is time the hypervisor gave to someone else, and on a shared cloud instance it is the number that explains "the app got slower and nothing changed".

# mpstat ships with sysstat: apt install sysstat / dnf install sysstat
$ mpstat -P ALL 2 3
Linux 6.8.0-45-generic (web-01)   07/25/2026   _x86_64_   (4 CPU)

10:12:41 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal   %idle
10:12:43 AM  all   18.24    0.00    4.11   22.68    0.00    0.51    0.00   54.46
10:12:43 AM    0   16.84    0.00    3.57   61.22    0.00    0.51    0.00   17.86
10:12:43 AM    1   19.10    0.00    4.52    5.03    0.00    0.50    0.00   70.85
10:12:43 AM    2   18.59    0.00    4.02    12.06   0.00    0.50    0.00   64.83
10:12:43 AM    3   18.43    0.00    4.33    12.41   0.00    0.53    0.00   64.30

The all row says the box is a bit over half idle. Core 0 says something on that core is blocked on disk 61% of the time. Averaging is what turned a clear signal into a vague one. I go deeper on reading these states in how to monitor CPU usage on Linux, and the CPU utilization glossary entry has the short definition.

The other trap is sampling window. A 5 minute average of a workload that spikes for 20 seconds every minute will read as comfortable while your p99 latency is falling apart. 30 second samples are a reasonable default for host metrics.

Load average only means something next to your core count

Linux load average counts processes in the run queue plus processes in uninterruptible sleep, usually blocked on disk or NFS. That second part is specific to Linux and it is why a load of 30 can appear on a box with idle CPUs during a storage incident.

$ cat /proc/loadavg
3.42 2.87 1.94 4/512 20481

$ nproc
4

The first three values are exponentially damped moving averages over roughly 1, 5, and 15 minutes. The fourth field is running/total kernel scheduling entities, and the fifth is the last PID created. On this 4 core box, 3.42 divided by 4 gives 0.86 per core, which is busy but not queueing badly. The 1 minute value climbing above the 15 minute value tells you the pressure is new.

I treat a sustained 1 minute load above 2.0 per core as worth investigating and above 4.0 per core as worth paging, but only after checking whether iowait is doing the inflating. Linux load average explained walks through the arithmetic and the D-state case, and the load average definition covers the basics.

Memory: "used" is the wrong number

Linux fills unused RAM with page cache because empty RAM does no work. used therefore climbs toward full on any long-running server and stays there. Alerting on it produces noise forever.

$ free -m
               total        used        free      shared  buff/cache   available
Mem:            7962        5421         318         142        2223        2098
Swap:           2047         128        1919

$ grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached)' /proc/meminfo
MemTotal:        8152364 kB
MemFree:          326112 kB
MemAvailable:    2148920 kB
Buffers:          128444 kB
Cached:          2149832 kB

free reads 318 MB free, which looks alarming and is not. MemAvailable reads 2098 MB, a kernel estimate of how much a new workload could claim without pushing the system into swap. That is the number to graph and to alert on, expressed as a percentage of total.

When memory actually runs out, the kernel picks a victim. The evidence lands in the kernel ring buffer:

$ sudo dmesg -T | grep -i 'killed process'
[Fri Jul 24 03:41:12 2026] Out of memory: Killed process 2841 (postgres) total-vm:4210348kB, anon-rss:3891204kB

If you find that line and your monitoring never fired, your threshold was on the wrong metric. How to monitor memory usage on Linux covers the cgroup case and per-service accounting, and memory utilization has the definition.

Filesystem: monitor per mount, never per host

A full filesystem is the most boring outage and one of the most common. It takes down writes, log rotation, and often the database, and it is completely predictable in advance.

$ df -hT -x tmpfs -x devtmpfs
Filesystem                 Type  Size  Used Avail Use% Mounted on
/dev/nvme0n1p2             ext4   78G   66G  8.1G  90% /
/dev/nvme0n1p1             vfat  511M  6.1M  505M   2% /boot/efi
/dev/mapper/data--vg-data  xfs   500G  120G  380G  25% /var/lib/postgresql

Averaged across the host, this server is at roughly 32% used and looks fine. Per mount, the root filesystem has 8.1 GB left and is one large log file away from trouble. Always alert per mount point.

Space is not the only way a filesystem fills. Inodes run out independently, and the symptom is confusing: df -h shows free space while writes fail with ENOSPC.

$ df -i /
Filesystem       Inodes   IUsed   IFree IUse% Mounted on
/dev/nvme0n1p2  5242880 4981233  261647   96% /

Millions of tiny session files or unrotated per-request logs get you here. How to monitor disk space and get alerted before a volume fills covers rate-of-change alerting, which beats a static percentage: 78 GB filling at 3 GB a day gives you a usable "time to full" instead of a threshold that fires at 3 a.m. with no lead time. See also filesystem usage.

Disk I/O: bytes moved, and the wait that comes with it

Disk I/O has two halves worth collecting. Throughput tells you how much data is moving per device. Latency tells you how long the requests are taking, which is what your application actually feels.

$ iostat -xz 2 3
Linux 6.8.0-45-generic (web-01)   07/25/2026   _x86_64_   (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          18.24    0.00    4.11   22.68    0.00   54.97

Device     r/s   rkB/s  rrqm/s  %rrqm r_await rareq-sz    w/s     wkB/s  wrqm/s  %wrqm w_await wareq-sz  aqu-sz  %util
nvme0n1  12.00  480.00    0.00   0.00    0.42    40.00 310.50  22436.00   18.00   5.48   14.82    72.26    4.63  92.40

w_await of 14.82 ms on an NVMe device is slow: that class of hardware normally answers writes in well under a millisecond. Paired with 22.68% iowait, this is a storage problem, not an application problem.

Two cautions on this output. %util means "percentage of time the device had at least one request in flight", which was a good saturation proxy on single-queue spinning disks and is close to meaningless on multi-queue NVMe that services many requests in parallel. And iostat -x needs the sysstat package, which is not installed by default on Ubuntu Server or on minimal RHEL images. Read await and queue depth instead of %util, and see troubleshooting high iowait and disk I/O for the full walkthrough. The glossary entries for iowait and disk I/O cover the terms.

Network throughput: counters, not rates

The kernel exposes cumulative byte counters per interface. Rates are derived by your tooling from the difference between two samples, which is why a monitoring agent that restarts mid-window can produce a spike or a gap in the graph.

$ cat /proc/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
    lo: 1284512    9421    0    0    0     0          0         0  1284512    9421    0    0    0     0       0          0
  eth0: 91238471203 84120394 0  0    0     0          0      1204  40218374102 61203941 0  0    0     0       0          0

For live rates without writing your own subtraction, sar from sysstat does it per second:

$ sar -n DEV 1 3
10:22:31 AM  IFACE  rxpck/s  txpck/s   rxkB/s    txkB/s  rxcmp/s  txcmp/s rxmcst/s  %ifutil
10:22:32 AM   eth0  8421.00  6210.00  9812.44  71204.10     0.00     0.00     0.00    58.24

71 MB/s outbound on a 1 Gbit link is roughly 58% of capacity, which is the point at which I want to know about it rather than the point at which it breaks. On physical NICs you can read the negotiated speed from cat /sys/class/net/eth0/speed, though virtualized interfaces often report -1 there, so on cloud instances take the link speed from the provider's instance spec instead. How to monitor network throughput on Linux covers per-interface baselining.

Once all six families are collected on the same host, they line up on one page over the same time window, which is what makes "when did this start" answerable.

Hyperping server page for prod-web-01 charting CPU utilization split into user and system, memory used in MB, 1, 5 and 15 minute load average, disk IO in MB/s, network bandwidth download and upload, and disk usage, all over the same one hour window

Where the numbers actually come from

Every tool in this guide is a formatter over the same kernel interfaces. Knowing the files is useful when you are on a stripped-down box with no packages installed.

Source file What it holds
/proc/stat Cumulative CPU jiffies per state, per core, since boot
/proc/loadavg 1, 5, 15 minute load, running/total tasks, last PID
/proc/meminfo Total, free, available, buffers, cached, in kB
/proc/diskstats Reads, writes, sectors, and time per block device
/proc/net/dev Cumulative bytes and packets per interface
/proc/mounts Mounted filesystems, devices, and types
/sys/block/<dev>/ Per-device queue settings and scheduler
/sys/class/net/<if>/ Interface state, MTU, speed, MAC

Two properties trip people up. Most of these are counters since boot, not gauges, so a rate requires two samples and a subtraction. And CPU values in /proc/stat are in USER_HZ units (100 per second on almost every distribution), not seconds.

The CLI tools worth knowing

Tool Use it for Package
top Quick look at load, per-state CPU, top consumers preinstalled
htop Same, with per-core bars and a tree view htop
vmstat 1 Run queue, blocked tasks, I/O blocks, CPU split over time preinstalled (procps)
mpstat -P ALL Per-core CPU split sysstat
iostat -xz Per-device throughput, await, queue depth sysstat
sar Historical replay of CPU, memory, disk, network sysstat
free -m Memory totals and available preinstalled
df -hT / df -i Filesystem space and inodes per mount preinstalled
ss -s Socket and TCP connection state summary iproute2

vmstat is the one I reach for first on an unfamiliar box, because it puts the run queue, blocked tasks, and CPU split on one line:

$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  1 131072 325904 128444 2149832   0    0    12   380  742 1310 18  4 56 22  0
 3  2 131072 322108 128448 2151004   0    0     8  4210  981 1644 19  5 52 24  0

Column b is processes blocked on I/O and column wa is iowait. Both non-zero at the same time points at storage before you have opened a single dashboard.

On RHEL and Rocky, sar history requires the collector to be enabled: systemctl enable --now sysstat. Without that, sar only shows the current session.

Why teams move from cron scripts to an agent

Nearly every team starts with a shell script in cron that pipes df -h into a Slack webhook. It works, and then it stops working for reasons that are all the same reason: a cron script has no memory.

  • No history, so you cannot answer "was this already climbing yesterday" or size a volume from six weeks of growth.
  • No cross-host view, so comparing 20 web servers means 20 separate scripts and 20 separate messages.
  • Alert logic duplicated per host, which drifts silently until one box has a 95% threshold and the rest have 80%.
  • No liveness detection, because a script on a dead machine sends nothing, and silence looks identical to health.
  • Nothing buffers during a network partition, so the window you most want to inspect afterwards is exactly the window with no data.

An agent solves those by keeping a persistent process on the host, sampling on a fixed cadence, buffering to disk when the network is down, and shipping to a backend that holds history and evaluates thresholds centrally. That last part matters: a threshold evaluated off the host still fires when the host is gone.

If you want the vendor-neutral version of this collection model, OpenTelemetry host metrics explained covers the hostmetrics receiver and its semantic conventions, which is what the Hyperping agent emits. See also agent-based monitoring and OpenTelemetry.

Thresholds that actually matter

Thresholds are where most setups generate noise. My rule is that a page has to be actionable at 3 a.m. and everything else is a ticket.

Signal Ticket Page Notes
Filesystem used per mount 80% 90%, or under 5 GB free Whichever fires first; rate-of-change beats both
Inode usage per mount 85% 95% Silent failure mode, easy to forget
1 min load / core > 2.0 for 15 min > 4.0 for 10 min Check iowait before blaming CPU
iowait > 10% for 30 min > 25% for 10 min On database hosts, treat as storage until proven otherwise
MemAvailable / MemTotal < 20% for 30 min < 10% for 10 min Never alert on used
CPU non-idle > 85% for 30 min > 95% for 15 min Per core matters more than the average
Network out per interface > 60% of link for 30 min > 85% of link for 10 min Requires knowing the link speed
Time since last metric 2 min stale 5 min offline The one alert that catches a dead box

Duration is doing as much work as the number. A CPU threshold with no for clause fires on every deploy. Server monitoring alert thresholds has per-workload variants for web, database, and queue hosts, and if you are running a single cheap box, how to monitor a VPS covers the smaller setup.

How to use Hyperping for Linux server monitoring

This is the setup I would run on a fleet of Linux servers, from install to on-call routing. It takes about ten minutes for the first host.

1. Install the agent on the host

Create a server in Hyperping, copy the install command, and run it over SSH. The installer sets up a systemd service on Linux and launchd on macOS, on both amd64 and arm64.

curl -fsSL https://hyperping.com/install.sh | sh -s HP_INSTALL_xxxxx

The agent is a compiled binary with an embedded OpenTelemetry collector, around 50 MB RSS, no JVM. It scrapes every 30 seconds and ships gzipped OTLP/HTTP. If the network drops, samples queue on disk at /var/lib/hyperping/queue and survive reboots, so you get the data back once the link returns. Re-running the installer rotates credentials while keeping the same server UUID and history. The agent install docs cover the flags and the uninstall path. Windows is not supported yet.

Repeat that command on the rest of the fleet and every host lands in the same list, grouped and filterable by agent state.

Hyperping Servers list showing the one line install command with Copy and Advanced buttons, status filters for All, Online, Stale and Offline, and a Production group of eight Ubuntu hosts with agent version, CPU, RAM and disk usage and last seen times

2. Read CPU, load, and memory on the server dashboard

The server page shows CPU time per state as a percentage of wall clock, including user, system, and iowait, plus a per-CPU breakdown and the logical core count. Load averages appear as 1, 5, and 15 minute values, so you can divide by cores without leaving the page. Memory shows used, available, and total in bytes alongside utilization.

Host metadata comes with it: hostname, OS, kernel, architecture, CPU model, boot time, and agent version. That is usually enough to answer "is this the box we resized last month" without opening a terminal.

3. Watch filesystems per mount and disk I/O per device

Filesystem usage is reported per real mount point with device, mountpoint, and filesystem type, and pseudo-filesystems are filtered out so you are not looking at tmpfs noise. Disk I/O is reported as read and write bytes per block device. The full list of what is collected lives in the metrics collected reference.

Worth stating plainly: the agent does not ingest per-process or per-container metrics, swap, or packet error counters, and it does not do logs, traces, or APM. On macOS, disk available is an approximation.

4. Track network throughput per interface

Bytes in and out are collected per interface and the dashboard derives TX and RX rates from consecutive samples. Compare that against the link speed for the instance type to see how much headroom is left, and watch for an interface whose egress climbs while request volume does not.

5. Set server alerts and route them to an escalation policy

Server alerting is available on Essentials and above. Set thresholds on CPU, memory, and filesystem usage using the table earlier in this guide, then attach them to an escalation policy and an on-call schedule so the alert reaches whoever is actually awake.

Agent liveness is handled for you: metrics go Stale, then Offline after a threshold, which opens an outage and fires the escalation policy. That is the alert that covers the case where the host stops existing. The server alerting docs go through the configuration.

6. Pair host metrics with uptime checks and a status page

Host metrics tell you the machine is unhealthy. Uptime monitoring tells you customers cannot reach it, which is the part they care about. Run HTTP checks against the services on the box from multiple regions, and when both signals fire together you know it is the host rather than the network path.

Publish a status page on top and the same incident that pages your on-call also tells customers what is happening, without anyone writing a manual update first.

Plans start at $24 a month billed yearly on Essentials with 5 server agents included and 7 days of metric history, $74 for Pro with 20 agents and 14 days, and $249 for Business with 100 agents and 30 days. Extra agents are $2 a month on Essentials and Pro, $1.50 on Business. The free plan includes 1 server agent with 2 days of history and no server alerting, which is enough to see whether the data is useful to you. Full details are on the pricing page and the server monitoring product page.

Where to start

If you are setting this up from scratch, collect all six families from day one but only alert on three: filesystem used per mount, memory available, and host liveness. Those three cover the outages that take a server down without warning, and they produce almost no false positives.

Add CPU, load, and iowait alerts once you have two weeks of history and can see what normal looks like on your own hardware. Thresholds copied from a guide, including this one, are a starting point you should be adjusting by the end of the first month.