🐧 Troubleshooting Linux

Troubleshooting is the disciplined process of moving from a symptom to a verified cause and a safe correction. Good administrators do not start by restarting random services or changing permissions. They establish the impact, collect evidence, test a hypothesis, apply the smallest useful change, and verify the result.

A Linux server can fail at several layers:

User or application
        |
DNS, network, and cloud firewall
        |
Service manager and service configuration
        |
Process, permissions, and environment
        |
CPU, memory, storage, and filesystems
        |
Linux kernel, devices, and physical or virtual host

By the end of this lesson, you should be able to:

Triage a Linux server problem without destroying evidence
Investigate logs, processes, services, resources, storage, networking, and permissions
Separate Linux failures from cloud-provider or application failures
Test hypotheses with focused commands and observable outcomes
Document mitigation, root cause, recovery, and follow-up actions

Start with impact and safety

Before changing anything, answer:

What is broken, and how do users observe it?
Is the problem isolated to one host, service, user, region, or deployment?
When did it start, and what changed just before it started?
Is the system still serving traffic, writing data, or accepting logins?
What access and recovery path is available if the next action makes things worse?

Classify the incident by user impact and urgency. A production data-loss risk deserves different handling from a development command that fails. Keep an existing SSH session or provider console open while changing access, networking, storage, or authentication configuration.

Prefer read-only inspection first. Avoid rebooting, deleting logs, killing processes, detaching disks, running fsck on a mounted filesystem, or changing firewalls before you understand the likely failure and recovery path.

Establish a baseline

Capture a small baseline before investigating details:

date -u
hostname
cat /etc/os-release
uname -a
uptime
who

This records the time, host, distribution, kernel, uptime, and logged-in users. Compare the output with a healthy host or an earlier known-good baseline where possible.

Check whether the system is reporting failed services:

systemctl is-system-running
systemctl --failed

These commands require a systemd-based system. Containers, WSL environments, Alpine/OpenRC systems, and minimal images may not provide them.

Build a timeline

A timeline often separates a root cause from a later symptom. Record:

First known failure and recovery attempts
Deployments, package updates, reboots, configuration changes, and certificate rotations
Changes in traffic, data volume, external dependencies, or cloud resources
Alerts and their first observed timestamps
Commands run and the result of each change

Use UTC when correlating systems in different locations. Do not rely only on the current clock if time synchronization is broken; compare journal timestamps, monitoring timestamps, and external events.

Logs are evidence

Logs can show what happened before a service failed, but a log message is not automatically the root cause. Filter by time, service, priority, and related identifiers.

On a system using systemd:

journalctl -b
journalctl -k -b
journalctl -p warning..alert --since "30 minutes ago"
journalctl -u example.service --since "30 minutes ago"
journalctl -u example.service -f

These show current-boot messages, kernel messages, warning-and-higher messages, a service's recent messages, and live service output. -b -1 can show the previous boot when persistent journal storage is configured.

Traditional logs commonly live under /var/log:

ls -lah /var/log
tail -n 100 /var/log/messages 2>/dev/null
tail -n 100 /var/log/syslog 2>/dev/null

File names and logging services vary by distribution. journald may forward to rsyslog, and the journal may be volatile unless persistent storage is configured.

Protect logs during an incident. Do not clear or rotate them manually to make a disk alert disappear, and redact passwords, access tokens, private keys, personal data, and other sensitive values before sharing output.

Check services and processes

If an application is unavailable, first determine whether its service and process exist:

systemctl status example.service
ps -ef | grep '[e]xample'
pgrep -af example

Then inspect the process that the service manager reports:

systemctl show example.service -p MainPID -p User -p Group -p ExecMainStatus
ps -p PID -o pid,ppid,user,group,stat,pcpu,pmem,etime,comm,args

Replace PID with the confirmed process ID. A service can be active while its application endpoint is unhealthy, and a process can exist outside the service manager. Check both the manager's state and the user-facing behavior.

If a process is repeatedly restarted, inspect the journal and unit's restart policy. Do not repeatedly send SIGKILL; it can hide the original error and interrupt cleanup. Stop or restart through the appropriate supervisor when possible.

Investigate CPU and load

Useful first commands are:

uptime
top
ps -eo pid,ppid,user,stat,pcpu,pmem,comm,args --sort=-%cpu
vmstat 1 5

High CPU may result from normal traffic, an infinite loop, inefficient queries, encryption or compression work, a runaway child process, or a kernel/network interrupt problem.

The load average is not a percentage. It includes runnable work and, on Linux, tasks in uninterruptible sleep. Compare it with the number of CPUs and inspect process states. A high load with many D-state processes points toward I/O or kernel waits, not necessarily CPU exhaustion.

Do not lower priority or kill the busiest process until you know whether it is providing a critical service. Capture the command, owner, CPU trend, and related logs first.

Investigate memory

free -h
ps -eo pid,ppid,user,stat,rss,vsz,pmem,comm,args --sort=-rss
grep -E '^(Mem|Swap)' /proc/meminfo
journalctl -k | grep -iE 'out of memory|oom|killed process'

RSS is an approximation of resident memory; VSZ is virtual address space and is not equivalent to RAM usage. Shared libraries and page cache complicate process-level totals.

Look for:

Available memory falling rapidly
Swap activity and high I/O wait
A process with a growing RSS, suggesting a leak or unbounded workload
Kernel OOM-killer messages
Container or service cgroup limits that are lower than host memory

Adding swap or killing a process may restore service temporarily, but it does not explain the allocation pattern. Record the OOM event, workload, limits, and recent changes before applying a permanent fix.

Investigate storage and filesystems

Start with the storage topology and capacity:

lsblk -o NAME,SIZE,TYPE,FSTYPE,UUID,MOUNTPOINTS
findmnt
df -hT
df -ih
du -xhd1 /var 2>/dev/null

Distinguish these cases:

Filesystem full: df -h is near 100 percent.
Inodes full: df -i is near 100 percent even when byte space remains.
Large directory: du identifies logs, caches, uploads, or application data.
Deleted-open file: A process still holds a deleted file, so df remains high after deletion.
Missing mount: Data is being written to the underlying mount-point directory instead of the intended volume.
Read-only filesystem: The kernel may have remounted it after errors.

Find deleted-open files when lsof is installed:

lsof +L1

Check storage and filesystem errors:

journalctl -k --since "30 minutes ago"
dmesg --level=err,warn

Do not delete files from an unknown directory to free space. Confirm ownership, retention rules, backups, and whether a service still needs the files. Do not run filesystem repair tools on a mounted filesystem unless the tool explicitly supports that operation.

Investigate networking

Test the network in layers:

ip addr
ip route
getent hosts example.com
ss -ltnp

For a known HTTP endpoint, inspect DNS, connection, and TLS behavior:

curl -vI --connect-timeout 5 https://example.com/

This is a diagnostic request; use an endpoint and method that are safe for the application. ping can be useful, but blocked ICMP does not prove that TCP or HTTP is unavailable.

Interpret common symptoms carefully:

Symptom	Likely area
Name-resolution failure	Resolver configuration, DNS service, search domain, or upstream DNS
Timeout	Route, firewall, security group, packet filtering, overloaded service, or dropped traffic
Connection refused	Host reachable but no listener, wrong port, or active rejection
TLS or certificate error	Certificate validity, hostname, trust store, clock, or protocol configuration
HTTP 4xx	Request, authentication, authorization, or application policy
HTTP 5xx	Application failure, dependency failure, proxy, or upstream timeout

Cloud networking adds layers outside the Linux guest: load balancers, security groups, network ACLs, routes, NAT, private endpoints, and provider health. A successful ss listener proves only that a local process is listening; it does not prove that traffic can reach it from the client.

Investigate permissions and identity

For Permission denied, use the actual service identity:

id appuser
namei -l /srv/example/config.yml
stat /srv/example/config.yml
getfacl /srv/example/config.yml
sudo -u appuser test -r /srv/example/config.yml

Check the full path, not only the final file. The process needs search permission on parent directories. Also check:

File owner, group, mode, ACL, and setuid/setgid bits
Service user, supplementary groups, and working directory
SELinux or AppArmor policy and labels
Read-only mounts, filesystem flags, and container namespaces
Whether a shell-only environment variable or secret is absent

Do not respond to a permission error with chmod -R 777. That can create a security incident without fixing the actual path, identity, ACL, or mandatory-policy problem.

Investigate boot and failed mounts

If the server recently rebooted or failed to boot, use a provider console or local recovery path when SSH is unavailable. On a running system:

systemctl --failed
journalctl -b -p warning..alert
journalctl -k -b
systemd-analyze blame
findmnt --verify --verbose

Check recent changes to:

/etc/fstab and filesystem UUIDs
Bootloader and kernel parameters
Network configuration and hostnames
Enabled services and unit drop-ins
Package updates and required reboots
Encryption keys, initramfs, and storage availability

Use rescue or emergency mode only with a recovery plan. Emergency environments may mount only the root filesystem and provide no network, so understand how you will preserve evidence and restore access.

Test one hypothesis at a time

A useful troubleshooting loop is:

Observe -> Form hypothesis -> Predict result -> Test safely
   ^                                             |
   +--------- Verify change and update theory ---+

Example: if a web service returns 502, do not immediately reinstall it.

Confirm whether all users or only one location is affected.
Check the load balancer or reverse proxy health.
Check the backend service state and recent logs.
Check whether the backend port is listening with ss.
Make a safe local request to the backend.
Check dependencies, resource pressure, and recent deployments.
Roll back or correct the smallest confirmed cause.
Verify the external user journey and continue monitoring.

This sequence narrows the failing boundary while preserving evidence.

Capture a diagnostic bundle

For a support case or handoff, collect only what is necessary:

Hostname, distribution, kernel, uptime, and time zone
Scope, impact, timeline, and recent changes
Service states, process listings, resource measurements, and network listeners
Relevant journal and application logs for a bounded time window
Storage topology, mounts, capacity, and error messages
Configuration versions and deployment identifiers
Commands run, changes made, results, and current mitigation

Tools such as sos report can collect standardized diagnostic information on supported distributions. Review and clean sensitive data before sharing a report. Diagnostic archives may contain hostnames, IP addresses, usernames, configuration, and other private information.

Recovery and incident response

After finding a likely cause, choose a mitigation that matches the risk:

Roll back a known-bad deployment.
Remove or isolate failing traffic.
Fail over to a healthy instance or zone.
Restart a service only when its state and data behavior are understood.
Restore data from a verified backup when corruption or deletion occurred.
Replace a damaged VM or rebuild from a known image.

Recovery is not complete when a process is running. Verify data integrity, authentication, network reachability, user-facing behavior, monitoring, and background work. Record the mitigation and any residual risk.

After service is restored, conduct a blameless review:

What failed and why?
Why did monitoring or controls not prevent or detect it sooner?
Which temporary actions should become permanent safeguards?
What tests, documentation, alerts, or runbooks need improvement?

Common dangerous shortcuts

Avoid these as first responses:

Running kill -9 without identifying the process and recovery impact
Running rm -rf with an unverified variable or path
Running mkfs, fdisk, parted, or fsck on an uncertain device
Changing a firewall or SSH configuration without a second access path
Applying chmod -R 777 or changing ownership broadly
Disabling SELinux, AppArmor, signature verification, or host-key checking to suppress an error
Rebooting before collecting evidence when the system is still accessible
Clearing logs, deleting caches, or rotating files without recording what was removed
Reinstalling a package or service before checking configuration and data

The fastest fix is not always the safest fix. Restore service in a controlled way, then address the underlying cause.

Practical design checklist

Can operators access the server through SSH, a bastion, provider console, or another recovery path?
Are logs, metrics, traces, configuration, deployments, and backups available during an incident?
Are CPU, memory, storage, inode, I/O, network, and file-descriptor limits monitored?
Can the team identify the process user, service manager, environment, dependencies, and failure domain?
Are diagnostic commands and destructive commands clearly separated in runbooks?
Are failover, rollback, restore, and reboot procedures tested?
Does every incident produce a timeline, mitigation record, root-cause hypothesis, and follow-up owner?

Summary

Troubleshooting starts with scope, impact, timeline, safety, and evidence.
Investigate from the user-facing symptom down through networking, services, processes, permissions, resources, storage, and the kernel.
Use logs filtered by time and service, and compare process, service-manager, and user-facing states.
CPU, memory, storage, inodes, I/O, network, permissions, and cloud controls can fail independently or together.
Test one hypothesis at a time and predict what evidence should change before making a correction.
Prefer graceful, reversible actions and avoid unverified kill, deletion, formatting, repair, firewall, or permission changes.
Recovery includes validating data, dependencies, access, monitoring, and the complete user journey.
A blameless post-incident review turns one failure into better detection, automation, documentation, and resilience.

Check your understanding

Why should impact and scope be established before changing a production server?
What is the difference between df and du when diagnosing disk usage?
Why can a service be active while its application is unavailable?
What does a D process state commonly suggest?
Why does a successful local listener check not prove that a remote client can connect?
What should you check before widening permissions after a Permission denied error?
Why should evidence be collected before rebooting or using SIGKILL?
What makes a recovery complete after a service process is running again?

Start with impact and safety​

Establish a baseline​

Build a timeline​

Logs are evidence​

Check services and processes​

Investigate CPU and load​

Investigate memory​

Investigate storage and filesystems​

Investigate networking​

Investigate permissions and identity​

Investigate boot and failed mounts​

Test one hypothesis at a time​

Capture a diagnostic bundle​

Recovery and incident response​

Common dangerous shortcuts​

Practical design checklist​

Summary​

Check your understanding​

Further reading​