Debugging Linux Servers

Introduction

The more serverless I do, the more old-school server debugging I risk forgetting. So here is a brain dump of the common commands I use.

• Check Linux Version
• Debug Resources
  • Networking Issues
  • Debug Processes
  • CPU Issues
  • Memory Issues
  • Disk Issues
    • Disk Performance
    • Disk Space
• Logging
  • Common Logs
• Extras
  • Webserver Config
  • Server Maintenance

Check Linux Version

The command to check your linux version varies per operating system:

cat /etc/os-release
lsb_release -a
hostnamectl

Networking Issues

Local port scan, to find which ports are open:

nmap localhost

External port scan, to find which ports are open. Refine the range to find what you're looking for.

nmap -A -p1-65535 <some_ip>

Ping the first port of call to confirm you can connect to an external IP:

ping <some_ip>

Curl is always the goto solution for testing a page loads:

curl -I http://localhost

But telnet is useful too:

telnet localhost 80
    GET / HTTP/1.1
    Host: localhost

sudo netstat -tulpn | grep LISTEN

Debug Processes

List all running processes:

ps -aux

Find a specific process using grep:

ps -aux | grep 'java'

Now that you have identified the problematic process, you can see what it's doing by using strace. Strace allows you to trace the system calls and signals a process is using.

strace -p <insert_some_pid>
# example: strace -p 1234

CPU Issues

The classic approach, which always be available:

top

The better approach, which you will likely need to install:

htop

Memory Issues

List free memory available:

free -mh

Again! Using htop to show memory is ideal.

htop

Disk Performance

iotop

iostat

If something is intermittent, then you can listen over a period of seconds and see the total:

pidstat -dl 10

Disk Space

The classic! See the disk space in human readable format.

df -h

It's not enough to have disk free, you all need inodes free. Check those by running:

df -i /

If you are running out of disk space, but you're not sure where then run this in the parent folder:

du -sh ./*

Common Logs

Because piping all logs to STDOUT and STDERR would make too much sense, sometimes you need to go digging.

# Debian
tail -fn 100 /var/log/syslog   # Global system activity data
tail -fn 100 /var/log/auth.log # Security related events

# RHEL / CentOS / Amazon Linux
tail -fn 100 /var/log/messages # Global system activity data
tail -fn 100 /var/log/secure   # Security related events

# Apache logs (apt)
tail -fn 100 /var/log/apache2/error.log
tail -fn 100 /var/log/apache2/access.log

# Apache logs (yum)
tail -fn 100 /var/log/httpd/error.log
tail -fn 100 /var/log/httpd/access.log

# Nginx logs
tail -fn 100 /var/log/nginx/error.log
tail -fn 100 /var/log/nginx/access.log

# Kernel logs
tail -fn 100 /var/log/kern.log

# Cron logs
tail -fn 100 /var/log/cron

Different servers will hide the logs in different locations, particularly on instances running multiple websites.

Webserver Config

Check your Apache config is valid first by running:

apachectl configtest

Check your Nginx config is valid first by running:

nginx -t

Server Maintenance

Deleting files over 14 days old:

find /some/large/directory/* -mtime +14 -exec rm {} \;

Find all non-images in a directory:

find . ! -name '*.png' ! -name '*.jpg' ! -name '*.jpeg' ! -name '*.gif' -type f