error linux

Linux Exit Code 137 - Killed (SIGKILL)

Understanding Linux exit code 137 - the process was forcefully killed by SIGKILL, often due to out-of-memory (OOM) conditions.

What It Means

Exit code 137 (128 + 9) indicates that the process was terminated by SIGKILL (signal 9). Unlike SIGTERM or SIGINT, SIGKILL cannot be caught, blocked, or ignored — the kernel immediately terminates the process.

This exit code is most commonly seen when the Linux OOM (Out Of Memory) killer terminates a process, or when someone runs kill -9.

Common Causes

  • Linux OOM killer terminated the process due to memory exhaustion
  • Someone ran kill -9 <pid> to forcefully stop the process
  • Docker or Kubernetes killed the container for exceeding memory limits
  • CI/CD system terminated a job that exceeded time or resource limits
  • Systemd killed a service that didn’t stop within the timeout period
  • Process consumed too much memory in a cgroup-limited environment

How to Fix

Check if OOM killer was involved

# Check kernel logs for OOM messages
dmesg | grep -i "oom\|killed process"

# Check specific process OOM events
grep -i "oom" /var/log/syslog
grep -i "oom" /var/log/kern.log

# Check system memory
free -h

Increase memory limits

# Docker - increase container memory limit
docker run -m 4g --memory-swap 4g your-image

# Docker Compose
# services:
#   app:
#     mem_limit: 4g

# Kubernetes - increase pod memory limit
# resources:
#   limits:
#     memory: "4Gi"
#   requests:
#     memory: "2Gi"

Fix memory leaks (Node.js)

// Increase Node.js heap size
// node --max-old-space-size=4096 app.js

// Monitor memory usage
setInterval(() => {
  const usage = process.memoryUsage();
  console.log({
    heapUsed: `${Math.round(usage.heapUsed / 1024 / 1024)}MB`,
    heapTotal: `${Math.round(usage.heapTotal / 1024 / 1024)}MB`,
    rss: `${Math.round(usage.rss / 1024 / 1024)}MB`
  });
}, 10000);

Fix memory issues in Python

import tracemalloc
tracemalloc.start()

# Your code here

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:10]:
    print(stat)

Prevent OOM in Docker

# Use multi-stage builds to reduce image size
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

FROM node:20-alpine
WORKDIR /app
COPY --from=builder /app/node_modules ./node_modules
COPY . .
CMD ["node", "app.js"]

Handle systemd timeout kills

# /etc/systemd/system/your-service.service
[Service]
TimeoutStopSec=30  # Give more time before SIGKILL
# systemd sends SIGTERM first, then SIGKILL after TimeoutStopSec