DebuggingReliabilitySRE

Debugging Strategies When Your App Gets Randomly Killed: Insights from Process Roulette

UUnknown

2026-03-10

9 min read

Diagnose and fix 'process roulette' with core dumps, eBPF tracing, watchdogs, and deterministic stress tests. Turn randomness into reproducible evidence.

When your service dies for no apparent reason: stop treating it as luck

Hook: You’re on-call. The service restarts again but logs show nothing helpful. Customers notice intermittent errors. It feels like someone is playing “process roulette” — processes die at random. Before you accept random as reality, use a structured diagnostics workflow that turns luck into reproducible evidence.

Top-level diagnosis: the quick checklist (do these first)

Start here to avoid wasted time chasing red herrings. These steps reveal whether a kill is external (OOM, watchdog, orchestrator), deterministic (bug in code), or environment-driven (resource limits, hardware faults).

Check the kernel log: dmesg for OOM killer entries or hardware errors.
Inspect process restarts: systemd, container runtime or orchestrator restart counts.
Gather crash artifacts: core dumps, coredumpctl entries, or crash reports (Windows Event Log/cloud provider crash logs).
Record exact conditions: time, request patterns, load, and sequence of API calls.
Set up a lightweight tracing shunt: strace for short runs or eBPF for production at scale.

Common immediate signals

OOM killer: dmesg contains "Out of memory" and "Killed process" entries, or the process has a sudden SIGKILL with no stacktrace.
Segfault / SIGSEGV: core dump present; process exited with signal 11.
SIGABRT / assert(): indicates assertion failure or abort() in code; core and logs may show stack.
External kill (SIGKILL by orchestrator): restart policy, liveness probes, or a debugging tool may send SIGKILL.

Evidence collection: enable core dumps and automated collection

Core dumps are the single most powerful artifact for post-mortem debugging of native crashes. In 2026, production workflows commonly rely on centralized core stores and automated triage.

Enable core dumps (Linux)

Quick commands to enable system-wide core generation:

ulimit -c unlimited               # shell for interactive session
sudo sysctl -w kernel.core_pattern=/var/crash/core-%e-%p-%t
sudo mkdir -p /var/crash && sudo chown root:root /var/crash

With systemd, prefer systemd-coredump. Confirm configuration in /etc/systemd/coredump.conf. Use coredumpctl list to enumerate cores.

Cores inside containers and Kubernetes

Set container ulimits: docker run --ulimit core=-1 or Pod spec with securityContext.
Give container permission to write cores and expose a volume for /var/crash.
Use kubectl debug or ephemeral containers to retrieve cores from running nodes.

Automated collection and upload

Use a coredump hook to automatically upload artifacts to a crash-collection service or an S3 bucket. For example, set kernel.core_pattern to a script that gzips and uploads the core plus /proc/PID/maps and logs.

Analyze the core: practical gdb steps

Once you have a core file and the matching binary and symbols, the following commands quickly expose the failing thread, stack, and local variables.

# inspect
gdb -q /path/to/binary /path/to/core
# in gdb
thread apply all bt full       # full stack for all threads
info registers                # registers at crash (useful for sigill/sigsegv)
frame 0
info locals                   # variables in crashing frame
list                         # view source around crash

Automate stack extraction for triage with:

gdb -batch /path/to/binary /path/to/core -ex "thread apply all bt full" -ex "quit" > /tmp/crash.txt

Tracing: short-lived and production-safe strategies

Tracing gives context: which syscalls were in-flight, what files were open, and what sequences preceded termination. Pick tools based on your risk tolerance.

Quick and dirty: strace / ltrace

Attach for short periods to capture syscalls: strace -ff -o /tmp/strace.out -p PID.
For new process runs: strace -ff -o /tmp/strace.out -- ./myapp args.

Production-grade observability: eBPF

By 2026, eBPF is the default for low-overhead production tracing. Use tools like bpftrace, bcc or observability platforms built on eBPF (Cilium Hubble, Pixie, or custom libbpf CO-RE scripts).

# trace process kills and OOM events with bpftrace (one-liner)
bpftrace -e 'tracepoint:oom:oom_kill { printf("OOM killed: pid=%d comm=%s\n", args->pid, comm); }'
# or trace kill syscalls
event:syscalls:sys_enter_kill /comm=="myapp"/ { printf("kill called: pid=%d sig=%d\n", args->pid, args->sig); }

Use recorded traces to correlate requests with failures and determine whether process termination is self-inflicted or external.

Perf and off-CPU analysis

For performance-related crashes or timeouts, use perf record and FlameGraphs to see if threads block and then time out under load.

Reproduction techniques: make failure deterministic

If something seems random, the next objective is deterministic reproduction. Without reproducing, fixes are guesses.

Build a minimal reproducible case

Strip the system down: disable unrelated services and features, run only the binary under test.
Isolate inputs: save the exact request sequences or message payloads that precede failure.
Use recorded traffic playback (tcpreplay) or client scripts to reproduce request timing and order.

Record-and-replay for concurrency bugs

Use deterministic record-and-replay tools like rr to capture a failing run and replay it under a debugger. rr is particularly effective for intermittent crashes caused by race conditions on x86_64 Linux.

rr record ./myapp <args>
# reproduce later
rr replay

Simulate the environment: fault injection and chaos testing

Controlled fault injection turns random fails into repeatable experiments.

System faults: stress-ng for CPU/memory/disk stress; example: stress-ng --vm 2 --vm-bytes 80% --timeout 60s.
Network faults: use tc or network chaos tools to introduce latency, packet loss or reordering.
Syscall failures and memory allocation failures: libfiu or application-level toggles to force failure paths.
Kubernetes chaos: use LitmusChaos or Chaos Mesh to run pod-level faults in CI.

Find memory corruption and race conditions

Random-looking crashes often come from memory errors or concurrency bugs. Use sanitizers and dynamic tools during a deterministic reproduction.

Use modern sanitizers in CI

ASan (AddressSanitizer) for buffer overflows and use-after-free.
TSan (ThreadSanitizer) for data races.
MSan / UBSan for uninitialized memory and undefined behavior.

Build a debug variant with sanitizers and reproduce the failure in CI or an isolated lab environment. Often the sanitizer will surface the root cause immediately.

Valgrind and memory-checking

Valgrind is slower but can find subtle leaks and misuse. Use it for longer repro cases where ASan misses issues.

Watchdogs and automatic mitigation

Watchdogs both detect and prevent process roulette from escalating. Treat them as part of your defensive architecture.

Use the right watchdog for your platform

systemd watchdog: configure WatchdogSec= and implement sd_notify("WATCHDOG=1"). systemd restarts unhealthy services and records useful restart metrics.
Kubernetes liveness/readiness: use graceful liveness checks and backoff to avoid aggressive restart loops. Ensure liveness checks are meaningful (not just TCP port checks).
External watchdogs: in some cases, a supervisor or process manager (runit, s6, supervisor) with exponential backoff and automated dumps is appropriate.

Configuring friendly restarts

Don’t mask the problem by always restarting immediately. Use backoff policies and alert thresholds so that repeated failures trigger post-mortems rather than noisy restarts.

Observability & SRE integration

Detecting and diagnosing process roulette is a team effort that should tie into your SRE workflows.

Instrumentation to improve diagnostics

Structured logs with correlation IDs and full context (request, user, feature flags).
Metrics: process restarts, crash counts, OOM events, thread dumps per minute.
Distributed tracing (OpenTelemetry) to link external requests to internal failures.

Alerts and runbook automation

Create runbooks that guide the on-call engineer through evidence collection (which logs to pull, where cores are stored, which traces to enable). Automate the creation of incident tickets with pre-attached artifacts.

Advanced workflows and 2026 trends

Here’s what teams doing this well are using in 2026.

eBPF everywhere: low-cost syscall and kernel-event tracing is standard in production; teams rely on CO-RE bpf programs for portability.
Automated core triage: AI-assisted crash triage that maps stack patterns to known issues and suggests suspects (memory corruption, specific libraries).
CI fault injection: injecting partial faults into CI pipelines with chaos frameworks to catch flaky failure modes before production.
Immutable, symbolized builds: publishing symbolicated binaries and debug info to centralized services so core analysis is reproducible across environments.

Decision tree: Is it external or internal?

Use this quick mental model when you’re triaging a new ‘random’ kill:

Was there an entry in dmesg about OOM or hardware? If yes, treat as external resource problem.
Is there a core dump or stack trace? If yes, analyze core and use sanitizers to reproduce.
Did Kubernetes or systemd log a kill due to failing liveness? If yes, adjust checks and instrument the lifecycle.
Are kills correlated with specific request patterns? If yes, record and replay those exact requests.
If none of the above, use eBPF tracing to capture kernel events and syscalls near the death window.

Real-world example (concise case study)

Scenario: A microservice in production was being randomly killed once per day. No logs indicated a crash; the service simply disappeared and restarted.

Steps taken:

Checked dmesg: multiple OOM killer messages tied to the process PID.
Configured systemd to keep cores and used coredumpctl to collect one core.
Analyzed the core in gdb and saw large allocations in a third-party image-processing library during a rare codepath.
Reproduced locally with stress-ng and crafted input using the recorded request; ASan showed heap overflow in the library.
Patched the library, added memory limit guardrails and an integration test that injects the malformed payload. Deployed and monitored: no further random kills.

Checklist for implementers (copyable)

Enable core dumps and centralize storage.
Record exact input sequences and timing for failing runs.
Use eBPF for low-cost production tracing.
Run sanitizers in CI and use Valgrind for deep memory checks.
Implement graceful watchdogs with backoff policies.
Simulate faults with stress-ng, libfiu, and chaos frameworks in CI.
Automate triage with scripts to extract stacktraces from cores.

Actionable takeaways

Don’t accept randomness: random kills are always diagnosable with core dumps, tracing, and controlled reproduction.
Collect artifacts early: enable core dumps and lightweight traces before incidents recur.
Use modern observability: eBPF and centralized crash stores make post-mortems faster and more reliable in 2026.
Make failures deterministic: record-and-replay, fault injection, and targeted load tests turn flakiness into reproducible bugs.

“Process roulette” is a symptom, not a cause. Treat it as a data-gathering problem: more artifacts, better hypotheses, faster fixes.

Next steps and call-to-action

If you’re fighting intermittent kills right now, start by enabling core dumps and adding a low-overhead eBPF trace for kernel kill events. If you want a ready-made runbook and scripts to automate core capture, stack extraction and upload, download our free debugging toolkit (includes gdb automation scripts, bpftrace snippets, and a CI fault-injection template) to get reproducible results in less than a day.

Get the toolkit and runbook: implement the checklist, run a controlled stress test in your staging environment, and tag an incident for a 1-hour triage session—turn your process roulette into a solved bug.

Unknown

Contributor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.