Find what a process is waiting on with strace
Trace the blocked syscall, follow its file descriptor and identify the thread or peer that must wake it. Use syscall duration rather than summary CPU time to measure waiting.
Start with a short trace of all threads, then identify the syscall that stays unfinished. Follow the descriptor or synchronization address to the code responsible for making progress. A quiet process can be healthy: an event loop waiting for clients looks much like a stalled one until you connect the wait to the application’s expected work.
Trace a running process
Set pid to the process you own and want to inspect:
pid=12345
strace -f -T -r -y -e trace=read,write,futex,epoll_wait,epoll_pwait,poll,ppoll -p "$pid"
-f follows children and, when attaching to a multithreaded process, its threads. -e trace= narrows the calls displayed. -T prints elapsed time inside each call; -r prints relative time between successive syscall entries. Neither is a timestamp for an application request. -y decodes descriptor paths, including pipe identifiers. Stop an attached trace with Ctrl-C to detach.
Attachment depends on ptrace permissions, namespaces and system policy. Launching your own scratch program under strace is often easier than attaching. Tracing changes timing, particularly when a program makes many tiny calls. Keep captures short and treat timings as observations under instrumentation. These flag semantics are documented in the strace manual.
Reproduce a blocked read
Save this as wait.py on Linux:
import os, time
r, w = os.pipe()
if os.fork() == 0:
os.close(r)
time.sleep(0.2)
os.write(w, b"ok\n")
os._exit(0)
os.close(w)
print(os.read(r, 3).decode(), end="")
os.wait()
strace -f -r -T -y -e trace=read,write,wait4,clock_nanosleep python3 wait.py
Example output from strace 6.19, trimmed to the pipe exchange:
[pid 3691455] 0.001112 read(3<pipe:[698422630]> <unfinished ...>
[pid 3691472] 0.000661 clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, {tv_sec=2541196, tv_nsec=806429682}, NULL) = 0 <0.199333>
[pid 3691472] 0.200229 write(4<pipe:[698422630]>, "ok\n", 3) = 3 <0.000085>
[pid 3691455] 0.000213 <... read resumed>, "ok\n", 3) = 3 <0.201025>
The parent’s read takes about 201 ms. The child sleeps for about 199 ms and then writes three bytes to the same pipe identifier. That establishes the causal chain: the parent needs data from its child. The delay is intentional, rather than a slow disk or an unknown lock.
Interpret common waits
| Call | First question |
|---|---|
read |
Which file, pipe or socket is this descriptor? Who writes it? |
futex |
Which application lock, condition variable or runtime queue owns this address? |
epoll_wait |
Are there outstanding requests, or is the event loop correctly idle? |
A futex wait alone does not prove deadlock. Inspect other threads for the owner or producer. A read returning zero means EOF, rather than an ongoing wait. EAGAIN on a nonblocking descriptor means no data is available now; repeated retries can explain a busy loop.
For counts, run strace -f -c python3 wait.py. Its default summary accounts for system CPU time, so a long sleep can rank below file metadata calls. Add -w when you need a wall-clock summary. For a call-site clue, run strace -k -e trace=read python3 wait.py; stack unwinding requires a supported strace build and useful symbols. Strace does not replace an application stack profiler.
Go deeper
The syscall dive follows a blocked read through scheduling and wakeup. Use that model after the trace tells you which resource is missing.