Jex’s Note

Debugging Tools

strace

Installation

$ sudo apt install strace   #Debian/Ubuntu
# yum install strace        #RHEL/CentOS
# dnf install strace        #Fedora 22+

Trace linux command

Simply run a command with strace, tracing of all system calls, e.g. df -h.

sudo strace df -h

Trace PID

sudo strace -p 5206

Get summary of process

Generate a report of total time, calls, and errors for each system call.

sudo strace -c -p 5206

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 90.41    0.001141         163         7           futex
  1.82    0.000023           4         6           write
  ...
  0.00    0.000000           0         2           gettimeofday
  0.00    0.000000           0         1           sendmmsg
------ ----------- ----------- --------- --------- ----------------
100.00    0.001262                    80         1 total

Print command time spent in system calls

sudo strace -T ls

open("/proc/filesystems", O_RDONLY)     = 3 <0.000024>
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 <0.000006>

Trace only specific system calls

sudo strace -e trace=write df -h
sudo strace -e trace=open,close df -h
sudo strace -e trace=open,close,read,write df -h
sudo strace -e trace=all df -h

qualifier: signal, abbrev, verbose, raw, read, or write

Trace system calls based on a certain condition

Trace all system calls involving process management.

sudo strace -e trace=process ls

execve("/bin/ls", ["ls"], [/* 17 vars */]) = 0
arch_prctl(ARCH_SET_FS, 0x7fb0a721f840) = 0
astra-worker  conf  jobctl  library  logs  main.go  README.md  service  test  topic  utility  vendor
...

Trace all system calls that take a filename as an argument

sudo strace -e trace=file ls

execve("/bin/ls", ["ls"], [/* 17 vars */]) = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
...

Trace all system calls involving memory mapping, type.

sudo strace -e trace=memory ls

brk(0)                                  = 0x62b000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9c0b0a1000
mmap(NULL, 26186, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f9c0b09a000
...

Trace all network related system calls

sudo strace -e trace=network curl google.com

socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
...

Trace all signals related system calls

sudo strace -e trace=signal -p 5206

--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=16988, si_uid=0} ---             // `sudo kill 5206` trigger this line
+++ exited with 0 +++

Other arguments

Help

sudo strace -h ls

Print instruction pointer during system call

sudo strace -i ls

[00007f2d6f9f81e0] openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3

Show time for each output line

sudo strace -t ls

01:37:47 getdents(3, /* 16 entries */, 32768) = 464

Redirect output to file

sudo strace -o /tmp/ls.log ls

Show debugging information for strace tool

sudo strace -d ls

TODO

System call

...
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
...

type of system call(argument) = return value

  • epoll_wait(5, {}, 128, 0) = 0
  • clock_gettime(CLOCK_MONOTONIC, {1696036, 90909291}) = 0
  • futex(0x1178818, FUTEX_WAKE, 1) = 1

ref:

Comments