Debugging Procedures

https://wiki.ubuntu.com/DebuggingProcedures

Info for the BugSquad

  1. This will create a minimal Precise system:

    1
    2
    sudo mkdir -p /chroots/precise
    sudo debootstrap precise /chroots/precise/
  2. Now change into this minimal precise system:

    1
    sudo chroot /chroots/precise
  3. Edit /etc/apt/sources/list and add all the repositories you need, including the ddeb repository.

  4. Execute the following in a terminal:

    1
    sudo apt-get update; sudo apt-get install gdb apport
  5. use apport-retrace as you’re used to.

Compiling with debugging -g option

1
2
3
./autogen.sh
./configure CFLAGS="-g -O0" #option A
./configure --enable-debug #option B

strace

1
2
$ strace ./a.out    //察看程序在执行什么系统调用
$ strace -p pid //附到一个已经在跑的进程上,实时观察

mtrace

valgrind

https://wiki.ubuntu.com/Valgrind

Backtrace

https://wiki.ubuntu.com/Backtrace

Generation

  1. Please ensure you have packages with debug symbols installed. You can do this by following the instructions at DebuggingProgramCrash.
  2. Make sure the GNU Debugger is installed.
  3. sudo apt-get install gdb
    Start the program under control of gdb via a terminal (some programs run as root, so one would use sudo gdb instead of just gdb below):
1
2
3
4
gdb <program> 2>&1 | tee ~/gdb-<program>.txt
(gdb) handle SIG33 pass nostop noprint
(gdb) set pagination 0
(gdb) run <arguments, if any>
  1. The program will start. Perform any actions necessary to reproduce the crash. If the program hangs but doesn’t crash you can press ctrl+c in gdb while the program is frozen and then continue with the next step.
    Retrieve a backtrace:
1
2
3
4
5
(gdb) backtrace full
(gdb) info registers
(gdb) x/16i $pc
(gdb) thread apply all backtrace
(gdb) quit

Attach the complete output from GDB, contained in gdb-.txt, in your bug report. You will find the file in your home directory /home//.

察看进程占用IO命令

只显示有I/O行为的进程

$ iotop -oP

查看特定进程 

$ iotop -p $PID

展示I/O统计,每秒更新一次

$ pidstat -d 1

查看该进程启动的完整命令行

$ ps eho command -p $PID

查看该进程启动时候所在的目录

$ readlink /proc/$PID/cwd

查看该进程启动时的完整环境变量

strings -f /proc/$PID/environ | cut -f2 -d ”

列出该进程所打开的所有文件:

$ lsof -p $PID
  cwd代表当前目录,这里是/root
  rtd代表根目录,这里是/
  txt代表执行的程序,这里是/bin/bash
  mem代表映射到内存的文件,这里是/lib/libc-2.7.so等动态链接库
  TYPE一栏表示文件/目录的类型,DIR代表目录,REG代表普通文件,CHR代表字符设备.
列出该进程所打开的网络连接:

$ netstat -pan | grep $PID

查看文件被哪个i进程占用

$ sudo fuser -v path/to/file

$ lsof path/to/file

察看多线程所有线程ID

$ ps -T
$ top -H -p <pid>  //让top输出某个特定进程并检查该进程内运行的线程状况
$ htop

查看程序或进程用到的库

1
2
3
4
$ ldd /path/to/program //可能会直接调用可执行程序来明确其库文件依赖关系,对于不可信的第三方程序来说不安全
$ objdump -p /path/to/program | grep NEEDED
$ sudo pldd pid-of-program
$ sudo pmap pid-of-program

监控文件

audit

inotifywait

inotifymoniter

PS

To print a process tree:

1
2
ps -ejH
ps axjf

To get info about threads:

1
2
ps -eLf
ps axms

To get security info:

1
2
3
ps -eo euser,ruser,suer,fuser,f,comm,label
ps axZ
ps -eM

To see every process running as root (real & effective ID) in user format:

1
ps -U root -u root u

To see every process with a user-defined format:

1
2
3
ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm
ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm
ps -Ao pid,tt,user,fname,tmout,f,wchan

Print only the process IDs of syslogd:

1
ps -C syslogd -o pid=

Print only the name of PID 42:

1
ps -q 42 -o comm=