linux-kernel-module-cheat/gdbserver.md

gdbserver

Step debug userland processes to understand how they are talking to the kernel.

In guest:

/gdbserver.sh /myinsmod.out /hello.ko

In host:

./rungdbserver kernel_module-1.0/user/myinsmod.out

You can find the executable with:

find buildroot/output.x86_64~/build -name myinsmod.out

TODO: automate the path finding:

• using the executable from under buildroot/output.x86_64~/target would be easier as the path is the same as in guest, but unfortunately those executables are stripped to make the guest smaller. BR2_STRIP_none=y should disable stripping, but make the image way larger.

• outputx86_64~/staging/ would be even better than target/ as the docs say that this directory contains binaries before they were stripped. However, only a few binaries are pre-installed there by default, and it seems to be a manual per package thing.

  E.g. pciutils has for lspci:

  define PCIUTILS_INSTALL_STAGING_CMDS
      $(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) $(PCIUTILS_MAKE_OPTS) \
          PREFIX=$(STAGING_DIR)/usr SBINDIR=$(STAGING_DIR)/usr/bin \
          install install-lib install-pcilib
  endef
  

  and the docs describe the *_INSTALL_STAGING per package config, which is normally set for shared library packages.

  Feature request: https://bugs.busybox.net/show_bug.cgi?id=10386

An implementation overview can be found at: https://reverseengineering.stackexchange.com/questions/8829/cross-debugging-for-mips-elf-with-qemu-toolchain/16214#16214

gdbserver different archs

As usual, different archs work with:

./rungdbserver -a arm kernel_module-1.0/user/myinsmod.out

gdbserver BusyBox

BusyBox executables are all symlinks, so if you do on guest:

/gdbserver.sh ls

on host you need:

./rungdbserver busybox-1.26.2/busybox

gdbserver shared libraries

Our setup gives you the rare opportunity to step debug libc and other system libraries e.g. with:

b open
c

Or simply by stepping into calls:

s

This is made possible by the GDB command:

set sysroot ${buildroot_out_dir}/staging

which automatically finds unstripped shared libraries on the host for us.

See also: https://stackoverflow.com/questions/8611194/debugging-shared-libraries-with-gdbserver/45252113#45252113

Debug userland process without gdbserver

QEMU -gdb GDB breakpoints are set on virtual addresses, so you can in theory debug userland processes as well.

• https://stackoverflow.com/questions/26271901/is-it-possible-to-use-gdb-and-qemu-to-debug-linux-user-space-programs-and-kernel
• https://stackoverflow.com/questions/16273614/debug-init-on-qemu-using-gdb

The only use case I can see for this is to debug the init process (and have fun), otherwise, why wouldn't you just use gdbserver? Known limitations of direct userland debugging:

• the kernel might switch context to another process, and you would enter "garbage"

• TODO step into shared libraries. If I attempt to load them explicitly:

  (gdb) sharedlibrary ../../staging/lib/libc.so.0
  No loaded shared libraries match the pattern `../../staging/lib/libc.so.0'.
  

  since GDB does not know that libc is loaded.

Custom init process:

• Shell 1:

  ./run -d -e 'init=/sleep_forever.out' -n
  

• Shell 2:

  ./rungdb-user kernel_module-1.0/user/sleep_forever.out main
  

BusyBox custom init process:

• Shell 1:

  ./run -d -e 'init=/bin/ls' -n
  

• Shell 2:

  ./rungdb-user -h busybox-1.26.2/busybox ls_main
  

This follows BusyBox' convention of calling the main for each executable as <exec>_main since the busybox executable has many "mains".

BusyBox default init process:

• Shell 1:

  ./run -d -n
  

• Shell 2:

  ./rungdb-user -h busybox-1.26.2/busybox init_main
  

This cannot be debugged in another way without modifying the source, or /sbin/init exits early with:

"must be run as PID 1"

Non-init process:

• Shell 1

  ./run -d -n
  

• Shell 2

  ./rungdb-user kernel_module-1.0/user/sleep_forever.out
  Ctrl + C
  b main
  continue
  

• Shell 1

  /sleep_forever.out
  

This is of least reliable setup as there might be other processes that use the given virtual address.