# gdbserver

Step debug userland processes to understand how they are talking to the kernel.

In guest:

    /gdbserver.sh /myinsmod.out /hello.ko

In host:

    ./rungdbserver kernel_module-1.0/user/myinsmod.out

You can find the executable with:

    find buildroot/output.x86_64~/build -name myinsmod.out

TODO: automate the path finding:

-   using the executable from under `buildroot/output.x86_64~/target` would be easier as the path is the same as in guest, but unfortunately those executables are stripped to make the guest smaller. `BR2_STRIP_none=y` should disable stripping, but make the image way larger.

-   `outputx86_64~/staging/` would be even better than `target/` as the docs say that this directory contains binaries before they were stripped. However, only a few binaries are pre-installed there by default, and it seems to be a manual per package thing.

    E.g. `pciutils` has for `lspci`:

        define PCIUTILS_INSTALL_STAGING_CMDS
            $(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) $(PCIUTILS_MAKE_OPTS) \
                PREFIX=$(STAGING_DIR)/usr SBINDIR=$(STAGING_DIR)/usr/bin \
                install install-lib install-pcilib
        endef

    and the docs describe the `*_INSTALL_STAGING` per package config, which is normally set for shared library packages.

    Feature request: <https://bugs.busybox.net/show_bug.cgi?id=10386>

An implementation overview can be found at: <https://reverseengineering.stackexchange.com/questions/8829/cross-debugging-for-mips-elf-with-qemu-toolchain/16214#16214>

## gdbserver different archs

As usual, different archs work with:

    ./rungdbserver -a arm kernel_module-1.0/user/myinsmod.out

## gdbserver BusyBox

BusyBox executables are all symlinks, so if you do on guest:

    /gdbserver.sh ls

on host you need:

    ./rungdbserver busybox-1.26.2/busybox

## gdbserver shared libraries

Our setup gives you the rare opportunity to step debug libc and other system libraries e.g. with:

    b open
    c

Or simply by stepping into calls:

    s

This is made possible by the GDB command:

    set sysroot ${buildroot_out_dir}/staging

which automatically finds unstripped shared libraries on the host for us.

See also: <https://stackoverflow.com/questions/8611194/debugging-shared-libraries-with-gdbserver/45252113#45252113>

## Debug userland process without gdbserver

QEMU `-gdb` GDB breakpoints are set on virtual addresses, so you can in theory debug userland processes as well.

- <https://stackoverflow.com/questions/26271901/is-it-possible-to-use-gdb-and-qemu-to-debug-linux-user-space-programs-and-kernel>
- <https://stackoverflow.com/questions/16273614/debug-init-on-qemu-using-gdb>

The only use case I can see for this is to debug the init process (and have fun), otherwise, why wouldn't you just use `gdbserver`? Known limitations of direct userland debugging:

-   the kernel might switch context to another process, and you would enter "garbage"

-   TODO step into shared libraries. If I attempt to load them explicitly:

        (gdb) sharedlibrary ../../staging/lib/libc.so.0
        No loaded shared libraries match the pattern `../../staging/lib/libc.so.0'.

    since GDB does not know that libc is loaded.

Custom init process:

-   Shell 1:

        ./run -d -e 'init=/sleep_forever.out' -n

-   Shell 2:

        ./rungdb-user kernel_module-1.0/user/sleep_forever.out main

BusyBox custom init process:

-   Shell 1:

        ./run -d -e 'init=/bin/ls' -n

-   Shell 2:

        ./rungdb-user -h busybox-1.26.2/busybox ls_main

This follows BusyBox' convention of calling the main for each executable as `<exec>_main` since the `busybox` executable has many "mains".

BusyBox default init process:

-   Shell 1:

        ./run -d -n

-   Shell 2:

        ./rungdb-user -h busybox-1.26.2/busybox init_main

This cannot be debugged in another way without modifying the source, or `/sbin/init` exits early with:

    "must be run as PID 1"

Non-init process:

-   Shell 1

        ./run -d -n

-   Shell 2

        ./rungdb-user kernel_module-1.0/user/sleep_forever.out
        Ctrl + C
        b main
        continue

-   Shell 1

        /sleep_forever.out

This is of least reliable setup as there might be other processes that use the given virtual address.