Debugging the Linux Kernel with Qemu and GDB
Last updated
Last updated
This tutorial will walk you through the steps needed to start debugging the Linux Kernel with a setup using qemu and gdb. If you want to know more about qemu, check the Official Repository definitely one of the best open source projects out there.
This Setup gives you the capability to remotely debug a qemu instance emulating The Linux Kernel, you can also debug Kernel Modules with this setup but we will not get into it in this tutorial.
Video Tutorial:
To start we need to build our own kernel to have the executable binary image bzImage so we can emulate it with qemu , and the Kernel File Image vmlinux with all debug info we need, to use it with gdb to trace through the Kernel Code. Both of these files are obtained when we compile the Linux Kernel from source.
Building the Kernel from scratch takes time AND space, so if you want to boot into your newly compiled kernel, when setting up your VM make sure to manually partition your disk and give /boot partition not less than 10GB to be able to boot the newly build kernel. THIS IS VERY IMPORTANT!
Detailed Steps for Building the Linux Kernel can be found in kernelnewbies.org
Use navigation to select drivers you want to build, and hit save.
We can edit the config file itself using a text editor and enable enough kernel config for debugging, I will not debug Kernel Modules in this tutorial, so as I said I would not bother build them.
For what we need, make sure to enable these configs:
There are other compiler configs to be enabled, but that is enough for our purposes.
Save the changes and build the Kernel:
After finishing, navigate to <kernel-dir>/arch/x86/boot you'll find the compiled kernel executable image bzImage this is what we need qemu to boot, it is a BIG image.
Now we have our compiled kernel, we can now restart our system, boot in the new kernel and hop into the next step.
Now before generating a RAM disk image, we will stop a second to understand who the Linux system boots, first we have the bzImage → the kernel as an executable binary, yet in order for the kernel to boot it needs an initial root filesystem initramfs to get stuff going and setup correctly initial binaries for the real system to work, like the init process aka the parent of all processes , and a bunch of other important binaries in /sbin , when the initramfs is done executing crucial binaries as root, it then looks for the REAL root filesystem to mount and pivot the root to it, free itself from memory, and the full system boots. but if it did not find the REAL root filesystem, it will boot the kernel and throw you in a recovery shell to boot it manually, or figure out what went wrong.
Have you experienced this lately?? 😉
So we have three components for booting a Linux system:
→ Kernel Image: bzImage OR vmlinuz
→ RAM disk: initrd
→ Root file system: /dev/sdY
but since we are not interested into making a full blown Linux system, we can stop at initramfs being the root filesystem, and not provide a root filesystem/device to qemu . as we will see everything will work fine with just an initramfs shell.
Making a RAM disk depends on your distro, since I'm running Debian for this setup, this can be done by:
Now we have what we need, Let's move on to compile qemu
qemu from source:I like building big software from source, because I can choose a minimal setup for just what I need, qemu is an emulation software and has a dozen of target systems to emulate, we don't need all that, we are just interested in a x86_64 bit Linux system.
Detailed steps for building qemu can be found here .
Although qemu has tons of options to enable, like enabling usb passthrough by using compiler options like -libusb and configuring the graphic options and multiple screens ...etc, we ONLY want to emulate and boot the Linux Kernel. So .. practicing Minimalism. 🤟🏻
Now we've successfully built qemu and can use it by executing qemu-system-x86_64 .
qemu:As a starter let's test booting the kernel:
for qemu to boot the Linux kernel it needs two parameters, -kernel <path-to-kernel-bzImage> and -initrd <path-to-ramdik.img> , -m 512 for memory and that is VERY MUCH enough.
And as we see, we've been thrown to an initramfs recovery shell with very limited functionality.
Now let's connect the qemu instance with gdb for remote debugging.
Since gdb is adopted everywhere, almost all important projects/software will have a GDBStub for debugging, in our case this task is done simply by adding -s to the qemu script.
That might look like as if we did nothing, because the kernel booted exactly the same as before. but if we connected with gdb to port 1234 as qemu uses this port for gdb remote debugging.
We see that gdb connected successfully, but we couldn't catch anything. and if we closed the qemu instance, gdb will complain about a remote communication error.
So we need to tell qemu to STOP booting until connected with gdb , simple as in a -S option to add to the qemu script.
Now we have the kernel waiting for us to connect remotely over port 1234 with gdb .
Let's make it cooler and redirect the qemu output to the main console window with a serial port terminal by adding a -nographic option to not view the qemu window and -append "console=ttyS0" to the qemu script, so we can scroll though the kernel log.
qemu kernel instance remotely with gdb:To be able to debug the Linux Kernel with gdb we need the Linux Kernel symbols to be able to trace through the kernel Code, Lucky for us since we've compiled the Linux Kernel ourselves, if we navigate to the compiled <kernel-dir> , we'd find a vmlinux which is yet another Linux Kernel Image File, but this file is statically linked, containing all debug_info and The Linux Kernel Symbols we need, So this is the Linux Kernel File we need to attach to gdb to load the Kernel symbols.
Now that the Linux Kernel symbols are loaded in gdb, and we have the Kernel qemu instance waiting for the gdb connection, we can connect with target remote :1234 .
Now we are connected to the qemu instance via its GDBStub and can walk through the main.c code and start exploring the Linux Kernel.
Let's start by setting up a hardware breakpoint at start_kernel() and hit continue to remotely control booting the kernel.
Now that doesn't seem like we control booting the kernel at all, because the kernel actually booted _ we entered the initramfs recovery shell _ and our breakpoint was not hit.
That's because we need to disable the kernel ASLR by adding nokaslr to the qemu script.
VOILA! now we actually control booting the kernel, from there you're free to walk through the code, learn the Linux Kernel by debugging, view and list the disassembly as well as the source code.
Compiling the Linux Kernel might be a tough job, yet it comes with its pros as there is a directory in your <kernel-dir> that has tons of helper scripts including gdb scripts you can add to your .gdbinit that's very much useful for debugging Kernel Modules.
you can view them in gdb after adding the path to the script in .gdbinit buy typing lx- and hitting TAB .
And that's debugging the Linux Kernel with gdb and qemu for you!, I hope you had fun going this far! If you found this setup interesting, maybe you can try out debugging the Linux Kernel with virtualbox and the GDBStub !!
