> For the complete documentation index, see [llms.txt](https://ghostinthehive.gitbook.io/ghostinthehive-as-a-ghost-in-the-hive/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ghostinthehive.gitbook.io/ghostinthehive-as-a-ghost-in-the-hive/debugging-the-linux-kernel-with-qemu-and-gdb.md).

# Debugging the Linux Kernel with Qemu and GDB

This tutorial will walk you through the steps needed to start debugging the Linux Kernel with a setup using `qemu` and `gdb`. If you want to know more about `qemu`, check the [Official Repository](https://github.com/qemu/qemu) definitely one of the best open source projects out there.

This Setup gives you the capability to remotely debug a `qemu` instance emulating The Linux Kernel, you can also debug Kernel Modules with this setup *but we will not get into it in this tutorial.*

**Video Tutorial:**

{% embed url="<https://www.youtube.com/watch?lc=Ugy4QbPJCGutXh8G2q54AaABAg&v=2VcA5Wj7IvU>" %}

***

## <mark style="color:red;">Build your own Custom Kernel:</mark>

To start we need to build our own kernel to have the executable binary image `bzImage` so we can emulate it with `qemu` , and the Kernel File Image `vmlinux` with all debug info we need, to use it with `gdb` to trace through the Kernel Code. Both of these files are obtained when we compile the Linux Kernel from source.

{% hint style="warning" %}
Building the Kernel from scratch takes time **AND** space, so if you want to boot into your newly compiled kernel, when setting up your **VM** make sure to manually partition your disk and give `/boot` partition not less than **`10GB`** to be able to boot the newly build kernel. <mark style="color:red;">**THIS IS VERY IMPORTANT!**</mark>
{% endhint %}

Detailed Steps for Building the Linux Kernel can be found in [**kernelnewbies.org**](https://kernelnewbies.org/KernelBuild)

### Compilation Steps:

```bash
# after cloning the kernel tree from git, cd into <kernel-dir> 
# copy your distro's kernel config file into the cloned <kernel-dir>

cd <kernel-dir>
cp /boot/config-`uname -r` <kernel-dir>/config

# editing the custom kernel config
# you can choose your same kernel config you have by executing
make olddefconfig      # enough for what we need

# or you can tinker with the new config, enable different built-in modules ..etc
make menuconfig
```

**Use navigation to select drivers you want to build, and hit save.**

<figure><img src="/files/s1E4qWxGbM4HPSrCAEUH" alt=""><figcaption></figcaption></figure>

We can edit the config file itself using a text editor and enable enough kernel config for debugging, I will not debug Kernel Modules in this tutorial, so as I said I would not bother build them.

For what we need, make sure to enable these configs:

```
CONFIG_DEBUG_INFO=y
CONFIG_GDB_SCRIPS=y
CONFIG_DBUG_KERNEL=y
```

There are other compiler configs to be enabled, but that is enough for our purposes.

<figure><img src="/files/3pj54dLQhFs0fbVcakBx" alt=""><figcaption></figcaption></figure>

Save the changes and build the Kernel:

```bash
sudo make -j3
sudo make -j3 modules_install 

# if you want concurrency, make sure to use less cores than you have
# specially if you're running a VM
`nproc`  # to know how many cores you have  
```

After finishing, navigate to `<kernel-dir>/arch/x86/boot` you'll find the compiled kernel executable image `bzImage` this is what we need `qemu` to boot, it is a **BIG** image.

Now we have our compiled kernel, we can now restart our system, boot in the new kernel and hop into the next step.

### Making a RAM disk image:

Now before generating a **RAM disk image**, we will stop a second to understand who the Linux system boots, first we have the `bzImage` → the kernel as an executable binary, yet in order for the kernel to boot it needs an **initial root filesystem** `initramfs` to get stuff going and setup correctly initial binaries for the real system to work, like the `init` process aka *the parent of all processes* , and a bunch of other important binaries in `/sbin` , when the `initramfs` is done executing crucial binaries as root, it then looks for the **REAL** root filesystem to mount and pivot the root to it, free itself from memory, and the full system boots. but if it did not find the **REAL** root filesystem, it will boot the kernel and throw you in a recovery shell to boot it manually, or figure out what went wrong.

Have you experienced this lately?? 😉

So we have three components for booting a Linux system:

**→ Kernel Image:** `bzImage` OR `vmlinuz`

**→ RAM disk:** `initrd`

**→ Root file system:** `/dev/sdY`

but since we are not interested into making a full blown Linux system, we can stop at `initramfs` being the root filesystem, and not provide a root filesystem/device to `qemu` . as we will see everything will work fine with just an `initramfs` shell.

Making a **RAM** disk depends on your distro, since I'm running Debian for this setup, this can be done by:

```bash
mkinitramfs -o ramdisk.img

file ramdisk.img
# ramdisk.img: gzip compressed data ... original size 26352128
```

Now we have what we need, Let's move on to compile `qemu`

***

## <mark style="color:red;">Building</mark> <mark style="color:red;">`qemu`</mark> <mark style="color:red;">from source:</mark>

I like building big software from source, because I can choose a minimal setup for just what I need, `qemu` is an emulation software and has a dozen of target systems to emulate, we don't need all that, we are just interested in a `x86_64 bit` Linux system.

Detailed steps for building `qemu` can be found [**here**](https://en.wikibooks.org/wiki/QEMU/Installing_QEMU) .

```bash
# after cloning the repo, cd into qemu
git clone git://git.qemu-project.org/qemu.git
cd qemu
mkdir build && cd build

# configure target system we need / and other options
../configure --target-list=x86_64-softmmu --enable-debug

# build
make -j3 
```

Although `qemu` has tons of options to enable, like enabling **`usb passthrough`** by using compiler options like `-libusb` and configuring the graphic options and multiple screens ...etc, we **ONLY** want to emulate and boot the Linux Kernel. So .. practicing **Minimalism.** 🤟🏻

Now we've successfully built `qemu` and can use it by executing `qemu-system-x86_64` .

***

## <mark style="color:red;">Setting up the Environment:</mark>

### Booting the Linux Kernel in `qemu`:

As a starter let's test booting the kernel:

for `qemu` to boot the Linux kernel it needs two parameters, `-kernel <path-to-kernel-bzImage>` and `-initrd <path-to-ramdik.img>` , `-m 512` for memory and that is **VERY** **MUCH** enough.

```bash
qemu-system-x86_64 -kernel <kernel-dir>/arch/x86_64/boot/bzImage \
-initrd <path-to>/ramdisk.img \
- m 512
```

<figure><img src="/files/Xj5Cu89tsMwNrfScbboM" alt=""><figcaption></figcaption></figure>

And as we see, we've been thrown to an `initramfs` recovery shell with very limited functionality.

Now let's connect the `qemu` instance with `gdb` for remote debugging.

Since `gdb` is adopted everywhere, almost all important projects/software will have a `GDBStub` for debugging, in our case this task is done simply by adding `-s` to the `qemu` script.

```bash
qemu-system-x86_64 -kernel <kernel-dir>/arch/x86_64/boot/bzImage \
-initrd <path-to>/ramdisk.img \
- m 512 -s
```

<figure><img src="/files/RLBeou8mJUHKXKM1PCAD" alt=""><figcaption></figcaption></figure>

That might look like as if we did nothing, because the kernel booted exactly the same as before. but if we connected with `gdb` to port `1234` as `qemu` uses this port for `gdb` remote debugging.

<figure><img src="/files/zmCC8bR3KREDt8AnhQmP" alt=""><figcaption></figcaption></figure>

We see that `gdb` connected successfully, but we couldn't catch anything. and if we closed the `qemu` instance, `gdb` will complain about a remote communication error.

<figure><img src="/files/9zrevPkqOF2Lsx679SKH" alt=""><figcaption></figcaption></figure>

So we need to tell `qemu` to **STOP** booting until connected with `gdb` , simple as in a `-S` option to add to the `qemu` script.

```bash
qemu-system-x86_64 -kernel <kernel-dir>/arch/x86_64/boot/bzImage \
-initrd <path-to>/ramdisk.img \
- m 512 -s -S
```

<figure><img src="/files/QWpZyoLTWDAp3d3Nvugl" alt=""><figcaption></figcaption></figure>

Now we have the kernel waiting for us to connect remotely over port `1234` with `gdb` .

Let's make it cooler and redirect the `qemu` output to the main console window with a serial port terminal by adding a `-nographic` option to not view the `qemu` window and `-append "console=ttyS0"` to the `qemu` script, so we can scroll though the kernel log.

```bash
qemu-system-x86_64 -kernel <kernel-dir>/arch/x86_64/boot/bzImage \
-initrd <path-to>/ramdisk.img \
-m 512 -s -S \ 
-append "console=ttyS0"
```

<figure><img src="/files/oHJhSAHc3LLQRSk9OYXs" alt=""><figcaption></figcaption></figure>

### Connecting `qemu` kernel instance remotely with `gdb`:

To be able to debug the Linux Kernel with `gdb` we need the Linux Kernel symbols to be able to trace through the kernel Code, Lucky for us since we've compiled the Linux Kernel ourselves, if we navigate to the compiled `<kernel-dir>` , we'd find a `vmlinux` which is yet another Linux Kernel Image File, but this file is statically linked, containing all `debug_info` and The Linux Kernel Symbols we need, So this is the Linux Kernel File we need to attach to `gdb` to load the Kernel symbols.

```bash
gdb <kernel-dir>/vmlinux
```

<figure><img src="/files/JCIzVL8czUlLRPZer9qn" alt=""><figcaption></figcaption></figure>

Now that the Linux Kernel symbols are loaded in `gdb`, and we have the Kernel `qemu` instance waiting for the `gdb` connection, we can connect with `target remote :1234` .

<figure><img src="/files/pA2u9cdI7YqD1ny5u9Xf" alt=""><figcaption></figcaption></figure>

Now we are connected to the `qemu` instance via its `GDBStub` and can walk through the `main.c` code and start exploring the Linux Kernel.

***

## <mark style="color:red;">Debugging the Linux Kernel:</mark>

Let's start by setting up a hardware breakpoint at `start_kernel()` and hit continue to remotely control booting the kernel.

<figure><img src="/files/CnOSjM7CfsvHoBT1yrDq" alt=""><figcaption></figcaption></figure>

Now that doesn't seem like we control booting the kernel at all, because the kernel actually booted \_ *we entered the `initramfs` recovery shell* \_ and our breakpoint was not hit.

That's because we need to disable the kernel **ASLR** by adding `nokaslr` to the `qemu` script.

```bash
qemu-system-x86_64 -kernel <kernel-dir>/arch/x86_64/boot/bzImage \
-initrd <path-to>/ramdisk.img \
-m 512 -s -S \ 
-append "console=ttyS0 nokaslr"
```

<figure><img src="/files/x8jLpcicEFIKuwmUHIf5" alt=""><figcaption></figcaption></figure>

**VOILA!** now we actually control booting the kernel, from there you're free to walk through the code, learn the Linux Kernel by debugging, view and list the disassembly as well as the source code.

<figure><img src="/files/QVK3P5wG5OL0z0rgyeFZ" alt=""><figcaption></figcaption></figure>

Compiling the Linux Kernel might be a tough job, yet it comes with its pros as there is a directory in your `<kernel-dir>` that has tons of helper scripts including `gdb` scripts you can add to your `.gdbinit` that's very much useful for debugging Kernel Modules.

you can view them in `gdb` after adding the path to the script in `.gdbinit` buy typing `lx-` and hitting TAB .

```bash
# add path to the linux kernel gdb script for safe auto loading 
 
echo "add-auto-load-safe-path <kernel-dir>/vmlinux-gdb.py" >> ~/.gdbinit
```

<figure><img src="/files/cNhY8xulLkVs0w8TmTOR" alt=""><figcaption></figcaption></figure>

And that's debugging the Linux Kernel with `gdb` and `qemu` for you!, I hope you had fun going this far! If you found this setup interesting, maybe you can try out debugging the Linux Kernel with `virtualbox` and the `GDBStub` !!