MacOS Two-machine Kernel Debugging
If you are a macOS security researcher, chances are that at some point you’ll have to perform kernel-level code debugging. While there are many great and helpful blog posts on topic of macOS kernel debugging, the fact is that at this point they are slightly outdated and mostly focused on using virtual machines.
The focus of this blog post is to describe how to perform two-machine kernel debugging on newer devices including Apple Silicon and the latest macOS version, which at the time of writing is macOS Monterey 12.2.1.
The setup described in this blog post includes an:
Intel-based host device
Intel-based and Apple Silicon target devices
Intel-based devices are models released after 2020 that have only Thunderbolt 3/Thunderbolt 4/USB-C support, and the same applies for Apple Silicon.
The Kernel Debug Kit for macOS (KDK_ReadMe), used in this blog post as an official reference guide, describes that:
The target device is the Mac that runs the code you want to debug
The host device is the Mac that runs the debugger
Additionally, you cannot perform two-machine debugging using neither USB Ethernet adapters nor wireless networking on any Mac. You must connect the host and a target device to the same network, but there are no other restrictions on how these devices are connected to that network.
This means that we can use an ethernet cable and a total of four additional adapters:
- Two Thunderbolt 3 (USB-C) to Thunderbolt 2 Adapters
Figure 1 - Thunderbolt 3 (USB-C) to Thunderbolt 2 Adapter
- Two Thunderbolt to Gigabit Ethernet Adapters
Figure 2 - Thunderbolt to Gigabit Ethernet Adapter
When configuring macOS for kernel debugging, the first thing to do is to identify the build version by running the sw_vers command in terminal:
The build version is important to download the appropriate version of Kernel Debug Kit (KDK) from the Apple’s Developer Downloads site. For example, in this setup the appropriate KDK file is Kernel_Debug_Kit_12.1_build_21C52.
Once installed, it will be located under /Library/Developer/KDKs/KDK_12.1_21C52.kdk/. Inside the mentioned directory there is a KDK_ReadMe file which contains all the necessary information regarding the macOS kernel debugging setup and configuration, as well as a System directory with different kernel and kernel extension variants:
For two-machine kernel debugging it is necessary to:
install the KDK on both the host and target devices
configure the target devices
To configure an Intel-based Mac as a host device:
Install the appropriate version of KDK
Download and install Xcode 12.5.1
If a newer version of Xcode already exists on the system, rename the Xcode 12 and adjust the path accordingly
Set defaults to use Python 2 in Xcode 12 LLDB
defaults write com.apple.dt.lldb DefaultPythonVersion 2
Use the Xcode 12 LLDB version for kernel debugging
Additionally, to perform kernel-level debugging for Apple silicon devices, a core dump server should be configured by creating a directory, configuring the required permissions and loading the kdumpd Launch Daemon:
chmod 1777 /var/tmp/PanicDumps
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.kdumpd.plist
The last step in the configuration is to identify the correct ethernet device (en0, en1, etc.) by running the ifconfig command.
It is recommended to assign both the host and the target machine’s ethernet devices with a static IP address for simpler configuration, even when directly connecting the devices via an ethernet cable, especially if this is your first time configuring two-machine kernel-level debugging.
To configure an Intel-based Mac as a target device:
Disable System Integrity Protection (SIP) from the Recovery Mode
Launch a terminal and execute the csrutil disable command
Execute the csrutil authenticated-root disable command (required when using other kernel variants)
Set the Secure Boot policy to “Medium Security”
Reboot the Mac
Finally, the boot arguments should be configured properly. For example:
sudo nvram boot-args="debug=0x44 kdp_match_name=enX wdt=-1"
From the KDK_ReadMe file*:*
debug=0x44 – Tells the kernel to wait for a debugger to attach to the device when the kernel receives a non-maskable interrupt (NMI)
kdp_match_name=enX - Set the value of this key to the Ethernet device you will use (en0, en1, etc.)
wdt=-1 (minus one) – Disables watchdog monitoring
Additional boot-args can be configured if needed.
When using other kernel variants, several steps must be taken:
- Execute the mount command in Terminal to identify the device mounted at “/” and remove the final “sX” from the device name to get the actual name. For example:
/dev/disk1s5s1 on / (apfs, sealed, local, read-only, journaled)
devfs on /dev (devfs, local, nobrowse)
- Mount a live version of the system using the following commands:
sudo mount -o nobrowse -t apfs /dev/disk1s5 /Users/<USERNAME>/livemount (note the previously identified device name)
- Add kernel variant files to the newly mounted disk by copying the entire /System directory of the KDK into the /System directory of your mounted disk:
sudo ditto /Library/Developer/KDKs/<KDK Version>/System/Users/<USERNAME>/livemount/System
- Rebuild the kernel collections for the variants you added to your mounted disk and “bless” them to authorize booting from your modified kernels:
sudo kmutil install –volume-root /Users/<USERNAME>/livemount --update-all
sudo bless --mount /Users/<USERNAME>/livemount –bootefi –create-snapshot
To boot the desired kernel variant, it is necessary to update the boot arguments by specifying the desired kernel variant with the kcsuffix argument:
sudo nvram boot-args="debug=0x44 kdp_match_name=enX wdt=-1 kcsuffix=development"
To verify the configuration, reboot the device and execute the following command in the Terminal:
kern.osbuildconfig: development (the result will depend on the variant used)
When configuring kernel debugging using a two-machine setup note the following:
Be careful when typing in the paths
Make sure that the mount device and ethernet device names are correct
With Apple silicon, the situation is different and the KDK_ReadMe states the following:
“Apple silicon doesn’t support active kernel debugging. You may inspect the current state of the kernel when it is halted due to a panic or NMI. However, you cannot set breakpoints, continue code execution, step into code, step over code, or step out of the current instruction.”
“Apple silicon doesn’t support installing the kernel and kernel extension variants from the KDK.”
It is, however, possible to inspect a kernel panic by capturing the core dumps over the network by configuring additional boot-args:
debug=0xc44– Creates a core dump when a panic or NMI occurs
_panicd_ip=<CORE_DUMP_SERVER_IP>– Sets the IP address of the core dump server
To configure the Apple silicon as a target device:
Disable System Integrity Protection (SIP) from Recovery Mode
Launch a terminal and execute the
Reboot the Mac
Lastly, with SIP disabled, configure the boot arguments using the following command:
sudo nvram boot-args="debug=0xc44 kdp_match_name=enX wdt=-1 _panicd_ip=<CORE_DUMP_SERVER_IP>"
The setup used in this blog post is shown in the figure below:
Figure 3 - Two-machine macOS kernel debugging setup
macOS supports remote kernel debugging by implementing the Kernel Debugging Protocol (KDP), a UDP based client-server protocol natively supported in LLDB.
As previously mentioned, the LLDB used will be the one installed with Xcode 12, and it can be launched by running the following command from a Terminal on the host machine:
Next, kernel-code debugging can be performed in two situations:
If the target device panics
If you trigger an NMI on the target device, which is the case described in this blog post
Figure 4 – Triggering Non-Mackable Interrupt
On Apple silicon, if a NMI is triggered, a core dump will be generated and sent to core dump server (it can take a while to generate a core dump). The core dump can then be found in the previously set /var/tmp/PanicDumps directory and loaded in LLDB once unzipped, as shown in the following figure:
Figure 5 - Apple silicon core dump loaded in lldb
Similarly, if a NMI is triggered on an Intel-based Mac with development kernel variant, a core dump can be generated and sent to the core dump server by running following commands from LLDB:
(lldb) target create /Library/Developer/KDKs/KDK_12.1_21C52.kdk/System/Library/Kernels/kernel.development
(lldb) kdp-remote IP_ADDRESS_OF_THE_TARGET_DEVICE
Figure 6 - Kernel-level debugging using KDP on Intel-based MAC
It is recommended to execute scripts as suggested by LLDB.
Once you are finished with a debugging session, refer to the the KDK_ReadMe as a resource for information on how to restore the original Mac configuration as well as default system security settings.
Given a fact that you can’t debug macOS kernel using your own machine, two machine macOS kernel debugging can look intimidating at first, especially since it requires two machines and several adapters. However, once configured properly it is a straight-forward process as described in this blog post. Best practice is not to use production Macs to perform kernel debugging but to use a machine for testing purposes. Also, monitor the KDK_ReadMe for any changes and use it as a reference guide.