Building the XNU kernel on Mac OS X Sierra (10.12.X)

2017-06-17 21:46:00 +0200 - Written by Mikal Villa

Introduction to XNU compiling

From version to version, I always love to play around with the kernel. And it has always been a great lack in guides and documentation on how to build Mac OSX’s kernel, XNU. For those of you that already have tried compiling XNU for Mac OSX 10.12 (Sierra), you probably noticed that earlier build guides like ssen’s blog - Building xnu for OS X 10.11 El Capitan don’t work anymore. However, many thanks to ssen to put in time to write a guide.

The problem is that Apple introduced something named Circular dependency with the libdispatch library and the kernel headers. So the order of the build process just got really important.

If you tried compiling yourself, the header firehose_buffer_private.h might be quite familiar with some errors you hit on the road to success. This is a header from the libdispatch library which forces you to build libdispatch first. Copying the header won’t really help, since you need a static library as well ( libfirehose_kernel.a). Just to make it a little more hassle, the libdispatch source won’t even be close to compiling before the XNU kernel headers are in place.

Get the sources

You need to download in total five sources, including XNU itself. This is version numbers for Mac OSX 10.12.4, which you find at Apple downloads

curl -O https://opensource.apple.com/tarballs/dtrace/dtrace-209.50.12.tar.gz && \
curl -O https://opensource.apple.com/tarballs/AvailabilityVersions/AvailabilityVersions-26.50.4.tar.gz && \
curl -O https://opensource.apple.com/tarballs/libdispatch/libdispatch-703.50.37.tar.gz && \
curl -O https://opensource.apple.com/tarballs/libplatform/libplatform-126.50.8.tar.gz

# To use the XNU package do
curl -O https://opensource.apple.com/tarballs/xnu/xnu-3789.51.2.tar.gz
export XNUPATH=$(pwd)/xnu-3789.51.2

# (Note that using github is experimental, if you break anything, it's not my fault.)
# To use github do
git clone https://github.com/opensource-apple/xnu.git
export XNUPATH=$(pwd)/xnu

To extract all files at once, you can run for file in *.tar.gz; do tar -zxf $file; done && rm -f *.tar.gz. The guide will assume you managed to extract the tar files from now on.

Building time

First of all, make sure you got Xcode 8, try to have the last version to encrease the chance it will build without too many errors. Next up you need to make sure you have build tools from command line, you can check this by running clang -v. If the output is something like zsh: command not found: clang then you need to install the command line tools. If you followed my instructions and got Xcode 8, this can be easily installed with the command xcode-select --install.

Next up is to ensure the SDK is correctly set. A good way to check if your SDK path is set correctly is to run xcrun -sdk macosx -show-sdk-path and check the output.

By the way, if you strugle with this, kernel compiling might be dangerous for your data, since this can break your OS install. At least, test it in a virtual machine.

dtrace

Dtrace is a fantastic debugging tool which you should really lookup if you haven’t yet. It’s worth your time. I also wondered why FreeBSD, Solaris and Darwin (OSX) had dtrace in official source but not Linux, and it seems to be because of some stupid licensing issues.

cd dtrace-209.50.12
mkdir -p obj sym dst
xcodebuild install -target ctfconvert -target ctfdump -target ctfmerge ARCHS="x86_64" SRCROOT=$PWD OBJROOT=$PWD/obj SYMROOT=$PWD/sym DSTROOT=$PWD/dst
sudo ditto $PWD/dst/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain

AvailabilityVersions

This is just a perl script that’s required to build the kernel.

cd ../AvailabilityVersions-26.50.4/
mkdir -p dst
make install SRCROOT=$PWD DSTROOT=$PWD/dst
sudo ditto $PWD/dst/usr/local `xcrun -sdk macosx -show-sdk-path`/usr/local

XNU - Header install

You will need the XNU headers installed to the SDK directory to be able to build the libdispatch library.

cd $XNUPATH
mkdir -p BUILD.hdrs/obj BUILD.hdrs/sym BUILD.hdrs/dst
make installhdrs SDKROOT=macosx ARCH_CONFIGS=X86_64 SRCROOT=$PWD OBJROOT=$PWD/BUILD.hdrs/obj SYMROOT=$PWD/BUILD.hdrs/sym DSTROOT=$PWD/BUILD.hdrs/dst
sudo xcodebuild installhdrs -project libsyscall/Libsyscall.xcodeproj -sdk macosx ARCHS='x86_64 i386' SRCROOT=$PWD/libsyscall OBJROOT=$PWD/BUILD.hdrs/obj SYMROOT=$PWD/BUILD.hdrs/sym DSTROOT=$PWD/BUILD.hdrs/dst
sudo ditto BUILD.hdrs/dst `xcrun -sdk macosx -show-sdk-path`

Libplatform

This is also required to build libdispatch. It just need it’s source to be available, so you don’t need to build anything here.

cd ../libplatform-126.50.8
sudo ditto $PWD/include `xcrun -sdk macosx -show-sdk-path`/usr/local/include

Libdispatch

At last you can build libfirehose_kernel.a from the libdispatch library. You won’t build the whole library since you’re only out after libfirehose_kernel.a and headers.

cd ../libdispatch-703.50.37
mkdir -p BUILD.hdrs/obj BUILD.hdrs/sym BUILD.hdrs/dst
sudo xcodebuild install -project libdispatch.xcodeproj -target libfirehose_kernel -sdk macosx ARCHS='x86_64 i386' SRCROOT=$PWD OBJROOT=$PWD/obj SYMROOT=$PWD/sym DSTROOT=$PWD/dst
sudo ditto $PWD/dst/usr/local `xcrun -sdk macosx -show-sdk-path`/usr/local

XNU

Finally we can build the XNU kernel itself.

cd $XNUPATH

# You can choose between RELEASE, DEVELOPMENT or DEBUG, or all.
make SDKROOT=macosx ARCH_CONFIGS=X86_64 KERNEL_CONFIGS="RELEASE DEVELOPMENT DEBUG"

Depending on which kernel config you choose, your kernel should be found at either locations.

# If you built DEBUG
file BUILD/obj/DEBUG_X86_64/kernel.debug
BUILD/obj/DEBUG_X86_64/kernel.debug: Mach-O 64-bit executable x86_64

# If you built DEVELOPMENT
file BUILD/obj/DEVELOPMENT_X86_64/kernel.development
BUILD/obj/DEVELOPMENT_X86_64/kernel.development: Mach-O 64-bit executable x86_64

# If you built RELEASE
file BUILD/obj/RELEASE_X86_64/kernel
BUILD/obj/RELEASE_X86_64/kernel: Mach-O 64-bit executable x86_64

And there you go, fresh XNU builds. However I choose to end it here, if you wonder how to actually switch it with the pre shipped one, you’ll have to do something like this.

And just to say it one more time, if you do this and you experience data loss, crashes, or your mac turns into a badger and tries to kill you, or any other unexpected events, it’s not my fault.

sudo cp BUILD/obj/RELEASE_X86_64/kernel /System/Library/Kernels/
sudo kextcache -invalidate /

Enjoy your custom XNU!