Fuzzing Apache httpd server with American Fuzzy Lop + persistent mode
Intro
Recently, I reported . This is a cross-post from my where I explain how to fuzz network programs with AFL by porting techniques learned in into AFL.1 After a small chat with Dominic he asked me to re-post it here which, for me it’s an honour to do so!
The reported CVE was obtained with code analysis and instrumentation of the right parts of the code (mainly core and parsing) – First, with honggfuzz I got the initial dirty test cases and then, through generated a few thousands mutations and finally AFL with the technique described here.
Goal
When stumbling upon the great and trying its awesome deterministic fuzzing capabilities and instrumentation soon we find out that this fuzzer was built to fuzz programs that take input from the command line (or standard input) instead of a network socket. Because of that, I thought I would give it a try and make AFL fuzz against Apache’s httpd server. First the AFL way by adding a new option to Apache’s command line and the second way, by using the persistence fuzzing (afl-clang-fast) by shamelessly copying the way .
Takeaways for the reader
- Learn to fuzz network based programs with AFL
- Code to start fuzzing Apache with AFL in no time
- A push in the interwebz fuzzing race
Let’s do it!
Setup part 1
I will be using a Debian GNU/Linux 8 64bit with the kernel 4.9.0-0.bpo.2-rt-amd64. You don’t really need that setup. All that you need is an operating system (under a virtual machine or not) that can compile and run AFL with the afl-clang feature.
But, before getting into any compilations/installations/fuzzing, I encourage you to set an organised folder structure that suits you best but, in case you haven’t got one already, I am sharing mine.
Under the Fuzzing folder I have:- Victims – For the target programs that we are about to fuzz
- Fuzzers – AFL, honggfuzz, radamsa, etc. go here
- Testcases – The samples we are going to feed the fuzzer to throw against our Victim
- Sessions – For storing the fuzzing sessions
- Compilers – To store compilers such as clang-4.0 and binaries needed to compile
Getting clang-4.0 and llvm-tools
Getting pre-built binaries for clang-4.0 and the llvm-tools is fairly easy if you have Debian or Ubuntu. You can get these from here . In my case the tarball.
If you are following the structure mentioned above, you can cd into your Compilers folder and drop the tarball there, extract it and then add the binaries folder to your path by adding the following line to the end of your ~/.bashrc file (~/.profile nor /etc/environment worked for me – It seems that you need to logout and login for these changes to take place).PATH="$HOME/Fuzzing/Compilers/clang+llvm-4.0.0-x86_64-linux-gnu-debian8/bin:$PATH"
Now issuing the which command on a new shell we should have the following output:
$ which clang$ /home/javier/Fuzzing/Compilers/llvm-clang-binaries/clang+llvm-4.0.0-x86_64-linux-gnu-debian8/bin/clang
Compiling and Installing AFL
Compiling AFL should be pretty straight forward but, for the lazy, you can just copy paste these commands and you should be ready to go:
sudo apt install build-essentialwget http://lcamtuf.coredump.cx/afl/releases/afl-latest.tgz tar xzf afl-latest.tgz cd afl* make && sudo make install && echo-e "\n[+] AFL ready to fuzz at $(which afl-fuzz)"
That’s it, the binary afl-fuzz should be in your path now ready to be unleashed.
Compiling and installing Apache
First move to a folder where we are about to download Apache server and all the dependencies needed. In my case, the folder is at ~/Fuzzing/Victims/apache_afl/.
Before compiling Apache we are going to need the Apache Portable Runtime (APR), APR Utils and support for HTTP/2 through nghttp2.
No lazyness this time, go download:- The following dependencies:
- sudo apt install pkg-config
- sudo apt install libssl-dev
- (DON’T DOWNLOAD PCRE2!!)
Now we need to get the latest Apache build, which I recommend you do from their subversion repository by doing so:
sudo apt install subversionsvn checkout http://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x httpd-2.4.x
Now if you downloaded and unpacked everything, you should have a similiar output from ls -l command:
drwxr-xr-x 28 javier javier 4096 Apr 9 23:12 apr-1.5.2 drwxr-xr-x 20 javier javier 4096 Apr 9 23:13 apr-util-1.5.4 -rwxr-xr-x 1 javier javier 1445 May 6 20:56 compile_dependencies_with_flags.sh drwxr-xr-x 11 javier javier 4096 Apr 29 01:22 httpd-2.4.x drwxr-xr-x 14 javier javier 4096 Apr 9 23:14 nghttp2-1.21.0 drwxr-xr-x 9 javier javier 12288 Apr 9 22:53 pcre-8.40
I made the following script to compile and link it all since I found myself often changing flags for the compiler and it was too time consuming compiling each dependency one by one with its own flags. Get it with:
wget
Before editing any files yet lets run the bash script and see that we can compile everything cleanly without any missing dependencies whatsoever:
CC="clang" CXX="clang++" PREFIX="/usr/local/apache_clean_test/" ./compile_dependencies_with_flags.sh
Please see the next asciinema for reference of a nice compilation run (takes a while to load).
Fuzzing Apache with AFL through an inut file
As you might know by now, AFL in its basic usage feeds a file into the target program through its “argv” array in the following form:(通过argv参数传递文件名)
afl-fuzz -i testcases/ -o session_1/ -- ./victim -v -f @@
The problem with Apache is that it doesn’t have such functionality so we will have to patch it our own way.
Patching Apache
Taking into account the aforementioned problem, we need to write some lines into Apache’s main.c file to make it able to read files from input.（注意补丁方式：在apache main函数中加入读取文件的函数，相当于提供了一种直接从文件读取并运行的能力！）
You can patch Apache with the following . Now apply it by cd‘ing into the base path of Apache httpd’s source code and issuing the following command:patch -p0 -i apatching_apache_for_AFL_fuzzing.diff
I am not going to cover all the patch in detail but some parts are worth mentioning.
The first and only time that I have seen the following technique was by , an information security researcher at Google when fuzzing Apache with honggfuzz. It is pretty clever and pretty obvious once you see the way it is done. It basically consists of launching a new thread inside Apache that will create a connection to the web server itself and send our fuzzed input; all happening within the same unique process so we can get all the instrumentation data into AFL. Clever! Right?
To achieve this it uses the function that disassociates parts of this thread’s context from the others without the need of creating a new process. Specifically, the network and mount namespaces are separated. This is done so we can have several processes with the same settings (listening on the same loopback interface and port with the help of netIfaceUp on line of the patch file and writing logs to /tmp on line ) running at the same time on each process we launch.We can see that the (unsh();) function is indirectly called on line previous to firing the new thread that will receive the fuzzed input at line .
The process of reading a file through the “-F” switch starts on line and when the file is read into a buffer, this buffer is passed onto the function responsible to launch the new thread () that will, in turn, send the fuzzed file inside the function on line .
unshare() allows a process (or thread) to disassociate parts of its execution context that are currently being shared with other processes (or threads). Part of the execution context, such as the mount namespace, is shared implicitly when a new process is created using or , while other parts, such as virtual memory, may be shared by explicit request when creating a process or thread using . The main use of unshare() is to allow a process to control its shared execution context without creating a new process. unshare允许进程控制与其它进程或线程共享的上下文。
Fuzzing Apache
Yes! We are ready now! Let’s compile Apache:
CC="afl-clang" CXX="afl-clang++" PREFIX="/usr/local/apache_afl_blogpost/" ./compile_dependencies_with_flags.sh
If you are familiar with AFL and how it works you probably have your own testcases to feed it with, in case you don’t the following video shows how to launch AFL and create two very simple test cases – remember that we need to be root or use “sudo sysctl -w kernel.unprivileged_userns_clone=1” in order to use the unshare function AND MORE IMPORTANT TO LAUNCH APACHE WITH THE “-X” FLAG AND “-m none -t 5000” FOR AFL SO IT CAN BOOT APACHE:
Setup part 2
Compiling afl-clang-fast
Remember we downloaded clang-4.0 and the llvm-tools before and set it in our path? This is where it comes most handy. Inside your AFL folder, navigate to the llvm_mode and run make and sudo make install in the root folder of AFL. What we have just done is compiling an experimental feature of AFL that will run a certain number of fuzzed inputs against a program without having to run the whole program per fuzzing input.
Patching Apache for Persistence
Following the same dance as before, (与单个不同，怎么没有读文件呢？), patch it and ready to fuzz!
Fuzzing Apache with AFL on Persistent mode
Let the video speak for itself but again remember the previously mentioned “-X” flag for Apache server and the “-m none -t 5000” flags for AFL:
Update:
, you don’t need to run everything as root as I did in the examples which, obviously imposes security risks. You can make use of the following command line (this one to be run as root :P) to let non-root users make use of the unshare function:
patch -p0 -i apatching_apache_for_AFL_fuzzing.diff
Conclusions
We have learned how to effectively fuzz server programs such as web servers by using Robert’s technique of launching a different thread and different context through the unshare function.
While relatively fast, it is not as fast as honggfuzz, which can go up to 20k iterations per second with 8 processes running. Also after a few days of fuzzing AFL’s stability goes way down 50% because of the multithreading that Apache is implemented on and so, any reported crashes, can be false positives or would either be needing the last iterations launched which AFL lacks of.It is left as an exercise to the reader to implement into Apache a way to save the last 1k sent inputs into a file and to think which other ways would improve stability and/or speed. Hint: Don’t instrument everything.
Shouts to the Apache Security Team, it was a pleasure to work with such an efficient team.
Happy pwning! x)
For further doubts, ramblings or whatever you can contact me on javier at domain’s name