#【AIS3 Pre-Exam 2026 Writeup】Reverse - lua

There was no C source in the directory. Only these two files:

file luac_stripped.exe secret.luac

luac_stripped.exe: PE32+ executable for MS Windows 5.02 (console), x86-64
secret.luac:       Lua bytecode, version 5.1

So I treated secret.luac as the program source, and luac_stripped.exe as the custom runtime/compiler. A normal native-pwn route was not useful. The binary is a Windows PE, not a Linux ELF, so ELF protections like RELRO did not matter. GDB plus glibc or printf tricks had no real target here. No GOT overwrite, no saved RIP plan.

The first useful clue came from strings:

strings -a luac_stripped.exe | grep -i -E 'opcode|ctf|disabled'

This compiler has been disabled for the CTF challenge.
Reverse engineer this binary to discover the OpCode mapping!

At first I tried to think about running it directly, but Wine was not a good path on this machine. The environment complained about missing Wine parts, and I did not get a clean bytecode listing or useful debugger state. No trigger, no path. Static analysis looked more reliable.

The binary was not completely blind. It still had many Lua symbols:

objdump -t luac_stripped.exe | rg -i 'luaK_code|luaV_execute|luaP_opmodes|luaP_opnames|luaU_undump|DumpFunction'

Important names were still there: luaU_undump, DumpFunction, luaK_code, luaV_execute, luaP_opmodes, and luaP_opnames. That made the plan more concrete. I could compare this binary with Lua 5.1 behavior instead of guessing every bytecode field by hand.

The secret.luac header looked like normal Lua 5.1:

1b 4c 75 61 51 00 01 04 08 04 08 00

But my first parser did not line up. Some prototype fields became nonsense, and the code size was also wrong. I first thought maybe the constants were encrypted, but the problem was earlier. Each prototype had one extra byte after nups.

The normal Lua 5.1 proto fields are:

source, linedefined, lastlinedefined, nups, numparams, is_vararg, maxstacksize

Here it was:

source, linedefined, lastlinedefined, nups, opcode_key, numparams, is_vararg, maxstacksize

That opcode_key byte mattered later. After fixing the parser layout, the prototype tree and constants started to look reasonable.

Next part was the opcode transform. luaP_opnames only showed fake names like OP_00 to OP_37, so the printed names were not enough.

luaK_code showed how the compiler encoded the opcode before writing one instruction:

objdump -d luac_stripped.exe \
  --start-address=0x140005110 \
  --stop-address=0x140005220

The low 6-bit opcode was XORed with the per-prototype key and a PC-based value. So the decode formula became:

op = (inst ^ (opcode_key ^ 0x2b) ^ (0x11 + 15 * pc)) & 0x3f

pc is the 0-based instruction index. I restarted the parser a few times because I did not trust the offset at first. When this formula was applied per prototype, the disassembly stopped looking random.

The opcode number still needed semantic meaning. For that I used two places:

objdump -s -j .rdata \
  --start-address=0x140023c60 \
  --stop-address=0x140023d80 \
  luac_stripped.exe

and:

objdump -d luac_stripped.exe \
  --start-address=0x140013890 \
  --stop-address=0x140014900

luaP_opmodes gave the instruction format, like ABC, ABx, and AsBx. luaV_execute gave the real behavior for each dispatch block. Standard Lua 5.1 opcode order was not usable directly, because the challenge shuffled the slots.

The recovered mapping was:

OP_00 TFORLOOP   OP_01 ADD       OP_02 MOVE      OP_03 UNM
OP_04 LOADK      OP_05 LOADBOOL  OP_06 CONCAT    OP_07 LOADNIL
OP_08 SUB        OP_09 GETUPVAL  OP_10 JMP       OP_11 GETGLOBAL
OP_12 GETTABLE   OP_13 SETGLOBAL OP_14 SETUPVAL  OP_15 MUL
OP_16 SETTABLE   OP_17 DIV       OP_18 MOD       OP_19 NEWTABLE
OP_20 SELF       OP_21 POW       OP_22 LEN       OP_23 LT
OP_24 TEST       OP_25 LE        OP_26 TESTSET   OP_27 EQ
OP_28 CALL       OP_29 TAILCALL  OP_30 RETURN    OP_31 FORLOOP
OP_32 FORPREP    OP_33 SETLIST   OP_34 CLOSE     OP_35 CLOSURE
OP_36 VARARG     OP_37 NOT

After that, the Lua logic was readable enough. The main chunk builds several closures, prints > , reads from io.read(), calls a checker, then prints ok or no.

Some helpers were only table builders. One helper did XOR by looping over bits. Another decoded table values:

out[i] = (t[i] - a - ((i * b) % c)) % 256

Another helper interleaved two tables. Odd positions came from the first table, even positions from the second table.

Those helpers produced the real key table:

key = [23, 88, 41, 199, 17, 90, 250, 61, 143, 12, 77]

and the target table:

target = [
    158, 35, 172, 11, 160, 217, 123, 62, 248, 242, 88,
    51, 84, 125, 92, 152, 46, 62, 166, 147, 23, 73,
    80, 220, 153, 6, 67, 13, 195, 5, 91, 6, 32
]

The checker starts with state = 65. For each input byte, it computes:

k = key[(i * 5 + state) % len(key)]
x = ((ch + i + state) % 256) ^ ((k + i * 7) % 256)
x = (x + ((k ^ i) % 13)) % 256

If x is not equal to target[i - 1], it returns false. Otherwise it updates:

state = (state + x + k + i * 3) % 256

At the end it also checks:

state == 229

This is easy to invert byte by byte, because state, k, and target[i - 1] are known at each step. No brute force was needed for the flag characters.

Full solver:

#!/usr/bin/env python3

key = [23, 88, 41, 199, 17, 90, 250, 61, 143, 12, 77]
target = [
    158, 35, 172, 11, 160, 217, 123, 62, 248, 242, 88,
    51, 84, 125, 92, 152, 46, 62, 166, 147, 23, 73,
    80, 220, 153, 6, 67, 13, 195, 5, 91, 6, 32
]


def check(buf: bytes) -> bool:
    state = 65
    if len(buf) != len(target):
        return False

    for i, ch in enumerate(buf, start=1):
        k = key[(i * 5 + state) % len(key)]
        x = ((ch + i + state) % 256) ^ ((k + i * 7) % 256)
        x = (x + ((k ^ i) % 13)) % 256
        if x != target[i - 1]:
            return False
        state = (state + x + k + i * 3) % 256

    return state == 229


state = 65
flag = []

for i, t in enumerate(target, start=1):
    k = key[(i * 5 + state) % len(key)]
    add = (k ^ i) % 13
    ch = (((t - add) % 256) ^ ((k + i * 7) % 256))
    ch = (ch - i - state) % 256
    flag.append(ch)
    state = (state + t + k + i * 3) % 256

flag = bytes(flag)
print(flag.decode())
print(state)
print(check(flag))

Output:

AIS3{Lu4_0pc0d3_Shuffl1ng_1s_Fun}
229
True