When LLM Decompilers Recompile More and Preserve Less
This analysis explores the tradeoffs between recompilability and behavioral fidelity in large language model (LLM)-based decompilers, drawing on empirical findings from recent research. It highlights how current evaluation metrics may prioritize recompilation success over functional accuracy, with i
Recompilation Success vs. Behavioral Fidelity
Recent studies have revealed a critical tension in the evaluation of LLM-based decompilers: while these tools often achieve high recompilation rates, they may fail to preserve the original behavior of the decompiled code. Traditional decompilers like Ghidra and Hex-Rays leave unresolved elements as placeholders, but LLM-based approaches generate clean, idiomatic C code. However, this focus on recompilability can lead to divergent behavior on inputs not included in the original test suite.
Researchers have developed a new evaluation method called Decompile-Diverge, which compares the behavior of decompiled code against the original on a fuzzed input corpus. This approach uncovers discrepancies that standard test suites miss. For example, some decompiled functions may pass all original tests but fail on new inputs, and disclosed vulnerabilities may be absent in the recompiled output with no visible trace.
Tradeoffs in Evaluation Metrics
The study found that while LLM-based decompilers improve recompilation rates compared to traditional tools, they may reduce behavioral agreement. For instance, one LLM-based system increased Ghidra's build rate from 75% to 90%, but its matched rate dropped from 74% to 62%. This suggests that recompilation success does not always correlate with functional accuracy.
On real-world code samples, including 300 GitHub library functions and 287 CVE-grounded functions, the divergence between recompilation and behavior was significant. Up to 13% of decompiled functions showed behavioral differences on new inputs, and one in ten disclosed vulnerabilities was absent in the recompiled output.
Implications for Security and Software Analysis
The findings highlight a key limitation of current evaluation practices: reliance on fixed test suites may overlook critical behavioral changes. This has implications for security tasks such as vulnerability detection and malware analysis, where functional accuracy is essential. The introduction of new fields, types, and guards in the decompiled code may alter the original logic, even if the code compiles and passes existing tests.
For practitioners, this suggests that recompilation success should not be the sole metric for evaluating decompilers. Instead, behavioral comparison tools like Decompile-Diverge can provide a more comprehensive assessment of functional fidelity. This is particularly important for security-critical applications where even minor behavioral changes can have significant consequences.
Source: Lobsters. Analysis based on publicly available information; no hands-on testing is claimed.
