TRIVUL: Improving Precision of Static Vulnerability Detection through Multi-Agent Reasoning

An anonymous artifact repository for a multi-agent framework for improving the precision of Java static vulnerability detection.

TriVul is a multi-agent assisted static analysis framework for Java vulnerability detection. Its goal is to improve the precision of existing static application security testing (SAST) tools by refining their findings through staged deterministic validation and targeted LLM-based semantic reasoning.

Overview

TriVul integrates heterogeneous static analyzers with stage-specific LLM reasoning. It employs an LLM-based filter during rule-based candidate search, applies deterministic family-aware validation for structural pruning, and then uses a split semantic layer in which an LLM judge evaluates retained candidates while a separate LLM recovery module revisits deterministically rejected ones. The central design goal is to improve the precision--coverage balance of existing Java vulnerability detection tools without replacing their scalable static-analysis backbone.

Main Results

The table below summarizes the main benchmark-level results reported in the paper on the 113 matched CWE-Bench-Java samples.

Method	Det.	DR (%)	Avg FDR (%)	Avg F1
CodeQL	12	10.62	89.66	0.105
Semgrep	11	9.82	94.81	0.062
IRIS + GPT-4	55	48.25	83.96	0.188
IRIS + GPT-4o-mini	27	23.89	89.27	0.040
TriVul + Mixed-LLM	58	51.32	66.94 ↓ 20.3%	0.374 ↑ 99.0%
TriVul + GPT-4o-mini (t=0.2)	54	47.79	67.60 ↓ 19.5%	0.361 ↑ 92.0%
TriVul + Mistral-7B (t=0.6)	54	47.79	68.23 ↓ 18.7%	0.356 ↑ 89.4%
TriVul + CodeLlama2-7B (t=0.2)	53	46.90	68.72 ↓ 18.1%	0.350 ↑ 86.2%

In the best mixed-model configuration, TriVul uses GPT-4o-mini for Stage I filtering and Stage III judging, and Mistral-7B for Stage III recovery. This configuration achieves the strongest overall precision--coverage balance among the evaluated settings.

Additional Mixed-LLM Ablation Results

In the following, we provide the internal ablation results we conducted on different LLM combinations for Trivul.

Method	Det.	DR (%)	Avg FDR (%)	Avg F1
Trivul + Mixed-LLM (GPT-4o-minit=0.2 + Mistralt=0.6)	58	51.32	66.94	0.374
Trivul + Mixed-LLM (GPT-4o-minit=0.2 + CodeLlamat=0.2)	43	38.05	75.26	0.279
Trivul + Mixed-LLM (Mistralt=0.2 + CodeLlamat=0.2)	56	49.56	68.28	0.353

Framework Overview

TriVul follows a three-stage design:

1. Stage I: Candidate Search Multiple worker branches surface coarse vulnerability candidates, including heterogeneous static analyzers and a lightweight rule-based vulnerability search component. A conservative LLM filter is applied in this stage to reject clearly spurious findings while preserving plausible or ambiguous candidates.

2. Stage II: Deterministic Structural Validation Candidate findings are fused and converted into explicit source-to-sink paths. Family-aware structural rules then prune candidates that do not match the expected structural shape of the target vulnerability family.

3. Stage III: Semantic Judgment and Recovery A downstream LLM judge evaluates deterministic survivors, while a separate LLM recovery component revisits a limited subset of structurally rejected candidates. This split design allows TriVul to improve precision while still recovering borderline true positives that deterministic rules may prune too aggressively.

Why TriVul

Static analyzers remain valuable for broad vulnerability discovery, but their practical adoption is often limited by false positives. TriVul is built around a simple idea: broad candidate discovery and structural pruning should remain deterministic and scalable, while expensive semantic reasoning should be used selectively and only where it adds the most value.

In that sense, TriVul is not intended to replace existing analyzers with a standalone end-to-end detector. Instead, it operates on candidate findings produced by existing tools and improves the quality of the alerts that reach developers.

External Analysis Tools

TriVul combines findings from several external analysis branches. For full functionality, the following tools should be available:

• CodeQL CLI
  • used for CodeQL-based scanning
• Semgrep
  • used as a lightweight rule-based baseline and candidate source
• Joern
  • used for graph/dataflow-based candidate discovery

LLM Backends

TriVul supports both local and API-based LLM backends.

• Ollama
  • used for local inference in the main artifact workflow
  • requires a running Ollama service
  • example models used in our experiments include:
    • chevalblanc/gpt-4o-mini:latest
    • mistral:7b
    • codellama:7b

If Ollama is used, the corresponding models must be pulled locally before running the pipeline.

Configuration

TriVul uses a small set of configurable paths for benchmark inputs, metadata, auxiliary data, output directories, and CodeQL databases. Reviewers should check config.py for the expected meaning of each path and the corresponding environment-variable overrides.

In particular, config.py documents:

• where benchmark project directories should be placed,
• where benchmark metadata files such as fix_info.csv are expected,
• where TriVul-specific auxiliary CSV files should live,
• where generated outputs are written, and
• where CodeQL databases should be stored when CodeQL-based branches are enabled.

Setup

TriVul requires a Python environment together with several external analysis tools.

1. Create a virtual environment

python3 -m venv .venv
source .venv/bin/activate
cd trivul
pip install --upgrade pip
pip install -r requirements.txt

2. Install external tools

For full functionality, the following tools should be installed and available in PATH:

• CodeQL CLI
• Semgrep
• Joern (including joern-parse)
• Ollama for local LLM inference

3. Download the benchmark dataset

TriVul is evaluated on CWE-Bench-Java.

• GitHub repository: https://github.com/iris-sast/cwe-bench-java

After downloading the dataset, check trivul/config.py and adjust the benchmark, metadata, output, and CodeQL database paths if your local layout differs from the defaults.

4. Prepare local LLM models

If you use Ollama, make sure the required models are pulled locally before running TriVul. For example:

ollama pull chevalblanc/gpt-4o-mini:latest
ollama pull mistral:7b
ollama pull codellama:7b

How to Run

TriVul is currently invoked through the package entry point:

python -m trivul.main --sample <sample_id>

Example: run one sample

python -m trivul.main \
  --sample perwendel__spark_CVE-2018-9159_2.7.1 \
  --stage1-model chevalblanc/gpt-4o-mini:latest \
  --stage3-model mistral:7b \
  --temperature-stage1 0.2 \
  --temperature-stage3 0.6 \
  --json-out outputs/sample_result.json \
  --csv-out outputs/sample_summary.csv

Anonymity Note

This repository is provided in anonymized form for peer review. Some identifying metadata and author-specific materials are intentionally omitted from the current artifact and will be added back in the public version after publication.