Policy-Conditioned Firewall Enforcement Audit Reproducibility Package

Public reproducibility artifact for the manuscript:

LEAF: A Leakage-Aware, Explainable Audit Framework for Stress-Testing Policy-Conditioned Firewall Enforcement

Study name:

LEAF (Leakage-Aware, Explainable Audit Framework)

Repository short name:

threat-xai-action

GitHub URL:

https://github.com/canay/threat-xai-action

Related Study

This repository supports LEAF, an audit framework evaluated on five-class operational firewall enforcement using enterprise Palo Alto threat logs. Supervised models serve as measurement probes within a leakage-aware evaluation sequence. The study tests how the evidence changes when direct policy fields, threat descriptors, duplicate signatures, time ordering, and unseen policy contexts are examined separately. The evidence comes from one firewall and policy state over one observation window, so cross-device and longitudinal validation remain future work.

Target classes:

• Allow
• Block
• Drop
• Reset-Both
• Reset-Server

Keywords

• enterprise firewall logs
• policy-conditioned firewall enforcement
• operational firewall telemetry
• leakage-aware evaluation
• temporal validation
• diagnostic explainable AI

Authors And Contact

Author order:

1. Özkan Canay, corresponding author
2. Cem Özkurt
3. Fatmir Garri

Özkan Canay

• Department of Information Systems and Technologies
• Faculty of Computer and Information Sciences
• Sakarya University
• Email: canay@sakarya.edu.tr
• ORCID: 0000-0001-7539-6001

Cem Özkurt

• Department of Data Science and Analytics
• Faculty of Computer and Information Sciences
• Sakarya University
• Email: cemozkurt@sakarya.edu.tr
• ORCID: 0000-0002-1251-7715

Fatmir Garri

• Department of Information Technology
• Sakarya University of Applied Sciences
• Email: fatmirgarri@subu.edu.tr
• ORCID: 0009-0005-6762-6846

GitHub:

https://github.com/canay

This repository is the author-managed code-and-output replication package for the associated manuscript. The main branch is the current public package.

Repository Contents

• code/: the controlled-data conversion entry point and all manuscript-used evaluation, explanation, validation, benchmark, and figure-rendering scripts (02--35)
• scripts/create_public_sample.py: the single retained helper for generating the privacy-audited public sample
• data/processed/: controlled-data access and schema notes, the aggregate processing manifest, and a 1,000-row anonymized public smoke-test sample with audit
• results/: CSV/JSON aggregate evaluation outputs, reproducible explanation artifacts, and additional validation outputs
• requirements.txt: portable minimum compatible dependencies
• requirements-lock-primary-linux-aarch64.txt: exact complete resolution for the canonical Linux/aarch64 VPS evidence
• requirements-lock.txt: platform-labelled Windows x86-64 diagnostic environment retained for provenance only
• SEED_MANIFEST.json: explicit random seeds and resampling scope
• CITATION.cff: citation metadata for this artifact
• MANIFEST.md: release contract and public/controlled-content checklist
• TRACEABILITY.md: manuscript evidence to script/output mapping, including the controlled processed-file checksum for authorized reruns

Raw enterprise firewall exports are not redistributed in this repository. The full event-level processed dataset is also not included unless explicit institutional authorization confirms that redistribution is permitted after privacy and security review. A 1,000-row anonymized sample is included only for schema inspection and smoke testing; it is not used for the reported manuscript metrics.

Data

Controlled processed dataset path:

• data/processed/threat_five_class.csv

The file above is the expected local path for authorized full reproduction, but the full file is not included in this public artifact. The data were derived from institutional firewall logs and may expose organization-specific or security-sensitive operational context even after processing. The calendar date, 14 May 2026, is intentionally disclosed with author approval; organization identity, topology, policy names, raw exports, and event-level records remain protected.

Aggregate cohort flow:

• Raw threat-log rows: 186,271
• Excluded detection-only alert rows: 9,115
• Mapped five-class rows: 177,156
• Observation window: 6 h 30 min 26 s on a high-volume weekday
• Aggregate audit: data/processed/threat_dataset_processing_manifest.json

Rows:

• 177,156 labeled threat-log records

Controlled processed file identity for authorized reruns:

• Event rows: 177,156
• Header rows: 1
• Size: 44,929,133 bytes
• SHA-256: 1BE9896A996DD58A582D94319180405A8559AB5193EF692BD8D2D2D614693724

Label semantics: this checksum identifies the original canonical experiment file, in which raw block was stored under the internal alias Deny. Release-facing code, tables, figures, and the public sample use Block; a label-only byte round-trip check confirms that class membership and reported metrics are unchanged.

Class counts:

• Drop: 160,788
• Block: 10,187
• Reset-Both: 2,903
• Allow: 2,502
• Reset-Server: 776

Public anonymized smoke-test sample:

• File: data/processed/public_anonymized_sample_1000.csv
• Audit: data/processed/public_anonymized_sample_1000.audit.json
• Rows: 1,000
• SHA-256: 68672057A80CA70D5B52301E3BB9ED00D8A41D707C50EFFC8B53217A345253DD
• Class counts: Drop 731, Block 93, Reset-Both 62, Allow 61, Reset-Server 53

This sample preserves the 27-column processed-file schema and all five target classes, but it does not preserve original timestamps, raw firewall actions, policy names, zones, interfaces, ports, applications, threat names, countries, source row identifiers, original row order, or other event-level values. Except for target, event-level values are redacted or mapped within each column to stable generic tokens such as application_001, source_zone_001, and rule_context_001; sampled rows are shuffled before synthetic sequential timestamps on 2026-01-01 are assigned. The sample is suitable for checking file format, parser behavior, and lightweight script execution only. It does not reproduce the reported metrics.

Privacy note:

The package excludes raw enterprise log exports and excludes the full event-level processed CSV by default. Non-missing Rule values in the controlled processed file were converted to stable anonymous rule-context identifiers during internal reproduction, and the public smoke-test sample applies stronger column-wise tokenization before release. The vendor fields named Source Country and Destination Country have mixed semantics in the controlled processed data: 13,748 source-side rows (7.76%) and 7,243 destination-side rows (4.09%) contain coarse private, shared, or special-purpose address-range labels rather than country names. Event-level source and destination IP addresses are excluded from the predictors; the retained range labels are treated as residual topology proxies. Public release of the full processed file should occur only after the relevant institutional authority approves the exact file and any required data-use agreement.

Detailed event-level review-queue records are excluded from the public artifact. code/09_operational_review_and_context_audit.py writes only aggregate and aliased per-context outputs by default. Authorized users may request an event-level queue only with --private-records-out; the script refuses any such destination inside this public repository.

Public schema aids:

• data/processed/schema.csv documents the expected columns, modeling role, and public-release note for each field.
• data/processed/public_anonymized_sample_1000.csv is a real-row-derived but strongly anonymized sample for schema inspection and smoke testing only.
• data/processed/public_anonymized_sample_1000.audit.json records the sample seed, source checksum, output checksum, class counts, and anonymization rules.
• data/processed/threat_dataset_processing_manifest.json records aggregate target-construction counts and the controlled processed-file checksum.
• TRACEABILITY.md maps manuscript tables and validation claims to generating scripts and aggregate output files.

Main Evidence Snapshot

Core holdout benchmark:

• Leading two-error profile: LightGBM, CatBoost, and XGBoost
• Leading-profile accuracy: 0.999944
• Leading-profile balanced accuracy: 0.999312
• Leading-profile macro-F1: 0.998373
• Leading-profile weighted-F1: 0.999944
• Downstream XGBoost reference: two reset-boundary errors
• XGBoost accuracy: 0.999944
• XGBoost balanced accuracy: 0.999312
• XGBoost macro-F1: 0.998373
• XGBoost weighted-F1: 0.999944

Core five-fold cross-validation macro-F1:

• LightGBM: 0.996674 +/- 0.001951
• CatBoost: 0.996630 +/- 0.001930
• XGBoost: 0.996343 +/- 0.002282
• Extra Trees: 0.995018 +/- 0.002547

Leakage-aware ablation without direct threat descriptors:

• Removed fields: Threat/Content Name, Threat/Content Type, Severity
• LightGBM holdout macro-F1: 0.993126
• XGBoost holdout macro-F1: 0.990091

No-threat-descriptors cross-validation:

• XGBoost CV macro-F1 mean: 0.986870
• LightGBM CV macro-F1 mean: 0.985312
• Extra Trees CV macro-F1 mean: 0.981459
• CatBoost CV macro-F1 mean: 0.976976

Additional XGBoost validation checks:

• Minimal-context stratified holdout macro-F1: 0.982091
• Core chronological holdout macro-F1: 0.866740
• No-threat-descriptors chronological holdout macro-F1: 0.836739
• Core chronological holdout ECE, 10 bins: 0.004593
• SHAP top-10 feature overlap across repeated stratified samples: 1.000000
• The canonical SHAP rerun reproduces the selected XGBoost holdout at 0.998373 macro-F1 with two errors.
• A 64-record diagnostic reconstructs raw margins within 0.00000882 maximum absolute error and probabilities within 0.00000012.
• The public XAI release includes aggregate SHAP evidence and three deidentified manuscript figures; no row-level LIME table is released.

Model-independent policy-action entropy audit:

• Outputs are in results/policy_action_entropy_audit/.
• Rule-context dominant-action fraction, record-weighted: 0.992758.
• Rule-context normalized entropy, record-weighted: 0.017669.
• Rule + threat type + direction dominant-action fraction, record-weighted: 0.997217.
• Low-purity rule-context rows at a 0.80 dominant-action threshold: 1,248.
• Rule values are anonymized as rule_context_* in the output artifacts by default.

Strengthening checks:

• Feature-group ablation results are in results/strengthening/.
• Chronological core macro-F1 in the feature-group audit: 0.866740.
• Chronological no-threat-descriptors macro-F1 in the feature-group audit: 0.836739.
• Chronological no-application-context macro-F1: 0.941499.
• Confidence-based referral on chronological core retains 97.45% of records with 9 retained-set errors and 0.9829 macro-F1.
• These temporal results are within-day forward-shift stress tests, not long-term temporal validation.

Duplicate-aware grouped-split stress test:

• Exact feature signatures are grouped so the same structured feature vector cannot appear in both train and test partitions.
• Core feature signatures: 47,047 unique signatures; 151,361 duplicate rows (85.44%).
• No-threat-descriptors grouped split macro-F1: 0.983299 +/- 0.004429.
• Minimal-context grouped split macro-F1: 0.977351 +/- 0.006797.
• Core grouped split macro-F1: 0.996657 +/- 0.000804 (range 0.9953--0.9974).

Held-out policy-context stress test:

• Rule is used only as a held-out grouping variable and is not included as a predictor.
• Among seven named rule contexts with at least 50 records, the core model reached 0.804460 support-weighted accuracy and 0.653076 support-weighted observed-class macro-F1.
• The no-threat-descriptors and minimal-context settings reached 0.378214 and 0.372487 support-weighted observed-class macro-F1, respectively.
• These results are a stress audit for local policy-context sensitivity, not evidence of cross-organization generalization.

Reviewer-revision sensitivity checks:

• Aggregate outputs are in results/reviewer_revision_sensitivity/.
• reviewer_revision_provenance_manifest.json binds scripts 22--24, their controlled input hashes, and their aggregate outputs without exposing raw identifiers or event-level rows.
• code/22_reviewer_revision_sensitivity.py compares the core setting, most-frequent Rule imputation, and explicit missing-rule encoding on the same seed-42 partition.
• In the registered Windows sensitivity environment, all three configurations produce two errors and 0.998373 macro-F1; the two with-policy encodings generate identical predictions.
• Pooling saved confusion counts across seven disjoint named-rule holdouts gives standard five-class macro-F1 values of 0.663073 for core, 0.422397 without threat descriptors, and 0.411417 for minimal context.
• The seven named contexts contribute 16,815 held-out events; this is the effective support for named-context transfer.
• code/24_rule_context_regime_audit.py shows that all 159,992 packet records lack Rule and account for 99.82% of missing Rule values. It also reports a support-qualified six-context sensitivity after excluding the fold that removes 91.90% of Reset-Both training support; core pooled macro-F1 is 0.867243 across 14,127 held-out events.
• code/23_session_identifier_coverage_audit.py writes aggregate coverage only. A valid nonzero raw session identifier exists for 14,660 of 177,156 eligible events, and only 197 events belong to repeated identifiers.
• None of these scripts exports event-level predictions, session identifiers, enterprise rule names, or raw records.

Operational review and unseen-context audit:

• Outputs are in results/operational_review_context_audit/.
• The saved operational review aggregate is generated with the 300-tree selected-model configuration (--n-estimators 300), so its model-log disagreement counts match the selected-model validation table.
• Aggregate queue summaries and aliased per-context results are public; detailed event-level queue records are not distributed.
• In the stratified core setting, a review queue triggered by reset-related records or the lowest 1% confidence records routed 941 of 35,432 test records (2.66%) to review and contained both model-log disagreements.
• In the no-threat-descriptors setting, the same review rule routed 920 records (2.60%) and contained all 15 model-log disagreements.
• A broader rule using model-log disagreement, lowest 5% confidence, or reset-related records routed 1,802 core records (5.09%) and 2,231 no-threat-descriptors records (6.30%).
• The exploratory unseen-context holdout audit shows that performance can deteriorate sharply when entire high-support categorical values are absent from training. Treat this as a deployment-boundary stress test, not as independent external validation.

Q1 audit revision checks:

• Outputs are in results/q1_audit_revision/.
• Train-majority macro-F1 is 0.190312 under stratified holdout and 0.181301 under chronological holdout.
• A non-rule, non-direct-descriptor app-direction-zone context lookup reaches 0.953460 stratified macro-F1 but falls to 0.748216 under chronological ordering.
• A richer minimal operational context lookup reaches 0.976840 stratified macro-F1 and 0.620468 chronological macro-F1.
• A fair 20-feature exact-signature lookup reaches only 0.511682 stratified macro-F1 and 0.225351 chronological macro-F1 because unmatched high-cardinality signatures fall back to the training-majority action; the learned model's performance is therefore not reproduced by exact-signature memorization.
• Test-record bootstrap intervals from saved fixed-model XGBoost outputs give chronological core macro-F1 0.866740 with 95% interval 0.8537--0.8792.
• The corresponding chronological no-threat-descriptors macro-F1 is 0.836739 with a 95% interval of 0.8224--0.8508.

Additional selected-model sensitivity experiments:

• Outputs are in results/q1_audit_revision/. All use the 300-tree, seed-42 selected-model pipeline. The benchmark and every model-dependent selected-model check were generated under one locked Linux/aarch64 VPS environment so that the reported XGBoost rows share the same preprocessing, library, split, and model contract.
• classweight_sensitivity.csv: adding inverse-frequency sample weights leaves core macro-F1 unchanged at 0.9984; in the no-threat-descriptors setting, macro-F1 rises from 0.9901 to 0.9907 and balanced accuracy from 0.9928 to 0.9967, with the same 15 errors. The selected-model conclusion is stable under this tested weighting change.
• sourceport_nearduplicate_ablation.csv: Source Port is high-cardinality (10,345 distinct); grouping records by their signature excluding Source Port (which collapses 14,678 signature groups and raises the duplicate-row fraction from 0.854 to 0.923) still yields a median core macro-F1 of 0.9932 (range 0.924--0.997), so the reconstruction evidence does not collapse when burst-level source-port variation is removed from grouping.
• forward_chaining_chronological.csv: expanding-window chronological folds (train earliest 60/70/80/90 percent, test the following 10 percent) show that within-day temporal performance is cutpoint-dependent rather than a single value, consistent with the class-mix shift documented for the 80/20 cut.
• forward_chaining_class_support_attestation.json: a no-refit, hash-bound readback confirms that every saved test window contains all five canonical classes; the saved macro-F1 denominator was therefore five in every fold.
• category_code_order_sensitivity_summary.csv: under canonical most-frequent categorical imputation and fixed transformed-feature order, five category-map permutations change selected-model macro-F1 by at most 0.0016 in the core setting and 0.0017 without threat descriptors.

Experimental Environment

The enterprise-data benchmark, its model-dependent validation and explanation analyses, and the public UCI portability artifact ran on Oracle Cloud Ampere Linux aarch64 systems with Python 3.12.3 and no specialized accelerator. All CPMB/PTED stages ran CPU-only on Linux x86-64 with Python 3.12.3. The canonical Family C run, the secondary diagnostics, the split and identifier audits, and their exact repeats used network-isolated containers with frozen package versions and single-thread limits on the same physical host used for Families A and B. The Linux run manifests in results/cpmb_pted/ record the host, container image, architecture, runtime, input hashes, and output hashes. The registered canonical enterprise evidence bundle is 2026-07-13_codex_vps_selected_model_canonicalization; it reproduced the original XGBoost benchmark profile before the dependent battery was accepted. CPMB/PTED manifests independently record their code, estimator, environment, and output hashes. Inference latency, throughput, and live deployment performance were not evaluated.

The exact complete Python 3.12.3 Linux/aarch64 environment is recorded in requirements-lock-primary-linux-aarch64.txt and was reverified byte-for-byte for the canonical evidence bundle. requirements-lock.txt preserves the later Windows diagnostic environment for provenance only, and requirements.txt remains the portable minimum-version installer.

Controlled Policy-Mutation Boundary Test

code/28_counterfactual_policy_mutation_benchmark.py creates CPMB, freezes two synthetic rule-generator families, and computes the exact $2^3$ model-conditional counterfactual Shapley oracle before estimator development. code/29_pted_family_a_development.py develops and freezes PTED on Family A only. code/30_pted_family_b_evaluation.py performs the held-out Family B evaluation with the frozen estimator. Scripts 31 and 32 add explicitly secondary post-outcome diagnostics and a split/identifier leakage audit without refitting the estimator or changing a locked statistic. After the Family B outcome, scripts 33 and 34 evaluate the unchanged estimator under a third, independently seeded mixed-support geometry. The Family C adapter, lineage, statistics, thresholds, and execution contract were hash-frozen before its one-shot outcome; it is a prospective extension of its own test, not a retroactive preregistration of Family B. Script 35 performs a separate post-outcome split and identifier audit for the frozen Family C inputs. It neither refits PTED nor changes the negative primary decision, and it does not establish realism or production external validity.

The final Family B decision is negative. Median TV is 0.1332 versus 0.2463 for the strongest frozen baseline, but the bootstrap upper 95% bound is 0.2406 rather than at most 0.20, and paired median improvement is 0.0755 rather than at least 0.10. Family C is also negative: median TV is 0.2456 (bootstrap interval 0.1746--0.3209), paired median improvement is 0.0462 with one-sided $p=0.35683$, and degradation from Family A is 0.0899. It fails five locked gates while producing 0/72 control false positives. Exact-repeat hashes match for both evaluation families. The post-outcome Family C audit found zero exact cross-split feature overlap, zero within-split duplicates, zero rerandomized-alias overlap, and no forbidden identity or oracle feature among PTED's 20 inputs across 48 policy-evaluation units. results/cpmb_pted/ contains the locked protocol snapshots, generator and development manifests, frozen estimator, Family B and Family C scenario/control outputs, primary results, diagnostics, and family-specific leakage audits. All contents are synthetic or aggregate; no enterprise event, identifier, rule, or feature value enters CPMB.

Reproduce the staged benchmark in separate output directories:

python code/28_counterfactual_policy_mutation_benchmark.py --outdir <generator-out> --device cpu
python code/29_pted_family_a_development.py --generator code/28_counterfactual_policy_mutation_benchmark.py --baseline-results <generator-out>/family_a_development_results.csv --outdir <development-out> --device cpu
python code/30_pted_family_b_evaluation.py --generator code/28_counterfactual_policy_mutation_benchmark.py --development code/29_pted_family_a_development.py --estimator <development-out>/pted_frozen_estimator.json --baseline-summary <generator-out>/family_b_baseline_gate_summary.json --outdir <family-b-out> --device cpu
python code/31_pted_postoutcome_diagnostics.py --scenarios <family-b-out>/family_b_scenario_results.csv --primary-results <family-b-out>/primary_results.json --outdir <diagnostic-out>
python code/32_cpmb_leakage_audit.py --generator code/28_counterfactual_policy_mutation_benchmark.py --development code/29_pted_family_a_development.py --outdir <leakage-audit-out>
python code/33_pted_family_c_prospective_evaluation.py --family-b-evaluation code/30_pted_family_b_evaluation.py --generator code/28_counterfactual_policy_mutation_benchmark.py --development code/29_pted_family_a_development.py --estimator <development-out>/pted_frozen_estimator.json --baseline-summary <generator-out>/family_b_baseline_gate_summary.json --outdir <family-c-out> --device cpu
python code/35_cpmb_family_c_leakage_audit.py --family-c-evaluation code/33_pted_family_c_prospective_evaluation.py --generator code/28_counterfactual_policy_mutation_benchmark.py --development code/29_pted_family_a_development.py --outdir <family-c-leakage-audit-out>

The public rerun can verify the released results, but it cannot recreate the original outcome-blind timing after an outcome has been published. The locked snapshots and content hashes document the Family B separation and Family C's candidate--review--decision--protocol sequence; neither is represented as an externally timestamped preregistration.

Reproducing With Authorized Processed Data

The scripts below require the controlled processed dataset at data/processed/threat_five_class.csv. That file is not included in this public artifact by default. Use these commands only after placing an institutionally approved copy of the processed dataset at that path.

Install dependencies:

python -m pip install -r requirements.txt

For exact inspection of the reported outputs, use the platform-labelled Linux/aarch64 lock. The Windows lock is historical diagnostic provenance; neither lock file is a cross-platform installer.

Recreate the canonical experiment-file label surface and write an aggregate manifest (all controlled paths must remain outside this public repository). The expected checksum makes byte identity fail closed:

python code/02_build_threat_dataset.py --input <controlled-raw.csv> --output <controlled-output.csv> --manifest <aggregate-manifest.json> --label-surface canonical --expected-sha256 1BE9896A996DD58A582D94319180405A8559AB5193EF692BD8D2D2D614693724

The canonical surface stores raw block under the internal alias Deny, matching the controlled file used for the reported experiments. To create a release-labelled derivative instead, use --label-surface release; that derivative reports Block and is not claimed to have the canonical byte checksum.

Smoke-test the schema and parser on the public anonymized sample:

python code/03_benchmark_baseline.py --dataset threat --feature-set core --models "Decision Tree" --sample-per-class 20 --data-override data/processed/public_anonymized_sample_1000.csv --random-state 42 --tag public_sample_smoke --output-dir results/public_sample_smoke

The command above is the supported public-sample smoke test. Other scripts require the controlled processed dataset because the public sample deliberately tokenizes sensitive numeric values; do not rename or substitute the public sample to regenerate manuscript metrics or explanation artifacts.

Run the core holdout benchmark:

python code/03_benchmark_baseline.py --dataset threat --feature-set core --models "Decision Tree,Random Forest,Extra Trees,XGBoost,LightGBM,CatBoost" --random-state 42 --tag baseline --output-dir results

Run the no-threat-descriptors ablation:

python code/03_benchmark_baseline.py --dataset threat --feature-set core --models "Decision Tree,Random Forest,Extra Trees,XGBoost,LightGBM,CatBoost" --exclude-features "Threat/Content Name,Threat/Content Type,Severity" --random-state 42 --tag no_threat_descriptors --output-dir results

Run the no-threat-descriptors top-model CV:

python code/03_benchmark_baseline.py --dataset threat --feature-set core --models "Extra Trees,XGBoost,LightGBM,CatBoost" --exclude-features "Threat/Content Name,Threat/Content Type,Severity" --cv --random-state 42 --tag no_threat_descriptors_top_cv --output-dir results

Run the matching core top-model CV:

python code/03_benchmark_baseline.py --dataset threat --feature-set core --models "Extra Trees,XGBoost,LightGBM,CatBoost" --cv --random-state 42 --tag top_cv --output-dir results

Run the policy-included upper-bound comparison:

python code/03_benchmark_baseline.py --dataset threat --feature-set with_policy --models "Decision Tree,Random Forest,Extra Trees,XGBoost,LightGBM,CatBoost" --random-state 42 --tag with_policy --output-dir results

Generate the SHAP and LIME explanation artifacts used for the manuscript figures:

python code/04_xai_explanations.py --data data/processed/threat_five_class.csv --outdir results/xai

The XAI renderer uses 8-point Lato Regular at normal weight for axis and applicable colorbar-axis titles and 8-point Inter Regular at normal weight for axis tick/feature labels; no internal label is bold. The summary beeswarm uses one constant point hue because the encoded categorical values are nominal, so no ordinal High/Low feature-value legend is shown. Set LEAF_LATO_REGULAR and LEAF_INTER_REGULAR to explicit font-file paths when those faces are not discoverable through the host font registry. The resolved filenames, SHA-256 hashes, and locked sizes are recorded in xai_generation_summary.json. The locked evidence environment uses SHAP 0.51.0 with XGBoost 3.2.0; older SHAP releases that cannot parse XGBoost's multiclass vector base score are not supported.

Run the additional validation checks:

python code/05_q1_validation_extensions.py --data data/processed/threat_five_class.csv --outdir results/extensions

Run the strengthening checks:

python code/06_strengthening_validation.py --data data/processed/threat_five_class.csv --outdir results/strengthening

If XGBoost is installed with CUDA support:

python code/06_strengthening_validation.py --data data/processed/threat_five_class.csv --outdir results/strengthening --device cuda

Run duplicate-aware grouped split robustness:

python code/07_duplicate_group_robustness.py --data data/processed/threat_five_class.csv --outdir results/duplicate_group_robustness

Run policy-context held-out robustness:

python code/08_policy_context_robustness.py --data data/processed/threat_five_class.csv --outdir results/policy_context_robustness

Render the held-out named-rule audit figure from the saved aggregate CSV, without fitting a model:

python code/21_render_heldout_context_audit.py --aggregate results/policy_context_robustness/policy_context_heldout_aggregate.csv --output results/policy_context_robustness/fig_heldout_policy_context_audit.png --metadata results/policy_context_robustness/fig_heldout_policy_context_audit.metadata.json

The held-out-context figure preserves the three reported settings, their support-weighted accuracy and observed-class macro-F1 values, and their error counts. Renderer v2.0 uses the same visual system as the other validation figures: blue accuracy circles, teal macro-F1 squares, coral error markers, gray connectors, alternating row bands, and dashed grids. Lato is used for panel and axis typography, while non-bold internal text uses Inter; explicit size locking keeps axis titles at 8 pt and ticks/categories at 7 pt.

Run operational review-queue and unseen-context audit:

python code/09_operational_review_and_context_audit.py --data data/processed/threat_five_class.csv --outdir results/operational_review_context_audit --n-estimators 300 --max-contexts-per-column 6

Run model-independent policy-action entropy audit:

python code/10_policy_action_entropy_audit.py --data data/processed/threat_five_class.csv --outdir results/policy_action_entropy_audit

Run Q1 audit revision baselines and bootstrap intervals:

python code/11_q1_audit_revision_checks.py --data data/processed/threat_five_class.csv --outdir results/q1_audit_revision --confusions results/extensions/q1_validation_confusions.csv --fit-xgb --bootstrap-iters 1000

If the controlled processed dataset is not available and only aggregate confusion-cell CI outputs are needed:

python code/11_q1_audit_revision_checks.py --outdir results/q1_audit_revision --confusions results/extensions/q1_validation_confusions.csv --confusion-only

Run the additional robustness and audit experiments:

The public UCI Internet Firewall dataset is not duplicated in this repository. Download log2.csv from the official UCI Internet Firewall Data page (DOI: 10.24432/C5131M; CC BY 4.0), place it at data/uci_internet_firewall/log2.csv, and retain the required UCI attribution. The canonical input SHA-256 is 0b42e7eb9a4d7c314f65810c447ebf7f09d5e3e106ec14d36950a9ebac61e9c0. The local data/uci_internet_firewall/ directory is ignored to prevent an accidental data commit.

python code/12_repeated_seed_robustness.py --data data/processed/threat_five_class.csv
python code/13_classweight_sensitivity.py --data data/processed/threat_five_class.csv
python code/14_sourceport_nearduplicate_ablation.py --data data/processed/threat_five_class.csv --outdir results/q1_audit_revision
python code/15_forward_chaining_chronological.py --data data/processed/threat_five_class.csv --outdir results/q1_audit_revision
python code/16_uci_leaf_instantiation.py --data data/uci_internet_firewall/log2.csv --outdir results/uci_leaf_instantiation --seed 42
python code/17_category_code_order_sensitivity.py --data data/processed/threat_five_class.csv --outdir results/q1_audit_revision
python code/18_full_feature_lookup_baseline.py --data data/processed/threat_five_class.csv --outdir results/q1_audit_revision
python code/22_reviewer_revision_sensitivity.py --data data/processed/threat_five_class.csv
python code/24_rule_context_regime_audit.py --data data/processed/threat_five_class.csv

Render the manuscript-facing descriptive, holdout/CV, feature-group, and temporal/uncertainty figures from the saved aggregate evidence:

python code/19_render_manuscript_figures.py --processing-manifest data/processed/threat_dataset_processing_manifest.json --core-holdout results/baseline_benchmark_threat_core_baseline.csv --no-threat-holdout results/baseline_benchmark_threat_core_no_threat_descriptors.csv --core-cv results/baseline_benchmark_threat_core_top_cv_cv_summary.csv --no-threat-cv results/baseline_benchmark_threat_core_no_threat_descriptors_top_cv_cv_summary.csv --strengthening results/strengthening/strengthening_summary.csv --rolling-origin results/q1_audit_revision/forward_chaining_chronological.csv --bootstrap-ci results/q1_audit_revision/xgb_fixed_model_bootstrap_ci.csv --outdir results/manuscript_figures

The descriptive, holdout/CV, feature-group, and temporal/uncertainty renderers use Lato Regular at normal weight for axis labels and record the resolved font hash in manuscript_figure_render_metadata.json. In the manuscript-facing class-distribution and combined holdout/CV figures, axis titles are locked at 8 points and axis tick/category labels at 7 points; the feature-group figure retains its prior sizes. The temporal/uncertainty renderer writes both a 600-dpi PNG and a vector PDF. LEAF_LATO_REGULAR may be used to provide an explicit font path.

Render the manuscript methodology workflow from the aggregate cohort manifest:

python code/20_render_methodology_workflow.py --processing-manifest data/processed/threat_dataset_processing_manifest.json --outdir results/manuscript_figures

The workflow renderer uses Lato Bold for phase and box titles and Liberation Sans Regular for the single-line box summaries. All eight boxes share one 8.6-point title size and one 7.8-point summary size. The dashed Phase I, II, and III bands use a subtle progression of neutral-gray fills, while their inner boxes retain very light blue, yellow, and green fills, respectively; burgundy phase labels, navy box titles, purple summaries, and neutral connectors preserve a consistent high-contrast hierarchy. The release figure records the resolved filenames, SHA-256 hashes, sizes, and palette in methodology_workflow_render_metadata.json; exact pixel reproduction requires those recorded faces. The renderer applies a content-aware 600-dpi PNG crop with a 12-pixel safety margin.

Public Package Maintenance

The main branch is the current public package. Before updating it:

• Confirm that the GitHub URL matches the public repository.
• Confirm that TRACEABILITY.md still maps the manuscript tables and figures to the final public outputs.
• Confirm that the controlled processed-file checksum is unchanged or update the checksum in README.MD, MANIFEST.md, TRACEABILITY.md, and data/processed/README.md.
• Confirm that data/processed/public_anonymized_sample_1000.csv matches its audit checksum and contains only anonymized sample values.
• Confirm that data/processed/ does not contain the full event-level firewall CSV unless institutional public-release authorization has been obtained.
• Confirm that no raw enterprise logs, manuscript drafts, private review notes, or local credentials are included.
• Keep the manuscript's Data availability statement aligned with this repository's actual public contents and controlled-data boundary.

License

This project's code and aggregate output artifacts are released under the MIT License (see LICENSE). The public anonymized 1,000-row sample is included as a limited reproducibility aid for schema inspection and smoke testing, not as a general-purpose benchmark dataset. The raw and full event-level processed enterprise firewall data are not covered by this license and remain controlled-access institutional material, as described in the Data availability section.