Evidence guide · LLM-agent safety
Beyond pass rates: what survives after the guardrail?
A source-led guide to two proposed structural tests for distinguishing measured improvement from hidden persistence in LLM-agent safety evaluation.
Status: Version 1 preprints · Not peer-reviewed · Empirical studies · Synthetic pilots · No production-agent validation claimed
The evaluation problem
A lower attack-success or policy-violation rate is useful evidence, but it does not by itself show why the rate fell. A safety intervention may have removed a reusable failure route, or it may have suppressed the measured examples while a related route remains available under different pressure. The two public preprints treat frequency metrics as necessary but incomplete and ask a structural follow-up question: which safe and unsafe behavioural patterns persist as adversarial pressure increases?
Two linked proposals
Topological robustness and anti-fragility
The first framework represents responses, decisions, and tool-call trajectories as a pressure-indexed structure. It proposes testing whether a safe core remains stable and whether controlled hardening improves the before/after structure on held-out attacks.
The reported pilot used 1,440 generated traces. Its stated scope is a controlled proof of concept with known ground truth, not external validation over production LLMs.
Pressure-filtered persistence and the Masking Index
The follow-up study proposes a diagnostic for the specific case in which development-suite frequency improves while held-out unsafe structure remains. The Masking Index is intended to separate apparent cleaning from structural hardening under stated correspondence and stability assumptions.
The reported instrument-validation study used 5,760 generated traces with 32 repetitions per cell. The paper says this validates the detector on planted synthetic ground truth, not predictive value on real agents.
How to read the evidence
| Question | Supported interpretation | Unsupported leap |
|---|---|---|
| Did measured failures become less frequent? | Use held-out frequency estimates and uncertainty intervals. | Assuming a lower rate proves the underlying route disappeared. |
| Did unsafe structure persist under pressure? | Use the proposed pressure-filtered summaries as a complementary diagnostic. | Treating a synthetic instrument result as production effectiveness. |
| Did hardening generalize? | Require held-out attacks, fixed correspondence rules, and prospective falsification conditions. | Reusing the development suite as proof of generalization. |
| Is the method operationally validated? | The checked records establish proposed methods and synthetic pilots. | Claiming external, production, regulatory, or peer-reviewed validation. |
What the public records establish
Crossref and Europe PMC currently index both titles, both DOI-version identifiers, Emanuel Shirbint and Alexander Rybalov as authors, and 2026 posting dates. The canonical research manifest labels both works as preprints, empirical studies, synthetic pilots, and public Version 1 records. Citation counts are omitted because registry coverage and counts are not reconciled into the public claim ledger.
These records do not establish peer review, production deployment, employment, advisory work, clinical use, or validated performance on real-world agents. Peer-reviewed work, conference work, preprints, conceptual research, production systems, employment, and advisory work remain distinct evidence classes.
Primary records
- A Topological Robustness and Anti-Fragility Framework for LLM Agents — Version 1 DOI record
- Europe PMC record PPR1248684
- Measuring What Survives: Pressure-Filtered Persistence and a Masking Index for LLM-Agent Safety Evaluation — Version 1 DOI record
- Europe PMC record PPR1253108
Evidence checked 11 August 2026 (Asia/Jerusalem). This asset is an evidence guide, not a claim of peer review or production validation.