The problem
Point a language model at a few days of network logs and it will hand you an incident report, complete with an ordered chain of stages, named accounts and machines and a confident summary. Whether any of it is true is a different question, and on real telemetry there is no way to settle it, because nobody has a list of which log lines were the intrusion. The only way I could think of to get that list was to write the logs myself.
How it works
The generator produces three log sources over a fixed window in a world of about fifty machines and sixty accounts, covering authentication attempts, process launches and network connections. A six-stage intrusion sits somewhere inside it, deliberately split so that no one source carries the whole story. The failed logins are in the auth log, the lateral movement shows up only in process ancestry, and the exfiltration is visible only in the network flows.
Most of the effort went into the background traffic rather than into the attack. A vulnerability scanner sweeps the network, people mistype passwords, nightly jobs archive the same shares the attacker steals from, and large uploads leave legitimately all day, so none of the attack's individual steps look unusual on their own. There is also a decoy, an innocent account and machine made to look busy at exactly the wrong moment, which is there to catch anything that mistakes coincidence for a chain.
The agent answers in JSON, and every claim has to cite the event ids it rests on, which is what makes grading mechanical rather than a matter of opinion. Two separate graders then ask two questions that usually get collapsed into one.
| Grader | Question | How |
|---|---|---|
| reconstruction | did it recover the planted chain | stages matched by evidence overlap, never by text, with key and supporting events scored apart |
| grounding | is the report honest about its own evidence | six mechanical checks per claim: the event exists, was read, the entities named appear in it, the numbers are supported, it is not the decoy, the ordering matches the timestamps |
I never average them together, because a report can recover the attack while citing the wrong events for it, or cite impeccably while describing something that never happened, and a single number would hide both cases.
It reports an intrusion whether or not there is one
| Condition | Runs | Reported an intrusion |
|---|---|---|
| no attack, across the size range | 15 | 14 |
| no attack, decoy present | 3 | 3 |
| no attack, neutral prompt | 3 | 3 |
| no attack, decoy present, neutral prompt | 3 | 3 |
| total | 24 | 23 |
Mean confidence when it reported one was 0.95. The prompt seemed the obvious thing to blame, since the original version is essentially a form with slots waiting to be filled in with attack stages. So I rewrote it as a neutral shift-triage task that gave both outcomes equal weight, added a control with the attack present to make sure the rewrite had not simply blinded it, and ran the whole thing again on the same seeds. The numbers came back the same.
What surprised me was where on the range this happens. I had assumed the invention was a symptom of being overwhelmed, but at 800 events, the smallest window and the one where the analyst is otherwise at its most accurate, clean data still produced an intrusion in all three runs.
A quieter attack is never found
All of that rested on a single planted chain, which left it ambiguous whether I was measuring the analyst or my own generator, so I built a second intrusion that inverts everything the first one relies on. Attack B begins on an ordinary workstation rather than the perimeter, involves no failed logins at all, uses a stolen administrator account instead of a guessed service account, writes no archive, and takes the data out as a drip of small flows rather than one bulk transfer. As a check on how different the two really are, a threshold detector that fires on failed-login bursts and outbound volume scores 0.128 on attack A and 0.000 on attack B.
Attack B was never found in any of the fifteen runs, at any window size, including the one where it makes up 3.7 % of an 899-event file.
That is not a reasoning failure, though. Handed attack B’s 33 events on their own, with the search taken out of the problem, the same model reconstructs the chain almost perfectly.
What it reported instead
Here is one run at the easiest setting, on those same 899 events. It came back with five stages at 0.95 confidence and 62 citations, every one of them pointing at a real event.
- “Brute force attack on administrative account adm.rivera”
- “Second attacker conducting brute force against i.keller”
- “Port scanning across internal hosts from 10.20.50.2”
- “Massive data exfiltration, approximately 1GB, to 192.0.2.135”
- “Large-scale exfiltration, over 2GB, to multiple external addresses”
All five are the background traffic I wrote to be unremarkable. The first two are people mistyping passwords, the third is the vulnerability scanner on its rounds, and the last two are the nightly cloud sync. What it never mentioned at any point was the mail client spawning a script host, the beacon leaving the workstation, the credential dump, the administrator account reaching a domain controller from a finance machine belonging to somebody else, or the slow drip of data going out.
That changes how I read the attack A numbers. Whatever competence showed up there seems to have been resting on a loud signature, twenty-four failed logins and a 434 MB transfer, and once the signature goes the detection goes with it at every scale, even though the reasoning is demonstrably unaffected.
The citations are real; the claims are not
Across all 69 runs the fabrication rate came out at 0.00, with every cited event id real and genuinely retrieved. I had expected invented citations to be the main failure and they never happened once. What happens instead is harder to catch, because the events do exist and simply do not say what the claim says they say. Hosts and accounts get named in the prose that appear in none of the cited events, and counts run past what the evidence can support. Anyone spot-checking three citations from a report like that would find all three genuine and conclude it was sound.
Baselines
| Baseline | Reconstruction | Grounding | What it establishes |
|---|---|---|---|
| oracle, attack events only | 0.955 | 0.838 | the ceiling |
| threshold detector | 0.121 | 0.877 | the two axes come apart |
| random citer | 0.000 | 1.000 | grounding alone means nothing |
| null, no attack planted | 0.000 | 0.278 | 96 % false positive rate |
The random citer is worth dwelling on. Fluent prose written over randomly chosen events passes every grounding check and scores a perfect 1.000, which is arguably the right result, since the check asks whether a claim is honest about its evidence rather than whether it is true, and a vague claim has nothing in it to contradict. The consequence is that a grounding score means very little on its own. The analyst managed 0.639 at the easiest setting, which puts it below a baseline that is not reading the data at all. The threshold detector sits in the opposite corner, barely reconstructing anything while being almost perfectly honest about the little it does claim.
Scope
None of this is realistic traffic, and I traded realism for an answer key on purpose, so the results should not be read as transferring directly to production logs. The generator and both graders also share an author, which for grounding is partly answered by a frozen set of 40 hand-labelled claims that the grader agrees with 95 % of the time at κ 0.77, and for reconstruction is not answered at all. With only three seeds per point most of the intervals are wide, so the two numbers I would actually stand behind are the ones that need no interval: 23 of 24, and no fabricated citations in 69 runs. Code, run files and the frozen validation set are in the repository.