How it reads, and how often it is wrong
Keystone does not summarise a paper and it does not ask a model what a citation meant. It looks for specific phrases in specific grammatical positions, and when it does not find one it says nothing at all.
Read the source, not the picture
Papers are parsed from the LaTeX arXiv ships alongside the PDF. That is why a quote here is the sentence the author typed, with its citation keys intact, rather than whatever a column-splitting text extractor made of a two-column page.
A cue decides, and shows its working
“Following Ba et al.” and “unlike Ba et al.” are opposite statements about the same work. The phrase that decided each reading is shown beside it, so a wrong reading is visible rather than load-bearing.
Pin it, or refuse to point
Each sentence is located in the PDF and stored as pixel rectangles, verified by re-reading the page geometry rather than trusting the matcher. Where a sentence cannot be located, nothing is drawn.
Scored on other people’s labels
Accuracy measured on the papers a rule was written against is not accuracy, it is a restatement. So the cue grammar is scored against public benchmarks it was never tuned on, annotated by other people, on papers outside this library.
Loading the evaluation…
Why the coverage is low on purpose. Most citations carry no cue phrase at all — they are a name in a list of names. A system that guessed at those would be right about as often as the base rate and would put a confident label on a sentence that does not support one. Silence is the correct output for a sentence that says nothing checkable, and it is the majority of them.
What that costs. Macro-F1 punishes abstention exactly as hard as error, so on these benchmarks it is low against neural classifiers around 0.84. That is the honest trade and not a rounding error: this is not the system to use if you need a label on every citation. It is the one to use if a label needs to be checkable.
What it will not tell you
- Whether a paper is any good. Nothing here is a judgement; it is a report of what the paper said about other work and about itself.
- Anything about a paper that does not ship LaTeX source. Roughly one arXiv paper in ten does not, and those are absent rather than approximated.
- What a citation with no cue meant. Those rows are blank on purpose, and a blank row is not a claim that the citation is unimportant.
- Whether an assumption is reasonable. Only whether the sentence stating it offered a citation, its own evidence, or nothing.