How to Catch AI-Fabricated References Before You Submit: arXiv's Ban and a .bib Checker

BlogEasyLaTeXPublished

To catch AI-fabricated references before you submit, work in three passes: list only the entries your text actually cites, check their DOIs and titles against Crossref in bulk, then open every flagged entry and verify it by hand. A clean compile proves nothing here. A made-up .bib entry typesets into the reference list like any other, and the log stays silent. You should also search your source for sentences a language model left behind, such as "here is a summary" or "fill in the real numbers". This article includes a copy-ready Python script and search commands; every example was run before publication.

What arXiv now does about fabricated references

In May 2026, Thomas Dietterich, chair of arXiv's computer science section, set out the penalty on X and later confirmed it to Nature: if a submission contains "incontrovertible evidence that the authors did not check the results of LLM generation", all authors receive a one-year ban. After the ban, any new arXiv submission must first be accepted at a reputable peer-reviewed venue.

He gave two kinds of evidence as examples:

  • Hallucinated references.
  • Meta-comments from the LLM, such as "here is a 200 word summary; would you like me to make any changes?" or "the data in this table is illustrative, fill it in with the real numbers from your experiments".

According to 404 Media, a moderator must first document the problem and the section chair must confirm it before a ban is imposed, and authors can appeal. The rule was announced through social media and press interviews. On arXiv's own moderation page, the relevant basis is that authors take full responsibility for all content, including "incorrect references" produced by generative AI, and that "general scrupulousness and care of preparation are required".

For a sense of scale, Zhao et al. (with arXiv founder Paul Ginsparg among the authors) audited 111 million references in 2.5 million papers from arXiv, bioRxiv, SSRN and PubMed Central. They conservatively estimate 146,932 hallucinated citations across those four corpora in 2025. They also estimate that about 78.8% of non-existent citations passed arXiv moderation, and that 85.3% of such citations in bioRxiv preprints survived into the published versions. The study is a preprint, and its method only detects references with non-existent titles. Real papers cited for claims they never make are harder to find automatically.

What fabricated references look like

TypeSignsCaught by automatic checks?
InventedPlausible title and authors, but the work cannot be found; the DOI does not resolveYes: the DOI returns 404 or the title has no match
SplicedA real DOI that points to a different paper, often with well-known real authorsYes: the DOI's title differs from the entry
MisattributedThe work exists, but the year, venue, or volume is wrongPartly: compare year and other fields
Unsupported claimThe work is real but does not say what you cite it forNo: you have to read the source

A single wrong year is usually a metadata problem, not fabrication. Merged versions in Google Scholar and faulty extraction from PDFs both produce such differences.

Step 1: list only the entries you actually cite

A .bib file often carries entries from older projects that never appear in the PDF. Compile once, then pull the cited keys from the .aux file:

Shell
grep -o '\\citation{[^}]*}' main.aux | sed 's/\\citation{//; s/}//' | tr ',' '\n' | sort -u

This works for classic BibTeX (\bibliography{refs}). With biblatex and Biber, the .aux file does not hold these records; check the whole .bib, or look at the entries Biber wrote to the .bbl file. Keys that are cited but missing from the .bib show up as Citation undefined in the log; see how to check citations in a LaTeX paper.

Step 2: check entries against Crossref in bulk

Crossref offers a free public REST API. The script below uses only the Python standard library and runs two checks per entry:

  • With a DOI: query api.crossref.org/works/{DOI}. An unknown DOI returns 404; a known one is compared by title with your .bib entry.
  • Without a DOI: run a bibliographic search on title, authors, and year, and compare the closest result.
  • Once the title similarity reaches 0.9, the year is compared as well.

Save it as checkbib.py and replace MAILTO with your email address. Crossref asks clients to include contact details, which routes requests to its more reliable "polite" pool.

Python
#!/usr/bin/env python3"""Check BibTeX entries against Crossref. Usage: python3 checkbib.py refs.bib [main.aux]"""import json, re, sys, time, difflib, urllib.parse, urllib.request​MAILTO = "you@example.org"  # replace with your email (Crossref polite pool)API = "https://api.crossref.org/works"​def entries(text):    for m in re.finditer(r"@(\w+)\s*\{\s*([^,\s]+)\s*,(.*?)\n\}", text, re.S):        if m.group(1).lower() in ("comment", "string", "preamble"):            continue        f = dict((k.lower(), re.sub(r"[{}\s]+", " ", v).strip())                 for k, v in re.findall(r"(\w+)\s*=\s*[{\"](.*?)[}\"]\s*,?\s*$", m.group(3), re.M))        yield m.group(2), f​def get(url):    req = urllib.request.Request(url, headers={"User-Agent": f"checkbib/1.0 (mailto:{MAILTO})"})    with urllib.request.urlopen(req, timeout=30) as r:        return json.load(r)["message"]​def sim(a, b):    norm = lambda s: re.sub(r"[^a-z0-9 ]", "", s.lower())    return difflib.SequenceMatcher(None, norm(a), norm(b)).ratio()​bib = open(sys.argv[1], encoding="utf-8").read()cited = Noneif len(sys.argv) > 2:  # only check keys that BibTeX actually used    aux = open(sys.argv[2], encoding="utf-8").read()    cited = {k.strip() for g in re.findall(r"\\citation\{([^}]*)\}", aux) for k in g.split(",")}​for key, f in entries(bib):    if cited is not None and key not in cited and "*" not in cited:        continue    title, doi = f.get("title", ""), f.get("doi", "")    try:        if doi:            w = get(f"{API}/{urllib.parse.quote(doi)}")            s = sim(title, (w.get("title") or [""])[0])            status = "OK" if s >= 0.9 else "CHECK: DOI exists but title differs"        else:            q = urllib.parse.urlencode({"query.bibliographic": f"{title} {f.get('author', '')} {f.get('year', '')}", "rows": 1})            items = get(f"{API}?{q}")["items"]            w = items[0] if items else {}            s = sim(title, (w.get("title") or [""])[0])            status = "FOUND (add DOI?)" if s >= 0.9 else "CHECK: no close match"    except urllib.error.HTTPError as e:        w, s, status = {}, 0, "CHECK: DOI not found" if e.code == 404 else f"ERROR {e.code}"    year = (w.get("issued", {}).get("date-parts") or [[None]])[0][0]    if status.startswith(("OK", "FOUND")) and f.get("year") and str(year) != f["year"]:        status = "CHECK: year differs"    print(f"{status:38} {key:22} sim={s:.2f} crossref_year={year} doi={w.get('DOI', doi or '-')}")    time.sleep(1)

Run it like this:

Shell
python3 checkbib.py refs.bib main.aux   # only entries cited in the text (BibTeX)python3 checkbib.py refs.bib            # every entry in the .bib

The script sends one request per second, so a few dozen references take a minute or two. It parses the .bib with regular expressions and handles the usual field = {value}, layout. If a value has deeply nested braces or an unusual line layout, the title may not be read, and the entry is flagged for checking.

What the output looks like

We built a test .bib with five entries: three real papers, one invented paper (authors Smith and Chen, with a DOI in the Nature Machine Intelligence format), and one spliced entry that borrows the DOI of the 2015 Nature review by LeCun et al. and pairs it with a title that does not exist. Compiled with pdfLaTeX and BibTeX, all five appeared in the reference list and the log had no warnings. Output of the script on 17 November 2026:

Plain text
CHECK: year differs                    vaswani2017            sim=1.00 crossref_year=2025 doi=10.65215/pc26a033OK                                     he2016                 sim=1.00 crossref_year=2016 doi=10.1109/cvpr.2016.90OK                                     lecun2015              sim=1.00 crossref_year=2015 doi=10.1038/nature14539CHECK: DOI not found                   smith2023fake          sim=0.00 crossref_year=None doi=10.1038/s42256-023-09999-xCHECK: DOI exists but title differs    mixed2015              sim=0.32 crossref_year=2015 doi=10.1038/nature14539

Both fabricated entries were caught. The first line deserves attention: Attention Is All You Need is a real paper, but the NeurIPS proceedings have no Crossref DOI, so the title search matched a different record from 2025 and the year check fired. This is common for conference papers, arXiv preprints, and book chapters. Use the output to decide what to check first. A flag does not mean fabrication, and OK only means the title, DOI, and year agree.

Step 3: verify flagged entries by hand

For each CHECK entry:

  1. Open https://doi.org/ followed by the DOI and compare the landing page's title and authors with your entry.
  2. Without a DOI, search the exact title in quotes on Google Scholar, Semantic Scholar, DBLP (computer science), PubMed (biomedicine), or arXiv. For conference papers, trust the conference site or DBLP.
  3. If you find it, fix the entry: add the correct DOI, year, and venue. Export BibTeX from the publisher page instead of editing by hand.
  4. If you cannot find it, delete or replace it. Do not keep a reference that "probably exists". If it supports a key claim, go back to sources you have actually read.
  5. Check the claim. Open the paper and confirm it supports the sentence you wrote. No script can do this; the record sheet in how to verify a research paper helps.

Put every entry that an AI tool suggested or wrote into your .bib through all five steps, whatever the script says.

Step 4: remove sentences left by the model

The second kind of evidence arXiv lists is meta-comments. They tend to sit at the start or end of paragraphs pasted back into LaTeX, in table notes, and in captions. From the project root, run:

Shell
grep -rniE --include='*.tex' \  'would you like me|here is (a|an|the) [^.]*(summary|revised|version)|as an ai( language model)?|illustrative|placeholder|fill (it )?in|\[citation needed\]|\bXX(\.X+)?\b|\bTODO\b|lorem ipsum' .

On a test file, this command found "Here is a 200-word summary… Would you like me to make any changes?", "The data in this table is illustrative; fill it in with the real numbers.", the placeholder XX.X\%, [citation needed], and a % TODO comment, without flagging the ordinary sentence. For manuscripts in other languages, add the equivalent phrases.

Review each hit: words like illustrative or placeholder can be legitimate. Comments starting with % do not appear in the PDF, but arXiv publishes your source, so remove them before submitting.

Finally, check that every number in your tables and figures comes from your experiment records. A model filling in a table may insert plausible sample values. grep cannot catch that; only a comparison with the raw data can.

Doing this in EasyLaTeX

  • The AI panel on the right of the editor (Chat mode) can read the .tex and .bib files in your project. Ask it to list the cited keys, point out entries without a DOI or with missing fields, and find sentences that look like meta-comments. The panel has no web access, so it cannot tell whether a paper exists.
  • To verify whether a reference exists, use the Academic Agent on the platform home page. It can search the web and look up literature, which suits checking flagged entries one by one.
  • When you let Agent mode edit the .bib, you can accept or reject each change. Before adding a DOI or changing a year, open the publisher page yourself.

The Python script runs locally; download your project source from EasyLaTeX and run it there. Whatever tools you use, all authors remain responsible for the references.

Pre-submission checklist

  1. The project compiles and the log has no Citation undefined.
  2. Every cited reference has been checked against Crossref, and each CHECK entry was verified by hand.
  3. Every reference suggested or written by AI was opened and confirmed.
  4. You have read the passages that support your key claims.
  5. The source has no meta-comments, placeholder numbers, or TODOs.
  6. AI use is disclosed as the venue requires; see AI disclosure statements for research papers.

References