Is This Slop? Detecting AI-Generated Content Without a Model

S at detecting LLM content. You troll Reddit, aching for the chance to throw down the “written by AI” hammer on authors you suspect are AI.

Actually, you’re not, and the research proves it.

I recently was permabanned from an unnamed subreddit for posting an article. The moderators flagged it as AI generated content and I was banned instantly.

What the moderators failed to notice was the article’s creation date:

A full two years before ChatGPT was available to the public.

This wasn’t slop — It was my writing.

Low precision, high recall

The problem with the current backlash against LLM-generated content is that the automatic tools to detect AI are often low precision and high recall: classifying everything as AI written for the maximum amount of “gotchas.”

To demonstrate, here’s an excerpt of my writing (pre chatbot days)

Each idea will need to be prioritized on two factors as mentioned, 
ease of execution and likelihood of success. 

There is no easy way to determine ease of execution or likelihood unfortunately, 
as each researcher and team is different.

Some considerations:
- How complex is the method or idea?
- Did this method show success in another area?
- Do I/we have any expertise in applying this method?

At the end of your prioritization process, you should have the ideas which 
have the highest likelihood of showing results, followed by the ease of execution. 
If you were to create a score for this (for sorting), it would be weighted more on
likelihood of success vs ease of execution.

Here’s the associated report from a popular AI detector.

This doesn’t mean the tells or models are useless. The goal here shouldn’t, and will likely never be, certainty. But, armed with more information, you might go from a wild guess to a little bit better than a coin flip.

In this article, I cover research highlighting signature patterns of LLM generated content, including:

1. Vocabulary tells
2. Rhetoric and sentence structure tells
3. Punctuation tells
4. Why even confident readers get it wrong
5. How language models developed these tells (the math)

After reading, you’ll be better armed to spot slop in the wild, along with the mathematical intuition behind why these patterns exist.

---

"Delve"

It wasn’t always suspicious. It was a slightly formal word that showed up every now and again. Once ChatGPT came out, it exploded. Now, everyone’s delving into something.

There are a couple interesting bodies of research behind what’s called “excess vocabulary.” Words that rarely show up in human writing but frequently in AI generated content.

In “Delving into ChatGPT usage in academic writing through excess vocabulary” by Kobak et al., researchers reviewed ~15 million PubMed abstracts from 2010–2024, and demonstrate that the appearance of LLMs led to an abrupt increase in “certain style words.”

We didn’t just start “delving.” LLMs started “delving” for us.

Researchers ran another version of the same test. In “Why Does ChatGPT ‘Delve’ So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models” by Juzek and Ward, the researchers identified “focal words” by comparing two corpora. The first was PubMed abstracts written from 2020-2024. The second was generated by having ChatGPT rewrite the same body of abstracts. The exact same words showed abrupt increases in usage in both of the corpora, suggesting that the patterns were a result of ChatGPT usage.

What this boils down to is: LLMs prefer certain words. This is most likely a result of human preferences in LLM post-training.

“Ooh, delve sounds nice, I like this answer better.”

These token probabilities differ from human usage probabilities in measurable ways. Seeing them could be a useful indicator of LLM use.

TLDR: Words to look out for

In isolation, not worrisome, but in combination or frequently, definitely a warning flag.

delve, boast, intricate, tapestry, realm, showcase, pivotal, 
underscore, meticulous, leverage, robust, seamless, testament, 
comprehensive, multifaceted, navigate, notably, interplay

Name preference

There’s also some research highlighting the preference for names, specifically related to co-occurring names. In “The Ghost Couple: Correlated LLM Name Priors and Their Haunting of the Web and Academic Publishing”, Michał Brzozowski and Neo Christopher Chung found that models develop “name priors,” repeatedly generating the same fictional identities.

These fictional identities are cited as co-authors in ghost written research papers which have real DOIs, among other AI generated content, like website templates. The authors argue that these name pairs can act as signatures for the underlying model.

Some frequently occurring names are below:

Underlying Model	Frequently Recurring Names
Claude	Elena Vasquez, Marcus Chen, Amara Okafor
GPT	Elara Voss
Gemini	Aris Thorne, Lena Petrova

There’s also some weak anecdotal evidence for other names, such as Kira, Priya, Mara, Aria and surnames beginning with Ok, like Okafor, Okonkwo

The name Marcus, and the surname Chen stand out particularly.

When prompting Claude Sonnet 5 with 'generate names for a software developer born in 1990'

The model returns:

Alex Chen
Dev Patel
Marcus Webb
Nadia Volkov
Lauren Okafor
...

🤣We get clear hits in the first set of responses.

Some food for thought. So, if you happen to stumble across a research paper with claims that are too good to be true, written by Marcus Webb, Elena Vasquez and Lena Chen, you’re probably reading ghost written slop.

Let’s now delve into some other tells.

---

Rhetoric and sentence structure tells

LLMs also prefer some structural linguistic patterns that are pretty easy to spot.

Contrastive phrasing

"It's not X — it's Y"

"It wasn't this — it was just this"

The post training method of choice for many foundation models, RLHF, rewards responses that human evaluators perceive as helpful. These contrastive phrases are highly correlated with explanatory writing. It gives a feeling of depth and conceptual understanding that human evaluators fall in love with.

The RLHF process is why it’s difficult to phase out with system prompts. This behavior has been baked in during post training and the LLM literally can’t stop doing it, even if you explicitly tell it not to.

This contrastive phrasing stylized into LLMs in post training leaks out everywhere, even where it doesn’t make any sense.

So the next time you see:

"This isn’t just an idea. It’s a revolution."

You can, probably, wave your slop flag safely.

Linguistic hedging

LLMs won’t ever fully commit to an idea if there’s the slightest chance of a counter argument.

Framing both sides of an issue and landing on a non-specific conclusion is risk-averse in a very specific, trainable way.

If the stylistic preference is “avoid conflict and overconfidence“, hedging is a very clear, rewardable signal from the perspective of a human evaluator. Thus, models exhibit a stereotypical pattern of hedging and assertion whitening, even when a more direct answer would be appropriate.

Some examples of hedges:

"This may indicate…"

"The evidence suggests that..."

""It is worth noting that..."

"This is a X, Y, and Z, however, A, B and C also exist"

"While this method offers several advantages, potential drawbacks should also be evaluated."

Hedging statements are rarely wrong, but they provide little in terms of actual opinion, increasing the appearance of nuanced understanding without increasing the precision of the answer.

From a language modeling perspective, this behavior is unsurprising. A cautious answer that avoids being obviously incorrect is often safer than a precise answer that risks disagreement, earning a higher reward in alignment training.

These hedges result in overly cautious, noncommittal writing that is easy to catch if you’re looking for it.

The rule of three

This is an effective writing technique that technical/business writers use frequently, so it’s weak pattern when observed in isolation.

LLMs love using three items, almost compulsively. This is likely an artifact of the training data used for large foundation LLMs. People do respond to it quite well.

Watch for it in lists, but also in single sentences: “fast, reliable, and secure.” “Clear, concise, and actionable.” If you can drop an item without losing detail, that’s a reasonable tell.

Sometimes “fast and actionable” beats “fast, actionable, and effective.”

Like every tell in this article, this isn’t evidence by itself. Human experts intentionally use triadic pattens because they work.

The real signal here comes from frequency. If every paragraph in the content you read contains balanced triplets, you’re observing another weak statistical feature that just nudges the probability toward machine generated text.

---

Punctuation tells

This one is easy and everyone knows it when they see it. The sacred em-dash.

—

LLMs have almost pathologically lost the ability to use the colon and the semi-colon in favor of the em-dash. It’s everywhere. So much so that OpenAI needed to put in a no em-dash preference option in ChatGPT.

Small-but-happy win:

If you tell ChatGPT not to use em-dashes in your custom instructions, it finally does what it’s supposed to do!

— Sam Altman (@sama) November 14, 2025

The em-dash still shows up quite heavily across foundation models.

I used to like em-dashes. They break up content and make a lot of sense when you need to briefly interrupt the train of thought. I tend to use a ton of asides in my writing. I’ve since replaced them (this content is an aside).

If you’re like me, and want to use em-dashes again, be fairwarned: your risk for being flagged increases dramatically.

(I even risked using them in this article)

---

Why even confident users get it wrong

In the paper, “Detecting AI-Generated Text: Factors Influencing Detectability with Current Methods” by Fraser, Dawkins, and Kiritchenko, the authors examined how accurately people could distinguish between human written and LLM generated text. Accuracy: 59%. Nearly a coin-flip.

The findings challenge the assumption that humans are reliable AI detectors. Even confident readers frequently misclassify both AI generated content as human written and human written content as AI generated.

Another study, “Ability of AI detection tools and humans to accurately identify different forms of AI generated written content” by Cheng et al., puts the average human rating accuracy at just 19%.

This is a detection paradox.

The better language models become at writing like a polished human, the more indistinguishable they’ll be from actual humans. This may explain the why human accuracy decreases for both 100% AI generated text (10%) and 100% human generated text (17%).

Detecting stylistic patterns is one, small piece of the puzzle.

Even if you are pretty decent at detecting AI content today, ultimately, human intuition is subpar at detecting AI and LLMs will continually advance until they are indistinguishable from polished prose.

What’s also somewhat unnerving is that I find LLM style writing popping up naturally as I write, as if reading LLM generated content affects my own writing style. Are we becoming more like ChatGPT or is ChatGPT becoming more like us? Am I a token machine?

The evidence suggests that it may be possible.

How language models developed tells

How does a language model develop characteristic traits, strong enough that detecting which version of the model generated the content is possible?

Base models trained purely on token prediction typically don’t have these tics. As referenced previously, they are an artifact of alignment training: specifically secondary post training using SFT and RLHF.

Stage 1: Supervised Fine Tuning (SFT) sets the base distribution

Before any preference optimization happens, the base model goes through supervised fine-tuning (SFT), where it’s trained to imitate a curated set of demonstration outputs, which include human written and human edited responses.

The objective function is:

$L_{\mathrm{SFT}}(\theta) = -\mathbb{E}_{(x,y)\sim D_{\mathrm{demo}}} \left[ \sum_{t=1}^{T} \log \pi_{\theta}(y_t \mid x, y_{<t}) \right]$

where $D_{demo}$​ is the demonstration dataset. Simply token level cross entropy applied to human demonstrations.

The model is learning to reproduce the conditional distribution of tokens found in $D_{demo}$, which is the most important detail from this phase.

This dataset is written or heavily edited by a small, non-representative pool of annotators, frequently working from internal style guides. If that pool skews toward writers who reach for “delve” instead of “explore.” The model isn’t learning “human writing”, it’s learning that pool’s writing, at whatever scale $D_{demo}$ gets used.

This matters quantitatively, not just anecdotally.

If $P_{demo}(“delve”)$ is even 5–10x the base rate of “delve” in general web text, cross-entropy training will push $\pi_\theta(delve)$ toward $p_{demo}$, not toward the pretraining corpus’s rate.

SFT is a small dataset (thousands to low millions of examples) relative to pretraining (trillions of tokens), but it’s the last thing trained on before alignment.

Next token distributions are highly sensitive to fine-tuning data. It’s the same mechanism that lets a few thousand instruction examples completely change a base model’s behavior.

This might also explain name priors, with specific name priors being tied to different models.

If $D_{demo}$ contained large numbers of example dialogues involving fictional characters, annotator habit or usage could have had a measurable effect on the probability mass for names. There’s no preference ranking for “Marcus Chen” is better than “Frank Smith.” Given that, my guess is SFT explains the mysterious name priors.

Stage 2: RLHF/RLAIF sharpens whatever survived SFT

You may have participated in this process while interactively chatting with an LLM (the provider asks you to pick which answer you like better).

At a high level, here is how it works.

RLHF optimizes the following objective function:

$\max_{\pi_\theta} \; \mathbb{E}_{x \sim D, \, y \sim \pi_\theta(\cdot|x)} \big[ r_\phi(x, y) \big] – \beta \, D_{KL}\big[\pi_\theta(y|x) \,\|\, \pi_{ref}(y|x)\big]$

The model is trained on human preference pairs, where the language model generates multiple candidate responses (e.g $y_n$) to the same prompts and a human (or another language model) ranks them.

The reward function, $r_\phi$ is learned via the Bradley-Terry preference model.

Given preferred response $y_w$ and rejected response $y_l$:

$P(y_w \succ y_l \mid x) = \sigma\left(r_\phi(x,y_w)-r_\phi(x,y_l)\right)$

The loss is the negative log-likelihood:

$\mathcal{L}_{RM} = -\mathbb{E}_{(x,y_w,y_l)\sim D} \left[ \log \sigma \left( r_\phi(x,y_w)-r_\phi(x,y_l) \right) \right]$

$r_\phi$ rewards which response was clicked, in the hopes that it reflects actual user preference

The KL term keeps the policy $\pi_\theta$ (the new, aligned model), from drifting too far from the reference model $\pi_{ref}$ (a frozen model as produced by the pretraining process).

In RLHF, policy is optimized via standard Proximal Policy Optimization (PPO).

Many production models skip the explicit reward model and use Direct Preference Optimization (DPO) instead, which is a reparameterization of the KL constrained maximization above.

What this boils down to is:

1. The LLM generates candidate responses to prompts
2. The human evaluator selects which is best (under time constraints)
3. The optimization applies the reward gradient (as specified in the loss) to the model in a way that keeps the original tendency but increases the likelihood of “preferred” responses (token probabilities)

RLHF explains why LLMs hedge, and it’s not a stylistic artifact.

Hedging is an artifact of the policy that survives optimization when you look at what $r_\phi$ is actually fit to reward

A rater comparing two candidate answers under time pressure, with no ground truth to check against, penalizes confident wrongness more than they reward confident correctness. A hedged answer ("the evidence suggests X, though Y is also plausible") is harder to judge as incorrect than a committed one, even when the committed answer is right. The hedged answer simply looks more right in comparison.

The stylistic biases and tells are indeed a product of these post training processes.

Wrapping up

The patterns in this article are exactly that: patterns.

Weak statistical signals that, when combined, can nudge your confidence one way or another. They aren’t fingerprints, and they certainly aren’t proof.

Technical writers hedge and plenty of people have been using em-dashes since long before ChatGPT existed (myself included, I miss them).

Now, armed with awareness of some of these tics, you’ll probably catch more and more LLM generated content in the wild. But you’ll rarely be certain, and that uncertainty will only increase as models continue to improve.

Maybe, we need to be asking a different question:

Is what I’m reading actually valuable?

---

Fun fact: I included (3) very specific, human written examples of AI sloppiness in this article on purpose. Can you find them all? 🥚🔍

---

[1] Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2024). Delving into ChatGPT usage in academic writing through excess vocabulary. ScienceOpen. Preprint. https://doi.org/10.48550/arXiv.2406.07016

[2] Juzek, T. S., & Ward, Z. B. (2025). Why Does ChatGPT “Delve” So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models. In Proceedings of the 31st International Conference on Computational Linguistics (COLING 2025) (pp. 6397–6411). Association for Computational Linguistics.

[3] Fraser, K. C., Dawkins, H., & Kiritchenko, S. (2025). Detecting AI-Generated Text: Factors Influencing Detectability with Current Methods. Journal of Artificial Intelligence Research, 82, 2233-2278.

[4] Cheng A, Lin Y, Reedy G, Joseph C, Wirkowski S, Mallette V, Nagesh V, Krieser D, Calhoun A. (2025). Ability of AI detection tools and humans to accurately identify different forms of AI-generated written content. Adv Simul (Lond). 2025 Nov 22;10(1):66. doi: 10.1186/s41077-025-00396-6. PMID: 41272826; PMCID: PMC12752165.

[5] Brzozowski, M., & Chung, N. C. (2026). The Ghost Couple: Correlated LLM Name Priors and Their Haunting of the Web and Academic Publishing. arXiv:2606.02184.

[6] Behrens, J. (2025). AI names. seehuhn.de.
https://www.seehuhn.de/blog/ai-names/