Reddit AI Search Picks Already-Popular Comments, Audit Finds

Reddit AI Search Picks Already-Popular Comments, Audit Finds

Reddit's AI search leans toward comments that were already winning on the platform, according to an audit by University of Illinois Urbana-Champaign researchers described by Search Engine Journal. The team ran 10,000 questions through the feature three times each, examined the 30,000 resulting answers, and traced them back through 14.68 million comments drawn from 20 advice and support subreddits. The work is a preprint and its findings apply to that slice of Reddit.

A comment's vote ranking inside its thread carried the largest weight in the selection model. One standard deviation of ranking multiplied the odds of being chosen by 2.88, and the median selected comment sat at the 91st percentile for score in its thread, against the 45th percentile for comments the system passed over. Selected comments also arrived early, a median of 1.2 hours after the original post versus 5.9 hours for the rest, and 92 percent of them were direct replies to the post. Longer comments and comments with an external link did better too, with odds ratios of 1.79 and 2.25.

Language mattered on top of popularity. Formality, scored by a text classifier, was the strongest linguistic predictor: one standard deviation more formal meant 49 percent higher odds of selection. Prescriptive wording along the lines of should and must helped slightly. Markers of personal experience, such as first-person pronouns and past tense, cut the odds, as did supportive language. These associations shrank but did not vanish once the researchers controlled for a comment's score, position and age. The authors caution against reading this as advice on how to write, stating, "Our study is observational and should not be interpreted causally."

The audit also measured what happens to voice after selection. Across 1,000 queries, words like I and my made up 3.3 percent of the quoted comments but 0.06 percent of the answers built around them, which the authors describe as personal testimony being turned into general advice. When the same queries went through GPT-4o-mini and GPT-5 via OpenAI's API with web search on, only 18 of 4,932 citations pointed at Reddit, though that API setup differs from the ChatGPT app.

The researchers used an LLM to rewrite real posts into short queries covering ten large communities, including r/personalfinance and r/AskDocs, and ten small ones such as r/UKJobs and r/AusLegal. They spaced the three runs about five hours apart. Near-identical answers appeared when the system retrieved the same posts, suggesting that differences came from retrieval rather than generation. A typical answer drew from about seven communities. Reddit launched the feature as Reddit Answers in December 2024 and added it to unified search on May 26, 2026. The test queries came from posts dated through July 2026, placing the audit after that change. Reddit CEO Steve Huffman told investors in February that the platform is strongest on questions answered through many people's perspectives, the type of query the researchers examined.

If the pattern holds, Reddit's AI answers are more likely to use comments that already ranked highly and use a formal register. The system also removes much of the first-person language when rewriting those comments. The authors caution that the findings may not apply to other communities or AI search systems. The preprint therefore identifies an association that still requires independent verification, not a general rule for AI search.

Related articles

WhatsAppReddit AI Search Picks Already-Popular Comments, Audit Finds – Geolix.ai