"That's likely a conservative estimate too, says Tollbit, as AI bots just keep getting better at seeming like humans when they navigate a website. "From the tests we ran ... many of these web scrapers are indistinguishable from human visitors on sites," Tollbit noted. "In light of this, the data below is conservative; it is likely worse than these numbers.""
"Training scrapes actually dropped by 15 percent between Q2 and Q4 of last year, Tollbit noted. Instead of scraping to develop models, bot traffic is being dominated by retrieval augmented generation (RAG) bots, which are what companies like OpenAI, Google, and others use to extract real-time information from the web to answer queries put to ChatGPT, Gemini, and the like. RAG bot traffic increased by 33 percent in the same period that training bot traffic declined."
AI bot traffic surged through 2025, reaching roughly one bot visit for every 31 human visits by Q4, up from one per 200 in Q1. AI bots are increasingly able to mimic human behavior, making detection and measurement conservative. Human visitors to websites declined five percent from Q3 to Q4 of 2025. Model-training scrapes fell 15 percent between Q2 and Q4, while retrieval-augmented generation (RAG) bot traffic rose 33 percent in the same period. AI search indexers that build RAG indexes increased traffic by 59 percent. RAG bots and AI tools are replacing traditional web search usage and dominating bot traffic patterns.
Read at Theregister
Unable to calculate read time
Collection
[
|
...
]