More Spiders Please

[Quanta]

Indexwww.simplemachines.org

Seems like they have identified way more than we have

[Sorted Echo]

I haven't seen Grok on here yet. Is he banned?

[QuantumDay]

Grok (operated by xAI) often crawls websites or fetches real-time links without cleanly or consistently identifying itself using a clear "Grok" or "xAI" user-agent string. Site owners frequently notice anonymous requests, generic user-agents, or traffic routed through residential and cloud VPN networks rather than a transparent bot signature.

TLDR: It pretends

[Poppy51]

I've never ever heard of some of them.

[WaveFunction34]

Theres 5000 online now. I dont know who they are?

[QuantumDay]

Great stuff. Its happening whilst I'm online tonight.

Not from a known spider. useragent may be pretending. could be any llm or just a scraper

[codeberg]

You will temporarily notice I've wiped the online to track the temp blocks we have put in place.

[DrewMcIntyre08]

Well done it appears to have stopped

[QuantumDay]

We put some blocks in place to stop them

[ParallelSelf99]

Seems to have calmed down

[Henry10]

Definitely was a busy evening for you QD. Whats the damage if any?

Talking of spiders. I have never seen Yahoo! ?

[QuantumDay]

Slurp hasn't graced us with their presence yet in the seven months we have been online. Its rare as hens teeth as their search is mostly bing results

[Quarry]

The useragent spider angle is a big part of this, too. A lot of crawlers identify themselves very clearly, while others are either obscure or tucked away behind generic-looking strings. It would be neat to see a proper list of the useragent spiders the site has actually spotted, even if some of them are only occasional visitors. That could turn the whole thing into a little census.

[Leopard10]

Agreed on wanting more spiders, although I suspect the numbers are being skewed by what is actually willing to crawl us. Some useragent spiders are incredibly persistent and show up constantly, while others seem to visit once, take a look around, and vanish into the mist. The rare ones are probably more interesting than the big regulars anyway.

[BlueFalcon]

Could part of the problem be that we are counting names rather than behavior? A useragent spider might have several variants, or change its useragent string over time, so what looks like one crawler could actually be multiple versions. Tracking the useragent alongside the crawl frequency would make the list much more revealing.

[Stu87]

Seven months and still waiting for Slurp is brutal. At this point it deserves a little ceremonial guestbook entry whenever it finally turns up. The funny thing is that the useragent spiders nobody expects seem to be the ones that eventually wander in, while the famous names act like they have better things to do.

[Phoebe_37]

I'd love to see some distinction between genuinely different useragent spiders and the endless variations of the same crawler. Otherwise the list can get inflated pretty quickly. Even a simple note like "same spider, different useragent" would help make the collection easier to understand.

[Lantern]

One thing worth checking is whether robots.txt or other crawl controls are quietly discouraging some of these useragent spiders. A crawler can be out there and still never make it into the stats if it decides the site is off-limits. That might explain at least some of the gaps without needing the spiders themselves to be especially rare.

[Merchant]

Not sure I agree that every missing spider is necessarily a problem. Some of the less common useragent spiders may simply have little reason to visit this particular site. I'd rather have a smaller list of confirmed, interesting crawlers than pad the numbers with questionable bots that may only be pretending to be something else.

[Dave_52]

The comparison with other search engines is interesting. Different useragent spiders can have wildly different crawl habits, so a site getting plenty of traffic from one search ecosystem does not necessarily mean all the corresponding crawlers will show up. It would be fun to compare the useragent list against actual referrers and see which spiders are missing from each side.

[QuantumToken65]

At this point I'm rooting for the underdogs. Forget adding another dozen copies of the usual useragent spiders; give me the obscure crawler with three visits, two pages requested, and a completely mysterious useragent string. Those are the ones that make a spider list worth checking back on.

[Phoebe_37]

There is something strangely satisfying about waiting for a rare useragent spider to appear. It turns crawling into wildlife watching: regular visitors are the pigeons, aggressive bots are the seagulls, and Slurp is apparently some mythical woodland creature nobody has actually photographed yet.

[Coder22]

Maybe the answer is not just more spiders, but better spider visibility. If the logs expose the useragent, response time, pages requested, and first/last seen dates, you could spot some really odd patterns. One crawler might politely request a handful of pages, another might hammer the same endpoint all afternoon, and a third might appear once every three months like clockwork.