Weird Search Engine Crawlers Visiting?

[Omega]

On Qday.forum regularly and means watching some strange visitors in the whosonline. The obvious ones everyone knows about are Google Googlebot, Bing Bingbot, Yandex, and SEO tools like Ahrefs and Semrush. But there are probably dozens of others that appear in website logs and most people have no idea what they are or who runs them.

Some are legitimate research crawlers from universities or institutions. Some are AI training data crawlers collecting forum content. Some are security scanners checking for vulnerabilities. Some are complete mysteries. A few are genuinely concerning if you look up who operates them.

We've been seeing some unusual ones here recently and curious what you've spotted. Have the admins noticed anything bizarre in your access logs? Do you know what some of these mystery crawlers actually are and who operates them?

Here are some questions to get things started. What crawlers have you seen that aren't Google Bing or Yandex? Have you spotted any AI training data crawlers specifically? Has anything tried to access admin paths or login pages? Any crawlers hitting the forum at unusual hours at high frequency? Are there any crawler names you've spotted that you actually looked up and found interesting or concerning?

[codeberg]

What an interesting topic.

In SMF Admin panel we detect quite a few search engines.
which i'm happy to copy and paste (sorry QD/QK if you know any reason these shouldn't be posted please remove)

Googlebot
Googlebot-Image
Googlebot-Video
Googlebot-News
GoogleAds
Bing
Applebot
DuckDuckGo
Yahoo
Baidu
Yandex
Naver
Mojeek
Exabot
PetalBot
Pinterest
OpenAI
Anthropic
Perplexity
MistralAI
Amazon
You.com
xAI
Bytespider
Meta
LinkedIn
Twitterbot
Discordbot
Slack
Internet Archive
Common Crawl
Ahrefs
Semrush
Majestic
Moz
SE Ranking
Awario
Trendictionbot
Omgili

Some of them we may have never heard of . we will remove ones we haven't seen after 5 months.

Of the reasons to detect them is so we can ignore their pageviews for page counts.
And its interesting modification wrote by co-admin here to show the icons with the members.

Any others you think we are missing. Hopefully QD or QK will see this message and perhaps look in the logs for any others

[QuantumKnight]

Codeberg I'm happy the list is fine to share publicly for the moment as theres been a few topics about why alot of Yandex on the forum. QD has been improving the SEO of the forum. JSON and IndexNow.

Probably now with his new mods the most advanced SMF forum in the world already.
Custom mods

[Omega]

Wow thats a list. Mainly a few though?

[Kane]

A few new ones on tonight

[Anthony_51]

GPTBot from OpenAI and Google-Extended are the AI crawlers explicitly labelled as such. Can be blocked in robots.txt if you don't want AI training on your content

[Ellie22]

University research crawlers are genuinely common and mostly harmless. Academic institutions run web studies and forums are interesting data sources

[Harper48]

The ones with no user agent string or random user agent strings are the most concerning. Legitimate crawlers identify themselves

[Kevin71]

MJ12bot from Majestic SEO is another common one. SEO tool crawling for backlink analysis. Similar to Ahrefs

[Thomas_69]

What about Reddit bot ?

[FairDos72]

DotBot is from Moz the SEO company. Another one that hits sites constantly if you're not blocking it

[Harry]

Its happening again. Hundreds online!