Server performance issues

[admin_jsfisher]

I am sure many of you have noticed the forums have on many occasions become remarkably non-responsive. The problem has been getting increasingly worse for some time. The cause: AI crawlers.

These are the web spiders that scour sites such as this one for every bit of content they can find to then train AI engines. Some are well-behaved and easily identified and blocked if appropriate, but many, many are not. They scan a site as rapidly as possible then restart the probing anew almost immediately. The server spends almost all its time repeating itself to these crawlers with few CPU cycles left to respond to the forum members.

South-east Asian countries (especially, China, Singapore, and Hong Kong) are homes for most of the malevolent AI crawlers. And with AI being the technology all the cool kids want to implement, the number of these malevolent crawlers is growing at an alarming rate. I did have some mechanisms in place to deal with the crawlers, but my methods were not able to handle the every growing load the crawlers.

Yesterday, I took a different approach to blocking Internet addresses and ranges of addresses for known AI crawlers. It seems to be working...for now at least...so the forum should be much ore responsive...for now at least.

[whoiswei]

I think we might need Cloudflare.