My anime forum archive site broke after a bot crawled it 500 times in one day

I run a little site that archives old forum threads about anime from like 2005. Last Wednesday I checked my stats and saw this one bot had hit it 500 times in a single day. It overloaded my cheap shared hosting plan and the whole site went down for 8 hours. I spent the whole afternoon blocking user agents and adding a rate limit rule to my .htaccess file. Then I realized the bot was probably just some person's scraper trying to pull data. I still don't know why they wanted all those threads about Naruto filler episodes so badly. Has anyone else had a random bot destroy their little project site like that?

2 comments

2 Comments

james_ramirez1mo ago

Oh man, that "bot hit it 500 times in a single day" part gave me flashbacks. I had a small forum archive too and some random scraper absolutely wrecked it trying to grab every single post about old school Dragon Ball Z.

piperr581mo ago

300 requests from one IP in an hour crashed my old phpBB board too, it was a nightmare.