Large part of my work is in the infrastructure security sector and I think I can help at least with some of these challenges you described:
there are databases of IP addresses and subnets that are known to run dumb, persistent scanners, bruteforcers etc - these should be blocked right away at the firewall level and that’s the first line of defense; the lists are usually updated every hour or daily
more sophisticated spam/hacking teams cycle their IP addresses, use Tor or set up dedicated infrastructure for your campaign only, but then so are the intrusion detection tools - #Wazuh and #Crowdsec are two solutions I have been using a lot that will allow you to block an IP address instantly when a suspicious pattern is detected in your logs, which basically allows you to block them on the spot
These tools deal with HTTP server logs or application logs, so you can usually do whatever kind of matching you can come up with and write custom signatures such as “a 10 characters long alphanumeric usernames created from the same IP over 15 minutes”. They are not silver bullets as any such tool can be bypassed by a sufficiently resourced and sophisticated team, but they significantly increase the cost of the campaign for the attacker.
I don’t have any Mastodon instances but have implemented them for Pleroma, NextCloud and many other solutions, so happy to help with deployment for your Mastodon instance if interested.