If you are running your own #Mastodon server, perhaps as a trial, and you decide to kill it, can you do one quick tidy up command before you pull the plug?
On the command line: tootctl self-destruct
This will notify the #fediverse that your server is on the way out. Otherwise, servers may try for days to communicate with you.
I've updated my #mastoadmin SQL and cron scripts a bit today to better keep my server clean.
It's getting pretty good after roughly a year of iterating on them, today in particular was a massive refactor and improvement to both query performance and capabilities.
Cleaned +70 spam accounts I had missed, thanks to the new tweaks.
I wonder if some internet sleuths could find where the bot's source code is at; that way, the bot and it's owner can be reported to the company hosting the source code.
To other #mastoadmins and #fediadmins -> We seeing these sorts of spam accounts trying (unsuccessfully) to register at our server, and not even trying to answer our "why do you want to join question" well. So all nuked before they get to in.
But keep a lookout for accounts like these trying elsewhere.
Esp if you don't have screening questions, and accounts can be created instantly.
#MastoAdmins I'm shutting down https://handmade.social on September 15th in case your server is a good place for the member accounts to migrate. I just don't have the funds to keep it going beyond that.
Please also do not render unsolicited advice about cheaper DIY hosting. I'm the primary caregiver to a special needs child and do not have the tech chops or time. That is why masto.host is so awesome.
If you'd like to take over the instance, I'm open to transferring. #MastoAdmin
I think we might have zeroed in on an actual solution to our server problems. Yesterday I noticed that when the server was struggling it was because it was constantly retrying the same batch of jobs that kept failing because they were trying to deliver to instances that were offline. The system was overwhelmed but it was spending most of its time waiting for timeouts on jobs that had already previously failed instead of processing new jobs that could actually be completed successfully.
I tooted about this and @victor pointed out that he has to periodically check the failed jobs and stop delivery to instances that are offline. Mastodon does this automatically After 7 days of failed attempts, but that's 7 days of generating jobs that will ultimately fail and be retried several times, clogging up your Sidekiq queues for no reason. After going through and checking which instances had the most failing jobs and stopping delivery to the ones that were offline, I was able to reduce the retry count from over 8,000 to about 200. More importantly, after I stopped delivery to offline instances, my failed job count fell to nearly zero. Without all these failed jobs clogging up the works all of the sidekiq queues have remained mostly clear, and items that get added to the queue get processed within a matter of seconds.
I don't think I'm ready to say we're completely out of the woods yet, but the results have been really encouraging over the last 24 hours. For any new #mastoadmins out there, this is something I'd recommend getting in the habit of checking on from the very beginning. I haven't seen this mentioned in any guides on how to scale Mastodon, but it will definitely become an issue as the number of Sidekiq jobs processed by your instance increases. If your retry counts get up into the thousands, you might want to check which jobs are failing because of connection errors and stop delivery to those instances that are down. If the instances come back online and attempts to make deliveries, the Mastodon application will automatically resume sending deliveries to those instances, so you don't have to worry about keeping track of sites that you have stopped deliveries to.
Thanks everyone for your patience while I try to work through this. And thanks to everyone who has offered support, ideas, and general good vibes. I truly appreciate it.
I've noticed that 8 out of the last 12 accounts that have signed up on our instance have done nothing since they signed up. No profile image, no banner, no bio, absolutely nothing other than sign up.
Do y'all experience the same types of numbers when dealing with new accounts? What do y'all do about it, if anything? Do you just let it happen and if they become active then so be it, and if they don't, so be it as well?
we must be due a Mastodon code update soon? It's been almost two months since the last one. It looks like there have been several good fixes and update to code, as yet unpublished.
Ok, question for #mastoadmins or anyone else out there with knowledge on the subject:
Adding pgbouncer on top of the postgres database seems to have at least reduced the problem with the sidekiq queues backing up. However, the push queue is still getting backed up to a few thousand jobs occasionally. This doesn't seem like it should be that big of a problem because I've added more sidekiq processes for that queue and they should be able to get through a few thousand jobs with minimal latency. The real problem, however, is that if the push queue starts showing as little as 1 minute of latency, all remote media previews break (they just show the fuzzy gradient with the "Not Available" caption), and all media uploads break. The only way to fix it, even after the latency drops back to zero is to do a hard reboot of the server.
It feels like I'm running up against some kind of timeout setting that's set too low. I could see a latency of 10 or even 5 minutes being a problem, but expecting that latency will never hit 1 minute seems unreasonable. Any ideas on what I might be doing wrong here? It seems to be tied to the database because if I reboot the web server without also rebooting the db server, the images stay broken.
@cosmality#MastoAdmins can mass search and block #thegx#spam accounts all at once to all of their server users easily by going to Preferences> Moderations> Accounts, and doing a search.
They pop up so often, I have the direct link to search bookmarked on my browser's favorite bar.
FediFetcher is a simple Python script that can help you pull missing responses from other #Fediverse instances into your own #Mastodon instance. It can also backfill profiles of new followers and followings.
It can be run as cron job, container, or even a GitHub action, meaning you don't need any infrastructure at all.