For the first time the #CoSocialCa Mastodon server has started to struggle just a little bit to keep up with the flow of the Fediverse.
We’ve usually been “push” heavy but we’ve started to see some spikes in “pull” queue latency. The worst of these spikes was today, where we fell behind by at least a couple minutes for most of the afternoon.
Eventually we’ll grow so big that we’ll need oodles of sidekiq queues and we’ll want to be able to customize how many types of them we want, and will run them as jobs across multiple servers and so-on.
But for now I’m just going to make the number of threads slightly bigger and see what happens.
@mick just note that article is completely out of date when it comes to the streaming server; I did changes & rewrote the scaling docs for that component.
That article from DigitalOcean suggests that 10-15 threads = 1 GB of RAM.
We also need to give each thread its own DB connection.
In staging the DB is local, so we don’t need to worry too much about a few extra connections.
In production, we’re connected to a DB pool that will funnel the extra connections into a smaller number of connections to the DB. Our Database server still has oodles of capacity to keep up with all of this.
Staging server only has 2 GB of RAM but it also has virtually no queue activity so let’s give it a shot.
Having confirmed that we have sufficient resources to accommodate the increase and then picked a number out of hat, I’m going to increase the number of threads to 40.
This looks better! Pull queue never got more than 41 seconds behind and that was only briefly.
I still am not clear on what has contributed to these spikes, so there’s no way of knowing for sure that the changes made yesterday are sufficient to keep our queues clear and up-to-date, but this looks promising.
We fell behind by less than a minute for most of the day yesterday, with some brief periods where we were slower still.
The droplet is showing no signs of stress with the increased Sidekiq threads, so I can toss a bit more hardware at the problem and see if we can reach equilibrium again.
Better would be to get a clearer picture of what’s going on here.
We have the capacity to run 40 workers (following the change I made last week, documented earlier in this thread.)
We have fairly huge backlog of pull queue jobs.
Why aren’t we running every available worker to clear this backlog? 🤔
It might be necessary to designate some threads specifically for the pull queue in order to keep up with whatever is going on here, but I am open to suggestions.
@mick I haven’t read the entirety of the thread, so forgive me if that’s already been covered, but have you tried defining your workers with different sequences of queues.
That wat you would have 10 workers prioritising the pull queue, but picking up other queues when capacity is available. And another 10 workers prioritising the default queue, but picking up other queues (including pull) when capacity is available.
You could permutate this for some different combination of queue priorities.
I’d hoped that just increasing the number of threads for the single service would be enough, but it seems like the default queue prioritization results in a backlog and idle workers.
So dedicating a number of threads per queue seems like the next sensible step.
Then that first command will process the push queue, after the pull queue has been emptied. And the second one will process the pull queue after the push queue has been emptied. Thus potentially wasting fewer resources.
After I was told to do this, it worked like a charm and my queues haven't been backed up at all. Before, I had gotten in the rut where several thousands were backed up in queue during certain busy hours of the federation day.
@mick i wonder if it'd make sense to produce log messages for each instance, actor, activity type, directionality and other properties over time & make that graphable?
What percentage of activities processed or sent in the last day were Delete's, what type of activities does Server Y send me?
(Maybe this also ties into @polotek's question earlier today)
@mick@thisismissem so the thing that has me thinking about this is I was using activitypub.academy to view some logs. I did a follow to my server and it showed that my server continually sent duplicate "Accept" messages back. I can't tell if that's an issue with my server or with the academy. Because I can't see my logs.
@thisismissem@mick one thing I know is a problem is retries that build up in sidekiq. Sidekiq will retry jobs basically forever. And when server disappear, their jobs sit in the retry queue failing indefinitely. I'm sure larger instances with infra teams do some cleanup hear. But how are smaller instances supposed to learn about this?
@thisismissem@mick where? Are they configurable? And again, how would I know? Is the recommended support channel complaining in mastodon until somebody tells you something that you can’t even verify?
@mick@thisismissem people told me that activity pub was very "chatty". I understand a lot better why that is now. But I now suspect that there's also a ton of inefficiency there. Because few people are looking at the actual production behavior.
@polotek@mick at Hachyderm we do do a lot of monitoring & alerting (helps when the infrastructure team is all really experienced), but there could certainly be more logs & data available
@polotek@mick@thisismissem what happened to the attempt to wire Mastodon up with OpenTelemetry? This kind of thing is what honeycomb.io is really good at exploring
Add comment