simon, 2 months ago

I did get a new TIL out of it - I moved to simonwillison.org for 24 hours and now I'm using Cloudflare to redirect that back to simonwillison.net https://til.simonwillison.net/cloudflare/redirect-whole-domain

simon, 2 months ago

I wonder how much of an impact this had on the invisible Mastodon infrastructure...

Every time someone I follow posted in the past 24 hours their instance would have tried and failed to deliver to my instance - how hard do these things get retried?

Am I about to see a massive flood of incoming deliveries that finally work?

clifff, 2 months ago

@simon by default, sidekiq jobs that throw an error will be retried 25 times over the course of 20 days https://github.com/sidekiq/sidekiq/wiki/Error-Handling#automatic-job-retry

it's possible mastodon has customized this but I haven't specifically looked. would expect you'll get more traffic than usual but shouldn't be all at once

JsonCulverhouse, 2 months ago

@clifff @simon

Here is the retry algorithm, if you were down a day. They will trickle in vs a massive flood

(0..16).each do |count|
delay = (count4) + 15
jitter = rand(0.5 * (count4))
total_delay = delay + jitter
puts "Retry #{count}: #{total_delay} seconds"
end

Retry 0: 15.713002155688077 seconds
Retry 1: 16.48915648567112 seconds
Retry 2: 31 seconds
Retry 3: 124 seconds
Retry 4: 382 seconds
Retry 5: 938 seconds
Retry 6: 1319 seconds
Retry 7: 2617 seconds
Retry 8: 6006 seconds
Retry 9: 9468 seconds
Retry 10: 12612 seconds
Retry 11: 19804 seconds
Retry 12: 26167 seconds
Retry 13: 31795 seconds
Retry 14: 55286 seconds
Retry 15: 67782 seconds
Retry 16: 89823 seconds

https://github.com/mastodon/mastodon/blob/71e5f0f48c3bc95a894fa3ad2c5a34f05c584482/app/workers/activitypub/delivery_worker.rb/#L13-L21

demiurg, 2 months ago

@JsonCulverhouse @clifff @simon May I ask why you add a jitter?

simon, 2 months ago (edited 2 months ago)

@demiurg @JsonCulverhouse @clifff at a guess it's to avoid accidentally having a whole bunch of retries fire at the exact same time - better to spread them out with a little bit of jitter