support,

We've previously made a strategic shift from Cloudflare to Fastly to tackle scalability issues at #tootcommunity.

This important move is designed to boost server stability and enhance overall performance, benefiting our users.

Discover the details of our DevOps strategies and their influence on the fediverse through this insightful post by our DevOps engineer:

https://jorijn.com/tackling-scalability-challenges-at-toot-community-from-cloudflare-to-fastly/

#Mastodon #DevOps #ServerScalability #Fastly #MastoAdmin

kevm,

@support @jorijn very cool stuff. Thank you for the blog and and keeping your infrastructure open. I may find myself using it to learn more about deployment practices.

jorijn,
@jorijn@toot.community avatar

@kevm thanks, Kevin :)

leoncowle,
@leoncowle@hachyderm.io avatar

@support @jorijn Nice writeup!!

I'm very (erm ... vary LOL) interested in the inclusion of “Cookie" in your Vary header:
vary: Accept, Accept-Language, Cookie

This means each cached object at is going to be indexed on path+hostHeader+AcceptHeader+AcceptLanguageHeader+CookieHeader. Doesn't that massively reduce your cache hit ratio (and increase your cached objects), because the Cookie header can possibly have so many variations inside it?

What we do instead (non-Mastodon) is...

1/2

jorijn, (edited )
@jorijn@toot.community avatar

@leoncowle @support

Hey Leon. You're right, this is a bit strange. I took the headers from a regular Chrome request, and that would explain the inclusion of the Cookie header.

(Edit: I explain this in the next post)

Usually, unauthenticated API traffic wouldn't include this header. I remember faintly about some custom VCL I deployed back then, but it mostly originated from here: https://github.com/mastodon/terraform-fastly-service

jorijn,
@jorijn@toot.community avatar

@leoncowle @support

Also, some paths, like /api/v1/timelines/public will return content (and headers!) depending on how you authenticate.

If requested with a valid session (in a cookie), it'll return Cache-Control: private, no-store and Vary on Cookie.

When requested without a cookie, it'll Cache-Control: max-age=15, and only Vary on Authorization, Origin.

We're leaving it in the capable hands of the Mastodon software, as it's opinionated about which paths are cacheable.

lifeofguenter,
@lifeofguenter@convo.casa avatar

@jorijn @leoncowle @support

I had created some tickets back then due to this. Most do not accept vary headers because caching keys become very arbitrary while most CDN have a "single pricing" for storage.

Ended up just nginx (which is also the official way) but I think it would be better to have distinct urls per content-type and to generally make it possible to just cache by url as a cache-key..

Glad its working out with Fastly, unfortunately most smaller instances won't be able to afford :P

haubles,
@haubles@fosstodon.org avatar

@lifeofguenter @jorijn @leoncowle @support hey Guenter - check out fastly.com/forward, we offer free or deeply discounted services to open source/ community projects

  • All
  • Subscribed
  • Moderated
  • Favorites
  • mastodon
  • DreamBathrooms
  • InstantRegret
  • ethstaker
  • magazineikmin
  • GTA5RPClips
  • rosin
  • modclub
  • Youngstown
  • ngwrru68w68
  • slotface
  • osvaldo12
  • kavyap
  • mdbf
  • thenastyranch
  • JUstTest
  • everett
  • cubers
  • cisconetworking
  • normalnudes
  • Durango
  • anitta
  • khanakhh
  • tacticalgear
  • tester
  • provamag3
  • megavids
  • Leos
  • lostlight
  • All magazines