I need to broaden my #Matrix discussions.
Here’s a 2.5 yo post about an 800 lb gorilla in the Matrix idea. I’ve heard people brush it off, but their needs were often niche, fanciful, or delusional. I need some engineers to help me separate the facts, legends, myths, and bullshit of a large scale federated matrix chat network. In our experience, just enabling presence can bring a Synapse server to its knees, if even if just 1 user follows a BIG room. What if many folks follow MANY big … 1/2
@shoq I have presence disabled on my synapse server, probably the largest room I’ve joined is the firefish support room. Performance hasn’t been too bad, storage hasn’t been an issue either, though I primarily use it for bridging and secure conversation. Considering the heavy constraints I have on storage for synapses, I would expect joining a large room to kill it
@insert it’s absolutely wonderful when audiences are small and spread out. But even if you host big rooms on big servers, the weak link is still the home instance if even a single user chooses to follow that big room. That instance is brought to its knees. This Way be Madness. Very expensive Madness
@shoq@insert
In my experiences using community-hosted servers, large rooms put a big load on the homeserver during a join, but once it's synced up with the room information (members list etc) and a small amount of room history, it's fine.
A large, encrypted room would take longer, what with exchanging keys and decrypting history (if enabled). But it's unusual for rooms to be both large and encrypted, and even that would be a temporary problem.
@shoq
If you're worried about storage from high traffic rooms, there are ways of automatically deleting surplus stuff, like old media, that keep it in check pretty well. ASCII text alone takes up a tiny amount of storage. You could be on your way to a Wikipedia or so worth of chat history before you start worrying about storing that.
@lightweight admins the Synapse instance I use, he might have thoughts on this.
@strypey@shoq referring more to high traffic considering the amount of data that would need to be added to the database, now that I fixed that though it should be fine if I join a large room.
@insert Even without any message traffic, you cannot have presence enabled for large rooms such as MatrixHQ (47000) without bringing even a bigger than average hardware to its knees. Imagine trying to follow Obama or AOC (or the entire House)? Even with presence off (it doesn’t exist on most the centralized apps), your server is still at the mercy of what any one of your users subscribes to. As one goes, so go we all. It’s just not very efficient.
@strypey Yes, but not sure it changes the core calculus. Someone has to pay for servers that stay vulnerable to the usage habits and whims of individual users choosing to join any given room, many of them potentially enormous. Challenging to fund, whenever I try to figure out how to scale it up. And the data redundancy and media handling are the same issues activityPub faces. I’ve never been comfortable squaring all that in my head, but I’m not an engineer. Still trying tho.
@shoq
> servers that stay vulnerable to the usage habits and whims of individual users choosing to join any given room, many of them potentially enormous
I'm not a server admin but this seems like a totally tractable problem. Isn't there are a way of configuring a matrix homeserver to only keep chat history for a certain numbers of months/ weeks/ days? Or to give each room a maximum storage space, so it has to delete history once it reaches that limit, starting with the oldest items?
@strypey The archives are a manageable problem. it’s the total number of subscribers in a room that creates the notifies overhead. If Presence is enabled, even a modest room can bring the server to its knees. Now one might argue that no big centralized chat servers have presence, and that’s true, but it’s always been a selling point of Matrix, but it’s expensive. As it is, I subscribe to 40+ rooms on my own (big) server and loading can take 60+ secs or more.
@shoq
> If Presence is enabled, even a modest room can bring the server to its knees
I presume you're talking about a Synapse server running matrix 1.8? So you're describing the performance limits of that software, in its current state, running a version of the protocol that will be out-of-date within about a year.
> I subscribe to 40+ rooms on my own (big) server and loading can take 60+ secs or more
This is why I drew your attention to matrix 2.0. Sliding Sync takes care of this.
@strypey Good to know, if it works and is actually rolled-out for real in less than a few years. They move glacially. And this doesn’t make their byzantine session verifications any easier, where the messages and tips are written & for engineers, but users are totally lost..
@shoq
> if it works and is actually rolled-out for real in less than a few years
Please read the linked blog post. It anticipates and addresses almost every criticism you've made in this thread, including this one. The protocols changes are already available for testing, using an add-on to Synapse and the Element X beta.
@shoq
> byzantine session verifications any easier, where the messages and tips are written & for engineers, but users are totally lost
You mean the one where if you open a verified session on a device with a camera, you can just scan a QR code on a new session to verify?
I'm finding it really hard to believe that you have used Element recently. Because most of your criticisms would have been fair a few years ago, but they don't describe the experience I have using it today.
@strypey Have you tried to explain the verification process, the keys, and the rest of the Element infrastructure to ordinary users? I have. Most of them want to hit me. Of course it’s better than it was. But it still has a long way to go. The error messages themselves could be a crowd project, but it’s unlikely to self-organize. The Element folks could craft projects like that easily. But they won’t.
@shoq
> Have you tried to explain the verification process, the keys, and the rest of the Element infrastructure to ordinary users?
I got one of my friend to use it to keep in touch with me while he was overseas. It wasn't hard to explain the bits he needed to know to use it. For the same reason, He got his brother onto it. A guy who isn't with all due respect, the sharpest tool in the shed. My friend was able to explain it to his brother, having only just learned it himself.
@shoq
But if you think matrix is too complicated, you could always try XMPP instead. The @snikket_im server and apps might be the best place to start. Although eJabberD now supports both protocols:
@strypey You convinced me that Matrix 2 has enough long term promise that I would try once again to fix our invite bug. I don’t want to sell it short. The Sydent ID people feel it’s a template config error. Waiting for admins to try once again, but overall, this kind of problem illustrates what a maze of issues Matrix can be. Have you configured Sydent properly yourself?
@shoq Without looking @ code but familiar with DO, that setup would be rough. Stuff not your users' is just cached, but cache needs RAM. At some point, cache is full & it has to toss the oldest item as more items are coming in. At the same time, your users are hitting the DB which needs to be on a separate instance on a private IF, which they sell. I suspect your VM was starved for RAM & beating itself up to store more stuff. Then other functions back up & the load rises. CPU thrashes. Not good.
@shoq Admins need to calculate how much RAM to use. With this stuff, the more the merrier. If the IF was actually 100Mb, and big instances were being loaded in a lot, I'd want to open my wallet for RAM and CPU. Also get the database server off of the instance. Your users, when they store stuff, can really clog the pipeline.
@stevesplace these needs go well beyond what a small instance operator would even think about paying for. I am looking for a viable concept for a large pubic network with thousands or millions of accounts with huge followings. It will take a Signal like network, dedicated to that need. I’d be ok with signal if they supported spaces. But they don’t. Could they? Can’t say.
@shoq Ah. I'd colo at someplace likw Hurricane Electric first after calculating how many servers I'd need. They mirror around the world. I'd use hardware, not VMs. Again, it's RAM & CPU-intensive. You need back-end DB servers for your instances, which I mentioned before. And you'd want to mirror in-house, too, in case of crashes, like on OSCAR config or something akin. If you do your own mail, that must be separate, too. Just my inclination. YMMV.
@shoq I could sing the praises of colo over VM here. t's easy to underestimate what you need in a VM and to overestimate its performance under load. The biggest downside tp a good colo is having to go there when you must. Hurricane Electric in Fremont, CA was good about handling reboots for me. I was on my own at 1 Wilshire in L.A..
Or you could trick out your place & not pay a colo (but not get mirroring, should ones near you offer it).
I'd skip tiny VMs. Go big or go home going forward.
Add comment