This is an interesting question. And I want to give a thoughtful answer. The reality is that the backend has all of these same problems. They have also experienced an explosion in complexity. I think the outcomes are different mostly because they have more support. The reality is that cloud vendors have assumed a ton of the complexity and the risk on the backend. So engineers aren't drowning to the same extent. https://mastodon.social/@floby/112107065645789780
The problem is still there though. For example, scaling a database to handle terabytes of data and millions of users is still really hard. Most companies have a team that can't actually do that. But they don't have to. They can use RDS or Aurora. And some really experienced engineers at Amazon will do it for them.
But these companies still experience the same grinding to a halt when they find the edges of what comes out of the box with AWS. Maybe it's just coming much later than with frontend.
I've already done that and have not worked at Amazon nor used RDS nor Aurora.
"Maximum Database Size. SQLite can have a maximum database size of 140 terabytes (TB)."
I mean sure, if you're using Micro$oft Excel it is constrained to 2GB (it used to be much smaller). But, people need to stop pretending that terrible spreadsheet is a database and treat it accordingly.
There are definitely viable alternatives even outside of SQL implementation realms.
If I had to guess, @hyc's LMDB also does quite a bit better.
(From the Wikipedia page on LMDB: "Most former modern computing architectures had a 32-bit memory address space, imposing a hard limit of 4 GB on the size of any database that directly mapped into a single-level store. However, today's 64-bit processors now mostly implement 48-bit address spaces, giving access to 47-bit addresses or 128 TB of database size,[6] making databases using shared memory useful once again in real-world applications.")
I can't think of having hit a 4 GB limitation in decades. Even Micro$oft Exchange, which had the absolutely atrociously bad JET storage engine (same as M$ Access, avoid these things. You have been warned) had a 16GB limit for normal (later raised to 50GB) and 16TB for "Enterprise" licensing (though IIRC, that may have required PAE [Page Address Extension] capable CPUs before AMD64/x86-64 became more widespread).
Personally, I wouldn't trust anything to Exchange, or any Micro$oft product, but I have been paid to administer such crap (and lost so much sleep as a result) and am guessing others have too. The idea of scaling something such as JET to TB and millions of users is horrifying.
IIRC when @bifrosty2k was grandfathered into working at Micro$oft as a UNIX Administrator due to having previously been part of Hotmail (which Micro$oft purchased) Micro$oft management had a hair brained idea to migrate from FreeBSD+whatever other stuff Hotmail was running to Exchange & it was not smooth.
@teajaygrey@polotek I don't think "32-bit address space" is the constraint people think of when productionising a DB for many TB for millions of users.
It's more stuff like managing DR, coping with user load exceeding what a single server will cope with, devs breaking stuff with an inefficiency query, stumbling over an obscure bug in the DB software etc.
All of which aren't insurmountable, but using a cloud solution means you can care a lot less about DB details, and more about the product.
Add comment