In an age of LLMs, is it time to reconsider human-edited web directories?
Back in the early-to-mid '90s, one of the main ways of finding anything on the web was to browse through a web directory.
These directories generally had a list of categories on their front page. News/Sport/Entertainment/Arts/Technology/Fashion/etc.
Each of those categories had subcategories, and sub-subcategories that you clicked through until you got to a list of websites. These lists were maintained by actual humans.
Typically, these directories also had a limited web search that would crawl through the pages of websites listed in the directory.
Lycos, Excite, and of course Yahoo all offered web directories of this sort.
(EDIT: I initially also mentioned AltaVista. It did offer a web directory by the late '90s, but this was something it tacked on much later.)
By the late '90s, the standard narrative goes, the web got too big to index websites manually.
Google promised the world its algorithms would weed out the spam automatically.
And for a time, it worked.
But then SEO and SEM became a multi-billion-dollar industry. The spambots proliferated. Google itself began promoting its own content and advertisers above search results.
And now with LLMs, the industrial-scale spamming of the web is likely to grow exponentially.
My question is, if a lot of the web is turning to crap, do we even want to search the entire web anymore?
Do we really want to search every single website on the web?
Or just those that aren't filled with LLM-generated SEO spam?
Or just those that don't feature 200 tracking scripts, and passive-aggressive privacy warnings, and paywalls, and popovers, and newsletters, and increasingly obnoxious banner ads, and dark patterns to prevent you cancelling your "free trial" subscription?
At some point, does it become more desirable to go back to search engines that only crawl pages on human-curated lists of trustworthy, quality websites?
And is it time to begin considering what a modern version of those early web directories might look like?
The piece I’m working on right now has taken me 60+ hours and counting.
And the hardest part, by far, has nothing to do with finding the time or the energy to do this - it is the immense sense of responsibility that I feel to get this RIGHT. Because most for the things I write under my name are bigger than me.
I’m writing to fight for what I believe in. For the internet. For my industry. For good ideas. For good people who want to do good in the world.
I want to tell you a story about how I, a marketer with a small social media following, accidentally blew up Mastodon with some war reporting.
And then I will show you what that experience, together with some recent developments on other social networks, can tell us about this weird moment we’re living through and what the internet might look like on the other side of it.
When I searched on: cell phone follow property lines
I got 4.7+ billion hits
The first hit is a PDF for my consulting service.
New personal record for number of hits where one of my things is at the top. #WebDev#SEO
The Verge asked “who ruined the internet”, and pinned the blame on SEOs.
I don't blame the Verge for falling into this trap. When they see us (SEOs), they see the people who work in explicitly helping businesses make money online. And they see our role in commercializing the web all too clearly, while missing the other parties who may have also played a part.
Mal ein "Ausrutscher" in einem anderen SEO-Projekt - in die andere Richtung. Im letzten Post hatten wir den "üblichen" Fehler bei WordPress und die verheerenden Folgen gesehen. 📉
Hier hatten wir Inhalte vorbereitet in Erwartung einer grossen Veröffentlichung. "Still und heimlich" Inhalte bereitgestellt, um etwas mitzunehmen vom Event.
Okay, Google just handed me the most perfect example for showing what I mean when I talk about cookie-cutter, bland, forgettable #content
A new blog post showed up on the search results page next to one of my own, both reviewing the exact same software. So... let's see how different it is to mine?
(Removing their website name because hating on them in particular isn't the point)
Every good #SEO gets up in the morning and checks GSC (and Bing WMT) to see if anything has changed overnight. But that can take an age if you have a few sites to check.
Here is my Looker Studio dashboard that i can glance at - three pages, this one is my side projects including my own site which is having a hard time with clicks and yet impressions strengthen. All fur coat and no knickers as my gran would say.
How does Google interpret JS redirects?
👉🏻 It indexes the start page
👉🏻 It doesn't index the destination page
👉🏻 It sees redirect as a DOM update and uses the content of the destination page
My thoughts about The Verge's article, The people who ruined the Internet and Danny Sullivan's response, and the conversation swirling around #SEO and the suckiness of the #Internet.
Another SERP thing for #seo folks to watch out for/ponder on.
When a SERP shows a lot of dates in results, are the ones ranking high missing dates? If so, are they:
🆕 pretty new (check source code even if not visible to users and within a month)
❌ doesn't have a date at all (usual "date" suspects in the source code not returning anything)
For keywords that I'd expect an article page, I'm seeing a lot of the newest ones rank well but the result didn't pull/show the date.
"Any user interaction data from a system this broken will become increasingly unreliable. So it's no surprise we're seeing a simulacrum of content, a landscape full of mediocre content that might seem tasty but isn't nutritional."
Wow. May have found a really obscure technical issue in my site that might have a profound effect on its #seo. Have just fixed it and am going to test over the next few weeks to see if there is any difference. If there is then I have yet another audit step in my arsenal and will write it up.
What Search Engine or terms do you use?
What search engine do you use? What terms might you use to filter out junk?...