In an age of LLMs, is it time to reconsider human-edited web directories?
Back in the early-to-mid '90s, one of the main ways of finding anything on the web was to browse through a web directory.
These directories generally had a list of categories on their front page. News/Sport/Entertainment/Arts/Technology/Fashion/etc.
Each of those categories had subcategories, and sub-subcategories that you clicked through until you got to a list of websites. These lists were maintained by actual humans.
Typically, these directories also had a limited web search that would crawl through the pages of websites listed in the directory.
Lycos, Excite, and of course Yahoo all offered web directories of this sort.
(EDIT: I initially also mentioned AltaVista. It did offer a web directory by the late '90s, but this was something it tacked on much later.)
By the late '90s, the standard narrative goes, the web got too big to index websites manually.
Google promised the world its algorithms would weed out the spam automatically.
And for a time, it worked.
But then SEO and SEM became a multi-billion-dollar industry. The spambots proliferated. Google itself began promoting its own content and advertisers above search results.
And now with LLMs, the industrial-scale spamming of the web is likely to grow exponentially.
My question is, if a lot of the web is turning to crap, do we even want to search the entire web anymore?
Do we really want to search every single website on the web?
Or just those that aren't filled with LLM-generated SEO spam?
Or just those that don't feature 200 tracking scripts, and passive-aggressive privacy warnings, and paywalls, and popovers, and newsletters, and increasingly obnoxious banner ads, and dark patterns to prevent you cancelling your "free trial" subscription?
At some point, does it become more desirable to go back to search engines that only crawl pages on human-curated lists of trustworthy, quality websites?
And is it time to begin considering what a modern version of those early web directories might look like?
I want to tell you a story about how I, a marketer with a small social media following, accidentally blew up Mastodon with some war reporting.
And then I will show you what that experience, together with some recent developments on other social networks, can tell us about this weird moment we’re living through and what the internet might look like on the other side of it.
The Verge asked “who ruined the internet”, and pinned the blame on SEOs.
I don't blame the Verge for falling into this trap. When they see us (SEOs), they see the people who work in explicitly helping businesses make money online. And they see our role in commercializing the web all too clearly, while missing the other parties who may have also played a part.
#introduction post! New nerd on the block, #nonbinary, they/them. Likes #opensource stuff, believes in using #SEO for evil, and mostly just wants to read good #scifi all the time... Just finished re-reading #Watchmen.
Will post re: books, tech, political rants, and whatever else is on my head... then might get overwhelmed and disappear for a while, lurking in the background for a while before coming back with a vendetta against myself.
We recently noticed a fair bit of traffic on www.bbc.co.uk & www.bbc.com from a User Agent which identifies itself as "ByteSpider" (& has a @bytedance.com email address).
Lots of docs on the web state it doesn't obey robots.txt but ByteDance have told us it does:
> ...in the robots.txt files > user-agent:Bytespider > Disallow:/
Thought that might be worth documenting as it might be a recent change & several of us searched but found zero docs from ByteDance
@deep: In this article, I address the topic of content generation by artificial intelligence and its impact on the publishing market and, more broadly, on the quality of information available to readers. I present concerns about the quality and authenticity of AI-generated content in the context of disinformation and potential...
Hi! Are you a freelance SEO expert?
I'm seeking someone to audit and suggest improvements for my business website (WordPress).
If you have the skills and experience, please contact me.
I'm in UK but location not important.
Don't have a huge budget but there is one!
Let's discuss further.
"Any user interaction data from a system this broken will become increasingly unreliable. So it's no surprise we're seeing a simulacrum of content, a landscape full of mediocre content that might seem tasty but isn't nutritional."
Erste Schritte - Quick Wins für mehr Sichtbarkeit in Google. https://www.youtube.com/watch?v=b2z-2IQoQq8 24min Basis-Schulung zu Search Console, das oft innert Tagen eine Steigerung bringt. #SEO
Exactly the same content, posted on the same time, to similar crowd, gets 10x more engagement here than on Twitter, even tho I have 10x less followers here.
Why are people still using Twitter for promotion? And some are paying money thinking they're getting a good deal.
Search engines are morphing into generative AI chatbots, and websites are facing a referral traffic drought. I spoke with Jim Yu from SEO company BrightEdge about what website operators and publishers can do as AI-based search products like Perplexity and Google SGE begin to take over the web. https://thenewstack.io/as-search-engines-become-ai-chatbots-what-can-publishers-do/#AI#search#SEO
Patient zero: if I search the web for a phone number, what I want is to know whose phone number it is.
The top results should be pages that contain that phone number and information about what person or business it belongs to (including, if it's a business, their own web site).
What I don't want—what NOBODY wants—are spammy, scammy, for-pay reverse phone lookup companies.
Afraid of content jobs being taken over by AI? Then write for humans.
Stop padding, stop vamping, stop over-explaining the background, stop putting the answer to the question below the fold, stop click-farming, stop writing for machine indexing, and stop putting a single word anywhere in the piece for any reason other than to make it a better piece of WRITING for HUMANS to read.
If you write for machines, you deserve to be replaced by an #LLM.
Between Innovation and Disinformation - An Analysis of the Market for Books Generated by Artificial Intelligence (dadalo.pl) Polish
@deep: In this article, I address the topic of content generation by artificial intelligence and its impact on the publishing market and, more broadly, on the quality of information available to readers. I present concerns about the quality and authenticity of AI-generated content in the context of disinformation and potential...