A question about what states were most-frequently represented on the HN homepage had me do some quick querying via Hacker News's Algolia search ... which is NOT limited to the front page. Those results were ... surprising (Maine and Iowa outstrip the more probable results of California and, say, New York). Results are further confounded by other factors.
HN provides an interface to historical front-page stories (https://news.ycombinator.com/front), and that can be crawled by providing a list of corresponding date specifications, e.g.:
So I'm crawling that and compiling a local archive. Rate-limiting and other factors mean that's only about halfway complete, and a full pull will take another day or so.
But I'll be able to look at story titles, sites, submitters, time-based patterns (day of week, day of month, month of year, yearly variations), and other patterns. There's also looking at mean points and comments by various dimensions.
Among surprises are that as of January 2015, among the highest consistently-voted sites is The Guardian. I'd thought HN leaned consistently less liberal.
The full archive will probably be < 1 GB (raw HTML), currently 123 MB on disk.
Contents are the 30 top-voted stories for each day since 20 February 2007.
If anyone has suggestions for other questions to ask of this, fire away.
NY is highly overrepresented (NY Times, NY Post, NY City), likewise Washington (Post, Times, DC). Adding in "Silicon Valley" and a few other toponyms boosts California's score markedly. I've also got some city-based analytics.
I'm working on parsing. Playing with identifying countries most often mentioned in titles right now, on still-partial data (missing the past month or so's front pages).
Countries most likely to be confused with a major celebrity and/or IT/tech sector personality: Cuba & Jordan.
Country most likely to be confused with a device connection standard: US (USB).
Raw stats, top-20, THERE ARE ISSUES WITH THESE DATA:
How Much Colorado Love? Or a 16-year Hacker News Front Page analytics
I've pulled 5,939 front pages from Hacker News, dating from 20 February 2007 to 25 May 2023, initially to answer the question "how often is Colorado mentioned on the front page?" (38 times, 5th most frequent US state). This also affords the opportunity to ask and answer other questions.
I've confirmed that the story shortfall does represent actual HN experience. Several days with fewer-than-usual stories, one day of complete outage, mostly in the first year of operations:
I'm wanting to test some reporting / queries / logic based on a sampling of data.
Since my file-naming convention follows ISO-8601 (YYYY-MM-DD), I can just lexically sort those.
And to grab a random year's worth (365 days) of reports from across the set:
ls rendered-crawl/* | sort -R | head -365 | sort<br></br>
(I've rendered the pages, using w3m's -dump feature, to speed processing).
The full dataset is large enough and my awk code sloppy enough (several large sequential lists used in pattern-matching) that a full parse takes about 10 minutes, so the sampling shown here speeds development better than 10x while still providing representative data across time.
I'm still trying to sort out a way to search / determine "statistically interesting terms", that is words or phrases which are disproportionately represented in submission titles.
@loke In part, and I can look for some specific strings...
"Who's Hiring" is a fairly regular post, I find 25 instances, oh, "Who is hiring?" is the more uniform version, 171 instances. About 13% of the initial total.
Filtering against "who is hiring", "is hiring" shows 24 hits for various YC ventures.
And excluding "is hiring" entirely ... mostly seems to land discussions about hiring (or stories about personnel changes at companies) rather than solicitations. 213 hits.
"Interview" also captures other forms of dialogue, e.g., "Interview with Michael Wesch", "Interview: Jimmy Wales", but also "Y Combinator interview tips".
The more specific "interviewing" gives 22 results, of which .... 21 are jobs-related ("Interviewing my mother, a mainframe COBOL programmer" is the exception).
"recruiting" seems to be entirely discussion / stories. 35 hits.
Note that some idiosyncrasies affect this, e.g., "New York City" appears rarely, whilst "New York" may refer to the city, state, any of several newspapers, universities, etc. "New York" appears 315 times in titles (mostly as "New York Times").
I've independently verified that, for example, "Ho Chi Minh City" doesn't appear, though "Ho Chi Minh" alone does:
Things about which Hacker News cares being down, and of which it has noticed:
Skype network is down, possibly under viral DoS attack. Lessons?<br></br>Is this why Twitter is down? Their Engineer Speaks<br></br>Amazon is down ... implications for AWS?<br></br>The Website Is Down (Hilarious 10 Minute Video)<br></br>Matthew Simmons: The only way is down<br></br>GitHub is down<br></br>KK on Unabomber: pounce on [technology] when it is down and kill it before it rises again<br></br>Yes, Rackspace Is Down And So Are Many Of Your Favorite Sites<br></br>Tell HN: Authorize.net is down<br></br>Dreamhost is down. All of it.<br></br>Most of Slicehost is Down<br></br>Ubisoft DRM authentification server is down, Assassin's Creed 2 unplayable<br></br>Dropbox is down<br></br>Heroku is down for the third time today<br></br>Tumblr is Down – Fans Angry<br></br>Great. Skype is down.<br></br>Reddit Is Down To One Developer<br></br>AWS is down, but here's why the sky is falling<br></br>Amazon EC2 EU-West is down<br></br>Reddit is down for 12 hours protest SOPA and PIPA.<br></br>Java.sun.com is down again - breaking bad apps across the land<br></br>Heroku is down<br></br>Tell HN: Heroku is Down (update: recovering as of 10PM PST)<br></br>AWS is down due to an electrical storm in the US<br></br>Heroku is down again<br></br>Google Talk is down<br></br>GoDaddy's DNS Service is Down<br></br>Github is down<br></br>Netflix is Down<br></br>Hacker News is down, so we made five issues free<br></br>This site is down because the owner stiffed the web designer<br></br>Dropbox is down<br></br>WhatsApp is down<br></br>DreamObjects is down<br></br>Facebook is down (09:08AM PDT Aug 1, 2014)<br></br>YTMND is down for temporary maintenance<br></br>Google Cloud Is Down<br></br>GitHub is down<br></br>DigitalOcean block storage is down<br></br>Firefox usage is down despite Mozilla's top exec pay going up<br></br>Slack is down<br></br>[dupe] Slack is down<br></br>Tell HN: GitHub is down again<br></br>Kiwi Farms is down across all domains as DDoS-Guard terminates service<br></br>Twitter's API is down?<br></br>
So ... after a side conversation where I said I'd probably not do it, I built a couple of command-line query tools to my HN front-page archive which let me look at specific sites or users, and summarize activity by year, weekday, and the corresponding other entity (sites submitted by users, users who submitted a specific site), along with story count, and point and comment totals and means.
It's ... a bit of a god-mode view onto the site.
Mind that HN does show user history at a detail level (posts and comments), but doesn't give a good way to get an overview.
I quickly discovered (divide-by-zero error) that one person never posts on a specific day of the week. And there are various comings and goings as well.
It is a handy reminder that our activities online are highly public, and can be assimilated with relatively little effort in many cases.
There are reasons I went all but completely pseudonymous well over a decade ago.
Many moons ago I made a conscious decision to abstain from working in fields involving intrusive data-based surveillance, including advertising, marketing, and (mostly) social networking. It's become apparent to me that my own personal boycott of such fields has probably had very little measurable impact on their development.
And so ... what I've been toying with over the past few days and the HN archive ... is something any modestly-motivated individual, let alone organisation, could do. The dataset itself is small (< 250 MB for the raw HTML, < 50 MB for the rendered content, slightly larger for my parsed output, and almost exactly 10 MB for the date summary (date, post number, site, submitter, votes, comments), and a few kB of awk code. My own not revealing this ... is a low hurdle for anyone else to do similar work. And in fact there are far more extensive analytic project, e.g.,:
And that's just one which is making itself known. It's the archives and analyses which aren't publicly acknowledged / accessible which are far more prevalent and as I see it, more troubling.
(This is one of the reasons that objections to public search/archival of Fediverse content strikes me ... and others (@alex has commented similarly a few times) as ... not only futile but largely misguided. The archives all but certainly do exist, they're just not serving the general public.)
So, more on that thing I said I wouldn't do but did anyway ...
Backstory: a dumb question lead me to crawl the HN front-page (FP) archive from 2007-present, just shy 6,000 pages, representing 178,162 stories, 52,400 distinct sites, and 43,491 distinct submitters. Each page has up to 30 stories, such that a fully-populated year has 10,950 or 10,980 (leap year) stories.
HN also provides a "leaders" page showing the top-100 members and "karma" (overall votes) --- latter being obscured for the top-10 members, though that can be found on their profile page. (https://news.ycombinator.com/leaders)
So ... I can get a summary of front-page activity for all leaders. It's ... interesting.
To assuage my guilt somewhat I'm only reporting overall / summary or anonymised stats. My goal isn't to out anyone specifically, but to give a sense of what HN front-page and "leader" member activity is like.
Seven leaders have no front-page posts at all, 17 have single-digit counts. The range is from 0 (obv!) to 1,183, mean 175.7, median 129, st.dev. 201.32, 10%ile: 3, 25%ile: 11; 75%ile: 253.5, 90%ile; 493.5.
Active years (years in which there is nonzero front-page activity) is ... all over the map -- there are members with results over 17 years, and with none at all.
What's ... peculiar ... is the points/karma% ratios. "Points' are votes on stories, "karma" is supposedly overall points (sum of story + comment moderation, less some for negative votes). The percentage of votes to overall karma ranges from 0 (no front-page activity) ... to 150.94%: more votes than cumulative karma. Points > overall karma (ratio > 100%) happens sixteen times, which is ... odd.
(Well, I mean, 16 is an even number, but the fact is odd-as-in-strange.)
One reason I've been doing this is to come up with some sense of overall quality metric. Engagements (votes and comments) are a highly-imperfect indicator, but looking at the arithmetic mean of votes and comments is interesting. I'm looking here at the average over all a member's front-page submissions:
Votes range from 0 to 634, mean 196.50, median 105,91, st.dev. 101.92, 25%ile: 150k 75%ile: 239.95.
Comments range from 0 to 323.75, mean 102.06, median: 96.38, 25%ile: 60.67, 75%ile: 123.16.
As might be expected, several members with lower-than-average submissions see high averages (there's more variance in small-n stats). One of the top-10 submitters (by average points and comments) has 514 FP stories, with an average of 236.37 points and 176.96 comments, and the most prolific submitter is very nearly median by votes and comments.
It's also possible to look at who's submitting a small or large range of sites by calculating a sites/stories% ratio. I'm finding, for example, one leader with 414 FP stories, from only 30 distinct sites, with the top site representing over half their submissions. (The site in question is legit and interesting, this does not appear to be spammy.) Several appear to favour their own personal sites / blogs, though again, not in a noxious way that I see. And 18 leaders have posted only a single item per site (each post is its own site), ranging from 1 to 20 FP items overall.
The ratio ranges from 0 (obv!) to 100 (obv!), mean 67.03%, median 71.83%, 25%ile: 51.82%, 75%ile: 89.72%.
In fact-checking my own comment, I found that my success rate in reaching the HN front page is not the roughly 10% I'd thought.
It's pretty much spang on 3%, which is the overall site average.
That's based on my archive's count of my own FP submissions (60) and Algolia search's results for all my article submission, whether or not they hit the front page (1,974).
So I guess I'm just about average.
This gives me the idea of checking against the HN Leaders list to see if anyone's markedly above 3% for FP placements.
HN Front Page: Foreign Policy Top 100 Global Thinkers (2014)
I pulled a copy of the "global thinkers" list I'd used as an indicator of website salience in a 2015 study.
The HN front page offers a limited opportunity for matches --- titles are 80 characters only, and HN's editorial policy is to not list authors of works, so what will show here is likely a subset of actual mentions.
That said: nearly a quarter of the list (23 entries) appear, from 1 to 11 times each. Paul Krugman (11), Lawrence Lessig (10), and Richard Dawkins (10) top the list.
1 Paul Krugman: 11<br></br> 2 Lawrence Lessig: 10<br></br> 3 Richard Dawkins: 10<br></br> 4 Freeman Dyson: 9<br></br> 5 Daniel Kahneman: 8<br></br> 6 Noam Chomsky: 8<br></br> 7 Jaron Lanier: 6<br></br> 8 Steven Pinker: 5<br></br> 9 Daniel Dennett: 4<br></br> 10 Christopher Hitchens: 2<br></br> 11 Craig Venter: 2<br></br> 12 Edward O. Wilson: 2<br></br> 13 Jared Diamond: 2<br></br> 14 Richard Posner: 2<br></br> 15 Steven Weinberg: 2<br></br> 16 Thomas Friedman: 2<br></br> 17 Gary Becker: 1<br></br> 18 Hernando de Soto: 1<br></br> 19 James Lovelock: 1<br></br> 20 Larry Summers: 1<br></br> 21 Martha Nussbaum: 1<br></br> 22 Peter Singer: 1<br></br> 23 Salman Rushdie: 1<br></br>
Hacker News Analytics: ~3% of submissions reach front page, with half of comments on FP articles
This is a finding based on maths and a previous study by Whaly in 2022 based on HN 2021 activity, rather than my own crawl, though it's informed by the latter.
The HN front page is a limited resource --- there are 365 * 30 == 10,950 front-page slots in a year, another 30, or 10,980, in a leap year, and regardless of site activity over a year, those slots are fixed. It's somewhat of a reminder that regardless of how much information we can access, our time to process that information is finite. Or as Herbert Simon observed: what information consumes is attention.
Whaly saw 386,663 total story submissions for 2021. I'm pretty sure that this is net of moderation (user flags, auto-kills, spam detection, voting-ring detection and the like). But it works out to a hair under 3% of stories not catching on any of those tripwires which then land on the HN front page.
Mind that that's actually a somewhat low estimate, as a story may appear for part of the day on the front page but not be represented on the end-of-day front-page archive.
I'm now thinking of doing some spot checks to see what kinds of success rates individual submitters have in landing on the front page. From what I've seen, even well-known and popular members have at best a modest chance of success.
Whaly also give a total number of comments: 3,769,520. That I can compare to my own front-page stats for 2021: 1,859,933, or 49.34% of all comments. That is, half of HN comments appear on the 3% of stories which reach the front page. That percentage is lower than what I'd have expected, though it's still a very strong bias toward the front page.
(Now I want to complete another analysis I'd thought of: mean votes and comments by story position (1--30), by year. Hrm...)
So ... I'm playing with a report showing how often F500 companies are mentioned in HN submission titles.
As I've noted, most of my scripting is in awk (gawk), and it's ... usually pretty good.
I'm toying with a couple of loops where I read all 178k titles, and all 500 company names, into arrays, then check to see if the one appears in the other.
The first iteration of that was based on the index() function, which is a simple string match. Problem is that there are substring matches, for example "Lear" (the company) will match on "Learn", "Learning", etc., and so is strongly overrepresented.
So I swapped in match(), which is a regular-expression match, and added W as word-boundaries.
The index-based search ran in about 20 seconds. That's a brief wait, but doable.
The match (regex) based search ... just finished as I'm writing this. 13 minutes 40 seconds.
Regexes are useful, but can be awfully slow.
Which means that my first go at this --- still using gawk but having it generate grep searches and printing the match count only ... is much faster whilst being accurate. That runs in just under a minute here. I'd looked for another solution as awk is "dumb" re the actually output: it doesn't read or capture the actual counts, so I'll either have to tweak that program or feed its output to an additional parser. Neither of which is a big deal, mind.
Oh, and Apple seems to be the most-mentioned company, though the F500 list omits Google (or YouTube, or Android), listing only Alphabet, which probably results in a severe undercount.
Note I didn't even try "Meta", though let's take a quick look ... yeah, that's a mess.
Up until 2021-10-28, "Meta" is a concept, with 33 entries. That was the day Facebook announced its name change. 82 total matches (so low overall compared to the earlier numbers above), 49 post-announcement, of which two are not related to Facebook a/k/a Meta. Several of the titles mention both FB & Meta ... looks like that's four of 'em.
So "Meta" boosts FB's count by 45.
There are another 296 mentions of Steve Jobs and Tim Cook which don't also include "Apple".
And "Alphabet" has 54 matches, six of which don't relate to the company.
gagejustins's HN analysis has inspired me to take a crack at typifying Hacker News front page stories by type.
Whilst he'd manually assessed each front-page story, I'm classifying by site, so that an NY Times article on, say, quantum computing would still be described as "general news".
I've classified 10,200 of 52,642 domains, the first 300 or so manually, much of the rest using regexes and imputation (e.g., ".edu", ".gov", and sites on Blogspot, Substack, Medium, etc.).
Results by story count:
1 13782 general news<br></br> 2 13398 software<br></br> 3 10473 tech news<br></br> 4 8677 blog<br></br> 5 7651 academic / science<br></br> 6 7294 n/a<br></br> 7 4750 ???<br></br> 8 4600 business news<br></br> 9 3546 corporate comm.<br></br> 10 1504 general magazine<br></br> 11 1291 general information<br></br> 12 1162 general interest<br></br> 13 1132 technology<br></br> 14 1099 videos<br></br> 15 1073 social media<br></br> 16 975 government<br></br> 17 568 corporate comm<br></br> 18 559 tech discussion<br></br> 19 505 tech law<br></br> 20 251 tech publications<br></br> 21 171 tech blog<br></br> 22 170 science news<br></br> 23 136 business education<br></br> 24 104 corporate comm. <br></br> 25 103 video<br></br> 26 99 corporate commm.<br></br> 27 96 general discussion<br></br> 28 80 misc<br></br> 29 71 technology / security<br></br> 30 61 law <br></br> 31 59 webcomic<br></br> 32 49 translation<br></br> 33 48 health news<br></br> 34 47 images<br></br> 35 46 podcast<br></br> 36 32 law<br></br> 37 7 legal news<br></br><br></br> Unclassified: 93213<br></br><br></br>"n/a" indicates no site, e.g., an Ask, Tell, or Show HN post.<br></br><br></br>'???' indicates I couldn't (quickly) assess a domain. Examples: 37signals.com, readwriteweb.com, thenextweb.com, archive.org, anandtech.com, avc.com, docs.google.com, righto.com, slideshare.net, infoq.com, hackaday.com, gamasutra.com, marco.org, smashingmagazine.com, highscalability.com, catonmat.net, centernetworks.com, jvns.ca, scribd.com, about.gitlab.com, cloud.google.com, alleyinsider.com, msn.com, firstround.com, axios.com, openculture.com, onstartups.com, ejohn.org, dadgum.com, shkspr.mobi, mixergy.com, geek.com, gmane.org, foundread.com.<br></br><br></br>"cproorate commm." is an obvious typo. This is very rough code & classification.<br></br><br></br>#HackerNewsAnalytics #MediaAnalysis #HackerNews<br></br>
I'm continuing to play with this, and have classified a whole mess more sites (reminder to self: update that count) (response to self: 13,150 sites classified).
So that's about 25% of all sites that are classified. Looking by story count ... it's about 55% of all FP stories. (Power laws are your friend here...)
Looking at my current breakdowns (and again, this is all VERY ROUGH):
My classifications are rough and I may revisit these. "blog" covers a lot of sins, though most are tech blogs (which makes "technology blog" redundant).
What I'd really like to do is to look at how trends vary over the years. Perhaps also by day of week / month of year. Finally answer that age-old question of whether HN is turning into Reddit....
As noted above, this is based on classifying the site rather than interpreting the title or reading the source article, so it's all a bit wobbly.
(This post formats better on toot.cat or on sites that render Markdown.)
I've got this to about 60% of posts classified (by submitted site). I can continue winnowing this down, though there's obviously diminishing returns.
I've also revised my analysis code so that anything that's not classified defaults to "UNCLASSIFIED", without having to explicitly code that in the sites file.
I'm thinking of how I might crossref / correlate the site-based findings with title-based analysis. I'm also thinking of looking at average comments / votes by classification, as well as looking at the ratio of comments to votes (HN uses this as a very rough "flamewar" heuristic, though on somewhat shaky grounds IMO).
My sense is that many of the less-frequently-posted sites will turn out to be blogs of some form. I'm thinking of how I might assess this without having to key all of them.
<stage_whisper> random sampling <\stage_whisper>
One issue issue for less-frequently-occuring sites is that it's easy to code those which match a pattern (twitter, blogspot, livejournal, medium, substack, etc.) than those which are idiosyncratic. Note that a lot of Medium blogs don't appear on Medium domains, as well.
OK, current stats are 63.5% of posts classified, with 29.8% of sites classified, a/k/a the old 65/30 rule. The mean posts per unclassified site is 1.765, so my returns for further classification will be ... small.
I could pick just under 4% more posts by classifying another 564 sites but ... that sounds a bit too much like work at the moment. Compromises and trade-offs.
Now to try to turn this into an analysis over time.
I've been working with a summary of activity by site, so running analysis has been pretty quick (52k records and gawk running over that).
To do full date analysis requires reading nearly 180k records, and ... hopefully not having to loop through 52k sites for each of those. Gawk's runtimes start to asplode when running tens of millions of loop iterations, especially if regexes are involved.
So ... I'm starting to get the reporting by site classification across years down and ... it is interesting.
Preliminary and buggy code yet. Also this is highly dependent on how I've actually classified sites.
I've got a few classifications I'd wanted to keep an eye on:
Programming-specific sites. A lot of this is github and gitlab, basically, software projects with code. I'm distinguishing software (which is mostly about use) and programming which involves, or at least anticipates, actual development.
"Political commentary". I used this as a description for ... highly political sites (spot-checking to see what stories actually hit the front page, though I should be more robust in that). The list: reason.com, rt.com, bostonreview.net, alternet.org, cato.org, rootsofprogress.org, breitbart.com, dailykos.com, mises.org, dailycaller.com, jacobinmag.com, rawstory.com, tribunemag.co.uk, hoover.org, heritage.org, theroot.com, wsws.org, adamsmith.org, manhattan-institute.org, theblaze.com.
And there's "academic / science" which is mostly university and academic press / journal sites.
Anywho....
... at least from initial takes, the trending on these does not suggest a trending toward sensationalistic topics and/or sites, but the opposite. Much more programming FP stories in recent years, fewer political commentary, and more academic/science items.
Presuming this holds up as I code further.
This is one of the fun things about data analysis: stuff jumps out at you, sometimes confirming hunches, but often radically violating preconceptions.
I want to look more closely at what happens in the lead-up and follow-on to the 2016 US elections cycle in particular....
Hrm. What does spike is cryptocurrency-specific sites in 2014. Though that falls off again. (I suspect as that discussion enters more mainstream sources.)
And "general info" and "general interest" sites seem to rise in recent years.
One concept Hacker News uses to moderate discussions is a "flamewar detector", which based on moderator comments over the years is triggered when a discussion has > 40 comments AND there are more comments than votes on the article.
That had long struck me as questionable, but it's now something I can look at and ... it seems reasonably accurate. I've calculated ratios of all 178,882 HN Front Page stories (as of 2023-6-31), and ... do I have some ratios.
Because of how I've parsed and processed data, it's not entirely straightforward to pull up the specific posts, though I can find those by the date and story position (ranked 1--30 on the page).
And ... yeah, the stories that tend to rate high based on this metric do tend to be sort of flamey.
The most ratioed post of all time was "juwo beta is released (at last!) Please use it and help improve it!", from 18 April 2007, at 21.0:
Sometime around 2009--2010 the flamewar detector seems to have been implemented and ratios tend to be much lower, though there are still some pretty spicy discussions. One from the National Institutes on Health titled "Mental illness, mass shooting,s and the politics of American firearms", posted on 26 May 2022 (for a story originally dating from 2015) is the highest-ratioed post after the flamewar detector came into use, at 5.99:
I find it interesting how being able to query my archive affords insights on HN which aren't available through the standard search tools. It's possible to look for specific keywords, or submissions or comments from a specific account, but searching for contentious posts isn't really A Thing.
I'm doing some further digging to see what patterns might emerge by site, though finding a good minimum number of front-page appearances is one question I'm looking at.
And, as usual for my data-dump posts, these read better on toot.cat than instances which don't pick up Markdown formatting. Click the date stamp to view original / formatted version.
Continuing my look at the comments/votes ratio, a look at sites which tend to focus on political commentary and their "spiciness". These tend to be well above mean (0.63), median (0.52), and tend to be a standard deviation or more from the mean (1 sd: 0.78, 2 sd: 0.92, 3 sd: 1.06).
Note that general news tends somewhat toward spicy, though not as much as the explicitly political sites. Of the 147 sites I'd identified as "general news", ratio statistics are:
HackerNews changed how it dealt with highly-active discussions around January 2009, based on evidence I see (far fewer spicy threads after that date).
I'm also seeing that spicy stories actually tend to rank slightly higher on the page (a lower "storypos", that is, story position, value), which is counter to my expectation. This may of course be due to selection bias --- moderators specifically lift limit on overheated stories, so that those stories that do survive are more appropriate to HN.
I'd like to look at semantic / sentiment elements here as well, words or phrases which seem more prevalent on high-ratio stories. Here my analytic methods work against me as the HN title of a post is often quite short and not especially descriptive, though with some examples (as with the mental health study mentioned earlier).
Add comment