tdp_org, 7 months ago to webdev

Google's indexers/crawlers/bots are ~75% of all indexer/crawler/bot traffic to www.bbc.co.uk & www.bbc.com.

I wonder what it is they do that's so drastically different to everyone else? Why do they need to make so many requests?

#webDev #webStats #SEO #Google

tdp_org, 9 months ago to Football

The Women's Football World Cup Final in HTTP access log lines processed (for a subset of our logs).
Funny how literally every football match follows this exact pattern - unless they go to extra time/penalties.
#WebStats #WWC #Football

tdp_org, 10 months ago to infosec

I ran the numbers for yesterday on how many HTTPS requests to www.bbc.co.uk & www.bbc.com did not include a TLS SNI header - by country. Highlights:

• Jordan is an outlier at > 17% non-SNI (3x any other)
• 90% (219) of countries have < 1% non-SNI
• 36% (88) of countries have 0 requests with no SNI
• H/T to Dominican Republic with ~800K requests, 100% of which include SNI
• DPRK (N. Korea) also had 0 non-SNI (but only 9 requests)
• UK has 0.06% non SNI (296M requests)
  #TLS #webStats #infoSec

tdp_org, 10 months ago to webdev

Still bends the mind a bit that we serve nearly 14M web pages every day to Search Engine indexers & bots (and those are just the bigger orgs indexers/bots). That's 4% of our traffic.

Also super strange how skewed towards Google this is, they're making 75% (over 10M) of those requests.

Might be interesting to plot the decline of Twitter actually...

Then there's "monitoring" which is everyone's "is the internet working" - nearly 32M of those every day.

#webStats #webDev #BBC

cazabon, 1 year ago to internet

#Canadian #media - well, the #legacy ones at least - are up in arms over the fact that #Meta / #Facebook is planning to block #news from appearing in #feeds in #Canada.

You managed to get a #law passed that means they would have to #pay you for the privilege of #linking to you, didn't you?

Now they get to choose whether to continue linking to you, or not.

You can't have it both ways with "You must link to us, but you must pay us to link".

I think they're in for a rude #webstats awakening.

tdp_org, 1 year ago to random

Yesterday I added a graph to track TLS ciphersuite usage over time & immediately spotted an anomaly.
In mid-Feb, on our commercial CDN, CHACHA usage dropped from ~10-12% to ~0.5% & stayed there. The same did not happen on our own CDN so it seemed unlikely to be client behaviour.
Raised it with the CDN vendor & they tied it to a release which accidentally changed the behaviour to pref CHACHA for clients for whom CHACHA is top pref.
Having good data is 💯
#TLS #CDN #Grafana #webStats