#SocialMedia#USA#Twitter#Copyright#WebScraping: "A US district judge William Alsup has dismissed Elon Musk's X Corp's lawsuit against Bright Data, a data-scraping company accused of improperly accessing X (formerly Twitter) systems and violating both X terms and state laws when scraping and selling data.
X sued Bright Data to stop the company from scraping and selling X data to academic institutes and businesses, including Fortune 500 companies.
According to Alsup, X failed to state a claim while arguing that companies like Bright Data should have to pay X to access public data posted by X users."
So, how can we get a proper answer? Ten years ago, when I wrote “The Milwaukee Soup App”, I used the Kimono (which is long dead) to scrape the soup of the day. You could also write a fiddly script to scrape the value manually. It turns out that there is another option, though. You could use Scrapegraph-ai. ScrapeGraphAI is a web scraping Python library that uses LLM and direct graph logic to create scraping pipelines for websites, documents, and XML files. Just say which information you want to extract and the library will do it for you.
Let’s take a look at an example. The project has an official demo where you need to provide an OpenAI API key, select a model, provide a link to scrape, and write a prompt.
As you can see, it reliably gives you the flavor of the day (in a nice JSON object). It will go even further, though because if you point it at the monthly calendar, you can ask it for the flavor of the day and soup of the day for the remainder of the month and it can do that as well.
I am running Python 3.12 on my Mac but when you run pip install scrapegraphai to install the dependencies, it throws an error. The project lists the prerequisite of Python 3.8+, so I downloaded 3.9 and installed the library into a new virtual environment.
Let’s see what the code looks like.
You will notice that just like in yesterday’s How to build a RAG system post, we are using both a main model and an embedding model.
At this point, if you want to harvest flavors of the day for each location, you can do so pretty simply. You just need to loop through each of Culver’s location websites.
Have a question, comment, etc? Please feel free to drop a comment, below.
#SocialMedia#SocialNetworks#WebScraping#Discord: "An online service is scraping Discord servers en masse, archiving and tracking users’ messages and activity across servers including what voice channels they join, and then selling access to that data for as little as $5. Called Spy Pet, the service’s creator says it scrapes more than ten thousand Discord servers, and besides selling access to anyone with cryptocurrency, is also offering the data for training AI models or to assist law enforcement agencies, according to its website.
The news is not only a brazen abuse of Discord’s platform, but also highlights that Discord messages may be more susceptible to monitoring than ordinary users assume. Typically, a Discord user’s activity is spread across disparate servers, with no one entity, except Discord itself, able to see what messages someone has sent across the platform more broadly. With Spy Pet, third-parties including stalkers or potentially police can look up specific users and see what messages they’ve posted on various servers at once.
“Have you ever wondered where your friend hangs out on Discord? Tired of basic search tools like Discord.id? Look no further!” Spy Pet’s website reads. It claims to be tracking more than 14,000 servers, 600 million users, and includes a database of more than 3 billion messages." https://www.404media.co/a-spy-site-is-scraping-discord-and-selling-users-messages/
#AI#GenerativeAI#MidJourney#StabilityAI#GeneratedImages#WebScraping: "On Wednesday, Midjourney banned all employees from image synthesis rival Stability AI from its service indefinitely after it detected "botnet-like" activity suspected to be a Stability employee attempting to scrape prompt and image pairs in bulk. Midjourney advocate Nick St. Pierre tweeted about the announcement, which came via Midjourney's official Discord channel.
Prompts are the written instructions (like "a cat in a car holding a can of a beer") used by generative AI models such as Midjourney and Stability AI's Stable Diffusion 3 (SD3) to synthesize images. Having prompt and image pairs could potentially help the training or fine-tuning of a rival AI image generator model."
Dear friends of the #BSDCafe and the #Fediverse,
Bytedance is connecting to our server every few seconds. As I don't understand why and, more, I've already had problems with their rude behaviour, I've added this rule to nginx.conf:
case sensitive matching
if ($http_user_agent ~ (Bytespider)) {
return 403;
}
case insensitive matching
if ($http_user_agent ~* (bytespider)) {
return 403;
}
#Meta#Facebook#WebScraping#DataScraping: "One year after Meta sued a data-scraping company, a federal judge this week threw out Meta's breach-of-contract claim because the defendant obtained only public data from Facebook and Instagram.
Meta sued Bright Data in January 2023, making claims of breach of contract and tortious interference with contract. Bright Data is an Israeli company that collects data from various websites and offers related products to businesses.
"Bright Data concedes that it was bound to Meta's Terms while it had Facebook and Instagram accounts, and that it sells data collected from Facebook and Instagram," US District Judge Edward Chen wrote in a ruling issued Tuesday. "However, even viewing the evidence in the light most favorable to the non-moving party (Meta)... the Facebook and Instagram Terms do not bar logged-off scraping of public data; perforce it does not prohibit the sale of such public data. Therefore, the Terms cannot bar Bright Data's logged-off scraping activities."
Meta alleged that Bright Data violated Facebook and Instagram policies by developing and using "unauthorized automation software to scrape data from Facebook and Instagram, including users' profile information, followers, and posts that users have shared with others." The case is in US District Court for the Northern District of California."
When actively scraping, the main starting function is
document.querySelectorAll()
This will return a NodeList, which typically one will use a for-loop to loop over each item.
On each item either the querySelector or querySelectorAll will be applied recursively until all specific data instances are extracted.
This data is then saved into various formats depending on future processing, either as on object in an array or as a string, which is then saved either to the localStorage, sessionStorage, IndexDB, or downloaded via a temporal link.
#SocialMedia#Twitter#WebScraping#CFAA#Musk: "The recent lawsuit that X Corp., formerly Twitter, filed against a nonprofit called the Center for Countering Digital Hate illustrates the ongoing threat to researchers—whether they’re nonprofit researchers, academics, journalists—who engage in public interest investigations of platforms and often speak critically about platforms. They will often find things that the platforms are not happy for them to publicize.
In this case, the Center for Countering Digital Hate published reports that talked about what it termed hate speech and misinformation that remained on the Twitter platform. In doing this research, they had to scrape public information on Twitter. They analyzed posts at scale and they argued that Twitter allowed content to remain up that violated Twitter’s own policies on content. X Corp. sued CCDH and their theory was that CCDH violated the terms of service and that that’s a breach of contract.
They’re seeking tens of millions of dollars in damages based on the reputational harm to them of these reports, which they say caused advertisers to flee."
I've been also thinking about writing a tutorial on #WebScraping for research, which is already a gray area, but now with all these "AI" startups and companies doing that to make money off of everything we put online...I don't know, I don't want someone to use my work for that 😬
Publishers including The Guardian, Daily Mail and The Telegraph are believed to have been targeted by a bot operated by the Beijing-based tech giant Bytedance.
The company has said its bot, dubbed Bytespider, has been deployed for “search optimisation” purposes.
However, news organisations are concerned that their articles are being used without permission to train chatbots and have raised concerns about copyright violations."
If you want to get started doing Web scraping with Python, this tutorial is very well done.
By Cody Winchester for the NICAR @IRE_NICAR (National Institute for Computer Assisted Reporting - that is, data journalism) conference early this year https://github.com/cjwinchester/nicar23-python-scraping
Until there's comprehensive regulation against the #webScraping of data used for training of #AI models (that benefits only the #bigTech), there are some tools that can help.
This #Kudurru looks interesting, and it offers a service in the form of a #WordPress plugin too.
#AI#GenerativeAI#Google#Bard#Copyright#WebScraping: "As compared to the class actions against Open AI, this class action seems to be directed even more precisely to the core issue of the Gen AI tools – their alleged training via resources made public on the internet and/or protected under copyright laws – combining potential legal issues on both the IP and the privacy fronts (not to mention due to bias in the algorithms). Whatever the result of such class actions, this seems to be a timely occasion for parties to clarify and judges to assess legitimacy of Gen AI tools based on a deep analysis of the technical functioning and composition of the training datasets. The fact that this is the main goal of the class action seems to be supported by the proposals introduced by the plaintiffs for a governance scheme for all Gen AI models."
I hate Google, but #GoogleColab can be handy.
We had some teaching examples of #webscraping with #selenium on colab because installing the webdriver locally can be challenging for some students.
Google broke selenium scraping on colab. :((
Any #MyBinder suggestions? Something else..?
#AI#GenerativeAI#WebScraping#DataMining#Privacy#DataProtection: "What kind of info are we comfortable forking over to the AIs, if any? Right now we are in the midst of a destabilizing moment. It’s alarming, yes, but it’s also an opportunity to renegotiate what we do and do not want to hand over to tech giants that have been gathering our personal data for decades now. But to make those sorts of decisions, first we have to know where we stand. What are the websites and apps we use every day doing with our data? Are they using it to train their AI systems? What can we do about it if so?
A good rule of thumb, to begin with: If you are posting pictures or words to a public-facing platform or website, chances are that information is going to be scraped by a system crawling the internet gathering data for AI companies, and very likely used to train an AI model of one kind or another. If it hasn’t already."
I've done a lot of work that I released as CC-BY, but I can't do that anymore, because I don't consent to have my work scraped for AI. That violates the BY part!
Twitter sues four unknown entities for 'unlawful data scraping' | Engadget (www.engadget.com)
Elon Musk blamed scraping by these unknown entities for Twitter's decision to put a cap on how many tweets a user can see per day.