Rather than investigate all the things where investigative journalism could potentially make a difference to real human beings' lives, the Guardian has chosen to pursue a plea for the majority owner of Meta's voting shares (61%) to step down from his position of power. Let us politely ask Sisyphus to stop pushing the boulder instead.
There's no reason to think this (silly) opinion piece is sapping the work of their investigative staff. In an attention economy it might even be drawing eyeballs to it.
No, the Guardian has chosen nothing, this is an OPINION piece by an external author, it has nothing to do with the Guardian staff. Learn the difference.
The Guardian (i.e. someone on its staff) chose to publish this opinion piece.
I spent 8 years of my life at a daily news organization, I know for a fact that reporters, editors, executives, and sometimes even the advertisers have a voice in what is published…whether opinion or not.
i kinda wish he would do that and then forget this reality. it did dawn on me that when he was talking about how great it would be to meet AI friends that this guy is just desperately trying to escape a world he feels rejected/hated at large in and get to one where he can feel beloved and accepted. All of these billionaires seem to have a similar weirdly sad story where they just want to haul everyone off to follow some dumb idea of a new world, buncha broken people.
My favorite real-world example that I worked on: I created a YouTube documentary series about corruption in a small town in Illinois. This required downloading thousands of hours of government meeting videos from YouTube, transcribing + chunking + embedding, summarizing meeting segments, running a couple passes of validation, then searching for interesting storylines. I also scraped thousands of public documents which exposed campaign finance violations and some truly nefarious stuff going on behind the scenes.
I did 90-95% of the work on this with a local GPU. I still ran Claude and Codex, but they were just writing code to orchestrate API calls against local models, local Whisper, etc. Doing the same with the public cloud would have cost thousands of dollars, instead it cost a few hundred dollars for a frontier lab subscription and tens of dollars for electricity.
I recently got into the DGX Spark and just a month ago, got an AMD Ryzen developer platform. Both are incredibly useful tools for some upcoming projects I'm working on (also related to government corruption), but in my limited experience of trying to run opencode on them, the "good models" are still completely unusable because the system prompt alone requires a minute of thinking.
Do you have a writeup of how you did it, which tools you used? I have a similar project on a much more mundane topic, where I want to build a searchable index of Youtube videos, blog posts, etc. based on their content.
I've been meaning to write something about how I did it, and have been putting it off for a long time. I wrote all the tools myself - by that, I mean that everything was a web app running in my browser.
My advice would be to first set up an automation for yt-dlp to pull the media, then use Whisper to build a transcription pipeline. Chunk the transcript based on desired result granularity, then store embeddings in local Qdrant. It helps to use an orchestrator to handle all of this - my current recommendation is Dagster (dagster.io).
My non-technical teammates use Claude Cowork extensively. We set up MCP servers to give them audited access to certain internal services, while they just have to think of MCP as a "connector" which they can configure in the GUI.
It's great to see SOC2 going in this direction. I've had to set up all kinds of compliance automation over the past decade (mainly Drata, Vanta, Oneleet), but at the end of the day it's a huge pain grabbing screenshots for what feels like a very performative process.
I asked a SOC2 auditor a while back about why they don't just make their own compliance automation dashboard, so we can pre-fill all their specific requests. They mentioned that certain rules are in place to separate evidence preparation from the actual auditing firms. However, after doing some armchair ChatGPT research, it seems like an audit firm could potentially be permitted to provide evidence management software, as long as you're not guaranteeing an audit result or preventing people from taking the evidence elsewhere. Perhaps you could shed light on the legal aspects and AICPA guidance here for providing evidence collection as AI skills to the companies you audit?
FWIW, Peter Norvig also gave an updated take on what programming will look like with LLMs around 2 years ago. It did give me some hope back then, but with the development of Fables and Kimis and Sols I think it's truly anyone's guess where the art of programming will go.
I've always spent 90% of my work time thinking and sketching what the architecture should be to solve a problem. I find that once I've worked that out, it's actually still faster for me to code it than to try to explain it and refine it in prompts. And the code ends up more succinct and covers the edge cases I've already considered. If I need a fast script to do one thing, or unit tests, sure an LLM is faster at writing those. But for clean code, nothing beats knowing what you're going to write and just writing it yourself.
Exactly. At my previous work we spent an awful amount of time planning and architecting the solutions and refining them. No matter the person who prompted or the model used, the solution was never close to satisfactory.
Layers of unnecessary abstraction, weird decisions (like wth it used triple pointers once) hard to debug issues and obvious race conditions. Segfaults are through the roof and people are burned out and gasping hard to solve the bugs. I called their attention to this at the time and I was ignored and a bit laughed about. I left and couple months later I know others are too.
... and 10x that for long term maintenance and evolution of the code. When you read your code later, understanding of it comes more naturally/faster than somebody else's code (at least for me). I break down bigger problems into small chunks in one way, other folks may view things differently.
> Another thing is that not many can afford Fable or similar and I don't think prices will go down, since it is already heavily subsidized?
Not sure if API usage is heavily subsidized, especially for Fable. Chat interface for free users is subsidized for sure. I think we'll see how costly it is to deploy a big model now that Kimi K3 weights have become available.
A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the entire underlying dataset (around 10 GB uncompressed) for free directly on the site.
Despite the "download all data" link being prominently placed on the front page, the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint. Even with CloudFront caching results and a fairly efficient backend setup, the monthly bill ended up with around $1k just going toward network ingress/egress, so I shut down the site the following month.
Yes, if I wanted to put a nice HTML interface over the query results then I would still end up with the same problem where some combinatoric explosion of query parameters to the `/search` endpoint, most of which are cache misses, leads to many many TBs of network egress.
Sure, but now you're moving the site from "Free data presented in a pleasant way to view" to a "pay-as-you-go database". Your audience shifts dramatically, and you lose the ability to share the data you're trying to present.
Because the crawlers would still have hammered their site, though.
(The GP post doesn’t actually meaningfully address the issue being raised. Adding BigQuery or whatever would not change the fact that (a) they already offered a method of getting all of the data in a cost effective way, and (b) the issue was that the crawlers hammered the site hard enough to make it economically unviable.)
The comment I’m speaking of was from a different user, and it’s giving a strong smell that both users are bots unfortunately. Maybe I’m wrong, maybe it’s a coincidence those exact strings of characters were formed in isolation from each other in the same thread. I hope I’m wrong, but it’s a red flag
I'm not a bot, and the other commenter was restating the question as the solution provided did not seem to actually solve the problem, just redesign the entire project (See comment from u/taneq).
We didn't form "those exact strings of characters in isolation from each other in the same thread" they are directly referencing my comment in a reply thread to said comment.
> The included traffic volume for outgoing traffic is 20TB. There is no bandwidth limitation. Overusage is billed with € 1.00 ($ 1.20)/TB. Incoming and internal traffic is unlimited.
the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint.
No, they decided that would be a great way to convince you of the narrative and persuade you to pay for "security" services that further the incumbent browser monopoly.
They're not "AI scrapers", they're DDoS'ers manufacturing consent.
There's been an explosion of vibe-coded scrapers that behave poorly and ignore robots.txt. Presumably OP disallowed crawling of the search endpoint for the reason stated (that crawling it would result in endless permutations of search filters.)
They were both more exhaustive and more frequent. I don't remember the exact numbers, but it was definitely over 100x the traffic from search engines. By the way, search engines were allowed under robots.txt - I did want all 11.5 million loans to be individually indexed so they would pop up in Google search results, and I actually did receive/forward multiple tips about fraudulent loans because someone searched a business name on Google. All of this traffic was barely a blip, and my AWS bill for my hobby data science account was only ~$50-$100/month.
When I looked at the logs after getting the billing alert, 99.99% of the requests were to the "/search" endpoint with virtually every permutation of ~10-12 facets in the query parameters. There was only one scraper, but it triggered an enormous amount of network egress since it ended up missing the cache on the majority of queries.
not the OP but I'd say that google crawls you once and AIs scrape your page every time someone asks them a question that they think your page might be relevant to.
Completely agree, I've worked at several companies where I saw the AWS bill balloon from five to six figures, usually because of pointless over-provisioning. However, it all became worth it when I saw Jeff Bezos go to space. [1]
I deployed a Lambda function behind CloudFront which rendered a simple HTML page with the query results from executing some SQL over the dataset. I served millions of page views for next to nothing because most pages were already in the cache.
I don't know where you get this expectation that people should anticipate that a crawler might try every possible combination of query parameters, thereby missing the cache on each one. Most people consider it a bitter and arrogant perspective, which is why this got downvoted.
If Boeing claimed a plane was airworthy, but it crashed because basic engineering controls were skipped, we have collectively put our faith in the NTSB to preserve evidence, run an independent technical investigation, etc. There is no such authority for software - most security auditors (SOC2, HITRUST, etc) are just looking at self-reported data.
Just take a look at the recent Epic vs. Health Gorilla lawsuit to see how nonexistent the protection is around exchanging your medical records, one of the most sensitive types of PII.
Missing from the article - the hacker first compromised Resolv Lab's AWS account, took a private key from KMS that was used to control minting, then managed to extract $25 million into ETH before all protocol functions were suspended.
^ this is a common security misconception in crypto. "We're using an HSM, they can't steal our private key." OK genius now you still have to secure the HSM.
There's no shortcut to MPC/multisig with 3+ keyholders.
> There's no shortcut to MPC/multisig with 3+ keyholders.
The whole concept of a stablecoin seems to be based on centralised trust.
Ultimately there is some org that has the fiat bank account, that mints and redeems the coins.
Nope, that is the foundation of bad stablecoin. Trustless decentralized stablecoin like DAI exist. People just largely don't do their homework and prefer scams that lure them in with promises of 'yield'
DAI and SKY are backed in large part by USDC, so they are not truly decentralized. It is possible in theory, but nobody has successfully done it so far.
It's possible in practice: that's how DAI worked originally. It's just not very competitive where the main customer -- traders -- want a lot of liquidity and razor thin spread.
DAI made some dumb decisions for market reasons recently but it was an actual stablecoin for a long time. It worked fine, they just decided to make it worse for some reason.
reply