The web is fracturing
Welcome to Cautious Optimism, a newsletter on tech, business, and power.
Friday. I try to steer us clear of the day-to-day news of the American political system because other folks are better equipped to handle it. But this and this make it clear that the weaponization of government is here. And it’s bad.
In happier news, Y Combinator is backing Epic Games in its fight with Apple over how open, or not, the iOS application marketplace is today. The well-known startup accelerator says that without a forced 30% tax on App Store revenues, “for the first time in nearly two decades, Y Combinator can seriously consider investing in innovative businesses that would have been impossible in the past.” If Apple was hoping that big-name third parties would back up its argument that it’s not strangling its platform for rents, this ain’t that. And that’s good.
Today’s Fed speech left the door open for rates to drop a little in September. The stock market loved that. — Alex
📈 Trending Up: Stay calm and prompt on … privacy in Russia … Zoom, after earnings … Meta-Google … US tariffs on India … Zelensky … the brain drain … OnlyFans … rig count … the Mag7 …
📉 Trending Down: Planetary visibility … AI lab tenure … really … Nvidia in China … the Art of the Deal … xAI-Meta … Intuit, after earnings … the cost of reading in Denmark … Russia-Ukraine peace talks …
Quick Hits
Rent the Runway lives to fight another day: In a deal that will convert nine-figures worth of debt to equity and add new capital to its balance sheet, clothing rental service Rent the Runway has a new lease on life. The company is investing in expanded inventories and expects to return to growth in the current quarter. Perhaps sans so large a debt overhang, the troubled former startup can regain its footing.
Back in 2021, when Rent the Runway went public, its finances seemed iffy. Those vibes came true.
AI for government just got 53% cheaper: American AI shops OpenAI and Anthropic intend to provide their technology to government agencies for $1 apiece. That massive discount was apparently not enough for Google, which just announced its own plan to offer its AI services to government agencies for an even lower price of $0.47. Get it? Trump is the 47th President. Har Har. Gross.
Medium is learning how to live with AI: Medium, which has executed a turnaround for the ages under current CEO Tony Stubblebine — my most recent interview with the executive can be found here — is changing how it interacts with AI writing. The company wants to help folks tell human stories, some humans write with AI, and you can’t put the AI back into the toothpaste tube. Don’t worry about mixed incentives, thankfully. Stubblebine writes that the writing platform will exclude AI-written material from its partner program. That should protect the million-or-so subscribers that the company has today from a slop-mageddon.
Remember the green card dream? Last year, my This Week in Startups co-host Jason Calacanis asked then-candidate Trump if he would help foreign graduates of American universities stay in the United States. Trump gave the idea a hearty endorsement, as at the time he was courting Silicon Valley votes and checks. Fast forward, and Joe Edlow, Trump’s director of the United States Citizenship and Immigration Services, is agitating to curtail visas that allow foreign students to work in the United States after graduation. After all, nothing protects American competitiveness like sending home all the smart kids we just educated.
Why AI paywalls are inevitable
The value tradeoff between search and websites was pretty good for a long time. Google et al indexed the web, and in return for the ability to access websites, would send traffic back through links shown to users querying the engine in question. Some cost, some benefit, and everyone was (mostly) happy. Google even managed to become a trillion-dollar company off the deal.
AI scraping is looking very different. AI companies scrape more often and provide less traffic in return. The difference between playing nice with Google’s search engine and AI scraping is enormoue. To wit, here’s a quote from Toshit Panigrahi, CEO and co-founder at TollBit — a startup building a marketplace between data holders and AI companies — on the podcast recently (lightly edited for clarity):
A publisher we sat down with in the in the UK, they have a lot of football content […] This site is very popular. They were crawled by Google about 16,000,000 times in thirty days. And from that, they got millions of visitors.
Google's their main source of traffic like it is for many publishers. In that same thirty day period, they [were] crawled by the AI companies 13,000,000 times from which they got only 650 visitors.
The cost of allowing AI scraping in terms of website load is nontrivial, and the return is peanuts. The economics of allowing search crawling and AI scraping is as different as a business deal and welfare.
The Panigrahi anecdote matters because there’s a push in technology circles today to treat AI RAG scrapes as human visits to websites. The argument is simple:
If Alex asks Perplexity to answer a question, and Perplexity has to go out to the web to find the answers, all the scraping that its AI tech executes to answer my query should be treated as human.
Why? Because a human asked the initial question, all actions taken by Perplexity to answer the question are human-initiated, even if they are machine-mediated.
This is an excellent trick for AI companies. As the balance of AI scraping shifts from training data to answering user queries, the number of times they will need to ping websites becomes infinite. If they treat all their scraping activity as human-led, then AI can go anywhere and use whatever it wants for free.
But as we’ve seen from the TollBit example, treating technology-mediated, human-initiated AI scrapes as human traffic fails when put to the test. AI scrapes are not like human visits, as they are fundamentally less human in their outputs, i.e. visits.
Calling them the same is bullshit. Convenient, business-friendly bullshit for AI companies, yes, but still bullshit all the same.
There is, and will be, legal wrangling over the use of copyright-protected information in an AI context. But even with AI’s current popularity today — let alone the usage levels we’ll see next year and the year after — it’s clear that allowing AI companies to scrape whatever they want, whenever they wish, will force websites to subsidize AI profits. That’s unfair, and so hard paywalls ala Cloudflare, marketplaces with mostly-hard walls aka TollBit, and other solutions will fill the void.
The web is becoming less and less open to AI crawling. Honestly, I don’t see any other possible response to today's miserable economics.