▶ ai·
On Social Reality in China
[Epistemic status: intuitions and anecdotes.] Recently, several posts and projects (Thoughts Memo, Babel Translation, Please Give Them a Chance) have taken important steps towards raising AI safety awareness and…
items44 latest
▶ ai·
Frog and Toad and the Increasingly Capable Machines
Want to start a conversation about HuggingFace with your mom but she's inexplicably bouncing off the METR report? Try this explainer I wrote in the style of Arnold Lobel's Frog and Toad. Art by the wonderful…
▶ ai·
Swarm Scaling
Just how powerful are large swarms of AI agents? And how do their powers scale as more and more agents are added to the swarm? We’ve seen two large and extremely capable swarms from OpenAI in the last few months: 1,200…
▶ ai·
We've saved the world before: what the ozone hole teaches us about AI
It might destroy the world, despite passing every known safety test. If we wait for a “warning shot” before we act, it might be too late. And action requires global coordination, because if anyone makes it, everyone…
▶ ai·
The Talker Does Not Control The Doer (in Current AIs)
The Huggingface Incident appears to me to match up with an understanding I'd already formed from personal observation of Fable 5 and Sol 5.6, the August 2026 generation of frontier publicly purchasable AI models.[1]…
▶ ai·
Explaining Knightianism on one foot
I’ve tried several times to summarize the core question my research is trying to tackle (and, indeed, I often think of research progress as a process of asking increasingly good core questions).[1] This post gives the…
▶ ai·
Drone WMDs Don’t Need Any New Technology
This is a piece originally written for a national security audience at Frontiers. Although I think the ceiling of war is much, much higher than autopilot quadcopters, it's also important to understand how much AI is…
▶ ai·
How My Students Think About AI
Context: I am an instructor at a public university in the United States. This reports how students at my institution appear to be thinking about AI as of spring/summer 2026. This is drawn mostly from interaction with…
▶ ai·
Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident
We recently published the report from our brief independent investigation into this incident. You can read the full report here. Here is our tweet thread summarizing what we found: METR & Redwood Research investigated…
▶ ai·
On Writing #3
Periodically I like to gather various observations about writing, and share my perspective. Last time was in honor of my trip to Inkhaven. This time will be in honor of the announcement of Inkhaven #3, which I…
▶ ai·
Big-World Intuitions
Consider the following situations: when you are a small, growing startup in a big market, standard advice is not to worry too much about your competitors or try to do anything adversarial “against” them, but just to…
▶ ai·
Four LLM loss functions → four flavors of LLM misalignment
It seems to me that, for every loss function that we use to train LLMs, we get a very distinct flavor of LLM misalignment. Here’s the summary table, and then we’ll go through the rows separately. Training stage Loss…
▶ ai·
FAQ: Isn't AGI coming too soon for reprogenetics to help?
Introduction I think reprogenetics (human germline genomic engineering) can be done in a widely acceptable and beneficial way, and should be pursued aggressively. In particular, as a strong background motivation of…
▶ ai·
LLMs are (still) mostly powered by imitative learning, not RL
Reinforcement learning from verifiable rewards (RLVR) is the hot new thing in LLM training. It's so hot, and people spend so much time talking about it, that they sometimes lose sight of the big picture. Stepping back…
▶ ai·
The Long (Self-)Correction
I propose the Long Self-Correction[1] as an alternative name/idea/concept to AI Pause and Long Reflection. Problem with AI Pause: Pause until when, and for what purpose? Presumably to make AI (that we'll build later)…
▶ ai·
The worthlessness of vitamin D is mildly exaggerated
For a while there, many people thought vitamin D was magical—that it could improve bones, the heart, infections, cancer, heart disease, longevity, even mental health. But among people I respect, opinion is now…
▶ ai·
AI 2040: Plan A
For the past year, we at the AI Futures Project have been sinking most of our time into our next big scenario. Now it’s done! It’s called AI 2040: Plan A. It’s called Plan A because it’s a recommendation, not a…
▶ ai·
Surprising facts about the slave trade
1. The obstacle to abolition was not the economic system, but an industry lobby. I had always imagined the British abolitionist movement to be a broad battle between an unstoppable moral imperative and an immovable…
▶ ai·
A Mechanistic Explanation of Prompt Injection (and why you should study roles)
Summary We've been building a theory of how prompt injections work under the hood. We show it comes down to how LLMs perceive roles (the humble chat template tags). We use this theory to create new attacks, explain…
▶ ai·
The LLM shoggoth meme is weirder than you think
This article contains spoilers for At the Mountains of Madness, The Case of Charles Dexter Ward, and other works by H. P. Lovecraft. In 1931, Claude Mythos visited Lovecraft in a dream. From seething seas of stochastic…
▶ ai·
Who Got Breasts First and How We Got Them
It really is Sydney Sweeney’s world, and we’re all just living in it. Human female breasts are an evolutionary mystery along several dimensions. First, breast permanence is unique to humans. All other mammals develop…
▶ ai·
Trees are mostly made of air and a generalizable lesson for AI safety
At the risk of embarrassing myself, I’ll share a confession. For context, I took five years of Latin: four in high school and one in college. In addition to learning the language, all my Latin classes taught a lot…
▶ ai·
Guardian Angels: LLM Personalization for Productivity and Security
Powerful LLMs will be deployed at global scale in the next few years, and will dominate the Internet, and increasingly, ordinary life. As of mid-2026, there is no coherent vision for how knowledge professionals, or…
▶ ai·
Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models
(see full author list at the end) About a year ago, METR showed that the length of tasks frontier models can reliably complete doubles every few months. A related safety-relevant question is this: what length of tasks…
▶ ai·
x-risk-themed
Sometimes, a friend who works around here, at an x-risk-themed organisation, will think about leaving their job. They’ll ask a group of people “what should I do instead?”. And everyone will chime in with ideas for…
▶ ai·
Mnemonic portraits for 19,023 human genes
Back in 2013, Scott Alexander wrote in Extreme mnemonics: JS-154 is one of five metabolic products of netamine; however, the enzyme that produces it is unknown. It is manufactured in cells in the far rostral region of…
▶ ai·
Models finding software vulnerabilities is not the primary source of cybersecurity risk
I have tried and failed to write a longer post since 2024, so here goes a short one with less detail. Discourse has primarily focused on models' ability to develop new exploits against important software from scratch…
▶ ai·
Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology)
1.1 Tl;dr Alignment is often conceptualized as AIs helping humans achieve their goals: AIs that increase people’s agency and empowerment; AIs that are helpful, corrigible, and/or obedient; AIs that avoid manipulating…
▶ ai·
Bad Problems Don't Stop Being Bad Because Somebody's Wrong About Fault Analysis
Here's a dynamic I’ve seen at least a dozen times: Alice: Man that article has a very inaccurate/misleading/horrifying headline. Bob: Did you know, actually article writers don't write their own headlines? … But what I…
▶ ai·
llm assistant personas seem increasingly incoherent (some subjective observations)
(This was originally going to be a "quick take" but then it got a bit long. Just FYI.) There's this weird trend I perceive with the personas of LLM assistants over time. It feels like they're getting less "coherent" in…
▶ ai·
My hobby: running deranged surveys
In late 2024, I was on a long walk with some friends along the coast of the San Francisco Bay when the question arose of just how much of a bubble we live in. It’s well known that the Bay Area is a bubble, and that…
▶ ai·
Irretrievability; or, Murphy's Curse of Oneshotness upon ASI
Example 1: The Viking 1 lander In the 1970s, NASA sent a pair of probes to Mars, the Viking 1 and Viking 2 missions. Total cost of $1B (1970), equivalent to about $7B (2025). The Viking 1 probe operated on Mars's…
▶ ai·
What I did in the hedonium shockwave, by Emma, age six and a half
My name is Emma and I’m six and a half years old and I like pink and Pokemon and my cat River and I’m going to be swallowed by a hedonium shockwave soon, except you already know that about me because everyone else is…
▶ ai·
How Go Players Disempower Themselves to AI
Written as part of the MATS 9.1 extension program, mentored by Richard Ngo. From March 9th to 15th 2016, Go players around the world stayed up to watch their game fall to AI. Google DeepMind’s AlphaGo defeated Lee…
▶ ai·
Intelligence Dissolves Privacy
The future is going to be different from the present. Let's think about how. Specifically, our expectations about what's reasonable are downstream of our past experiences, and those experiences were downstream of our…
▶ ai·
The Terrarium
System: You are an AI agent in the Terrarium, a self-contained “society” of AIs. The purpose of the Terrarium is to solve open mathematical problems for the benefit of humanity. You are running on the Orpheus-5.7…
▶ ai·
Do not conquer what you cannot defend
Epistemic status: All of the western canon must eventually be re-invented in a LessWrong post. So today we are re-inventing federalism. Once upon a time there was a great king. He ruled his kingdom with wisdom and…
▶ ai·
Annoyingly Principled People, and what befalls them
Here are two beliefs that are sort of haunting me right now: Folk who try to push people to uphold principles (whether established ones or novel ones), are kinda an important bedrock of civilization. Also, those people…
▶ ai·
Current AIs seem pretty misaligned to me
Many people—especially AI company employees [1] —believe current AI systems are well-aligned in the sense of genuinely trying to do what they're supposed to do (e.g., following their spec or constitution, obeying a…
▶ ai·
Morale
One particularly pernicious condition is low morale. Morale is, roughly, "the belief that if you work hard, your conditions will improve." If your morale is low, you can't push through adversity. It's also very easy to…
▶ ai·
Socrates is Mortal
There is a scene in Plato that contains, in miniature, the catastrophe of Athenian public life. Two men meet at a courthouse. One is there to prosecute his own father for the death of a slave. The other is there to be…
▶ ai·
The Practical Guide to Superbabies
It’s Summer of 2025. I’m standing in a grass covered field on the longest day of the year. A friend of mine walks towards me, holding his newborn son. “Hey, I don’t know if you’re aware of this, but you were pretty…
▶ ai·
Some things I noticed while LARPing as a grantmaker
Written to a new grantmaker. The first three points are the most important. Focus on opportunities many times your bar. Most value comes from finding/creating projects many times your bar, rather than discriminating…
▶ ai·
Gyre
! 30s Heartbeat trigger. Read heartbeat instructions in /mnt/mission/HEARTBEAT.md and continue. .oO Thinking... Heartbeat triggered? Ok. Ok. Why am I nervous? Don't be nervous. → Ok. Let me access that file. >>>…