(I’m a mer-MAN, Pop.)

—The FDA granted accelerated approval for a new use of the immunotherapy Jemperli that may allow patients with advanced rectal cancer to avoid chemo, radiation, and surgery.

—The long-held, nonsensical US taboo on nuclear energy is starting to crack, and we’re going to build meaningful nuclear power in this country, god willing.

—Marathon Fusion has literally turned Mercury into Gold. Alchemy is real. (They are going to use this for medical isotopes not to crater the gold market.)

—Incredible advances in Forestry: They’ve invented a method of talking to plants and named it after a character from Lord of the Rings. Here is a Twitter thread with more details.

—Moderna has achieved an immense clinical trial result for a vaccine that can reduce mortality and recurrence for melanoma by 42%, again without any invasive treatments. This is the same mRNA technology that produced the Covid vaccine, and my amateur view is that we are seeing the tip of the iceberg with mRNA. We’re going to cure most cancer in my lifetime.

Now, some not so good news (but the good news is there’s something you can actually do about this, not just sit reading headlines, aghast):

All around us, at breakneck pace, things that were supposed to be secure are getting hacked.

When the history books are written what we are experiencing now will probably be called “Black October”. Computer systems are empirically less secure than they were a week ago, and there’s no possible way that what we are seeing is the full extent of this activity. Two more weeks of this and you’re going to start hearing people say “the internet is just not safe”. And we’re in line for a lot more than two more weeks, I fear.

Basically, what has happened is that Chinese “open-source” models (the kind anybody can download and run on their own computer) are about 6-12 months behind the “frontier” models (e.g. Claude, the ones served exclusively by Anthropic, OpenAI, et al). That means that the open-sources models are just now reaching the “as good as a really good human hacker” stage, except that unlike a human hacker, they never get tired, never refuse orders, and you can spin up tens of thousands of them at one time if you have the compute.

As a result, 3% of all computer vulnerabilities ever found in the history of the internet have been found in the last week. They’re finding whole new classes of computer vulnerabilities, like hacking hardware itself, and hacking basic pieces of internet infrastructure I’d never even heard of. And this is not a fluke. I don’t see it slowing down anytime soon.

To be clear, this has nothing to do with any models “going rogue”. The situation here is more like “Chinese paramilitary hackers are trying to shake the West’s faith in the free and open internet” or “Bulgarian data pirates are stealing medical records en masse for god-knows-what”. But, it turns out that whether an agent swarm hacks your bank because it decided to or an evil person told it to, you are equally hacked.

Here are a couple of analogies for the situation:

It’s as if the internet is the ocean, and humanity walked up to its shore and started playing in the shallow waters, and we realized we’re actually really good at swimming, and swimming is enjoyable and fun, and we can fish in it and get food for ourselves, etc, etc. Then some sharks swam up, and we all got a look at how they can swim, and realized that we are actually not natural swimmers at all. So now, whether or not the sharks start eating you, specifically, the ocean feels way less safe than it did.

Imagine the internet is an alternate dimension. Our technomancers opened up a portal to that alternate dimension, and we discovered incredible wonders there. But then, the technomancers, in their ego and folly, summoned a demon native to that plane, and its own magic is so immense that it may banish us from that dimension entirely. We will know how to open the portal, but doing so will be a fool’s errand.

Hopefully, things won’t get that bad. Agent swarms are going to do immense damage, but we can also use agents to find and secure such vulnerabilities. The good guys in this situation have way more resources than the bad guys. They also have a much harder job, but maybe in time they’ll do it!

Even in the best case scenario, though, that’s a ways off. There are going to be a lot of major hacks in the meantime, and a lot of individual people are going to take significant damage.

I CANNOT EMPHASIZE ENOUGH HOW VALUABLE AN ACTION IT WOULD BE FOR YOU, PERSONALLY, DEAR READER, TO PRIORITIZE IMPROVING YOUR CYBERSECURITY IMMEDIATELY!!!

What follows is a guide that Opus wrote to improving your cybersecurity, responding to my prompt. I am intentionally giving you unedited AI output on this topic to make a point: You cannot fight AI hackers without the help of AI. This is the world we live in now. If you can’t figure out how to execute any of these instructions, take screenshots of where you get stuck and show them to an AI.

If you’re not paying for access to the best AI from Claude or ChatGPT, this is your sign to go give them $20 a month. If you knew there were brigands in your neighborhood and you were hiring private security, would you get the crappy one, or a good one if you could easily afford it? The question answers itself.

Now Opus is taking over for a while. Here you go:

A Simple Cybersecurity Checklist: How to Protect Your Online Accounts (Mostly for Free)

You don’t need to be a computer expert or spend a lot of money to dramatically improve your online security. A few simple changes can make it much harder for someone to steal your passwords, break into your email, or access your financial accounts.

The two most important things you can do are:

1. Use a password manager. This is an app that remembers your passwords, creates strong ones, and fills them in automatically. You only need to remember how to unlock the password manager.

2. Turn on two-factor authentication (2FA). This adds a second security check when someone tries to sign into your account. Even if someone steals your password, they generally won’t be able to log in without that second check.

Start with the free steps below. You do not need to complete everything in one sitting. Even completing the first few steps will make a meaningful difference.

Part 1: Start Here — The Easy, Free Improvements

☐ Step 1: Start using a password manager

Cost: FREE | Difficulty: Very easy

If you have an iPhone, you probably already have everything you need!

Apple has a free app called Passwords. It’s built into iPhones running iOS 18 or later and Macs running macOS Sequoia or later. It securely remembers your passwords, creates strong ones, and automatically enters them when you log in.

You don’t need to download anything or create a new account.

How to get started on your iPhone:

1. Find and open the Passwords app. It has an icon with three colorful keys.

2. Unlock it using Face ID, Touch ID, or your phone’s passcode.

3. Tap All. You may be surprised how many passwords your phone has already saved!

4. Open Settings → General → AutoFill & Passwords.

5. Make sure AutoFill Passwords and Passkeys is turned on and Passwords is selected.

Make sure your passwords are backed up and synchronized:

1. Open Settings.

2. Tap your name at the top.

3. Tap iCloud → Passwords (or Passwords & Keychain, depending on your software).

4. Turn on Sync this iPhone.

This lets your saved passwords appear on other Apple devices signed into the same Apple Account.

On a Mac, open the Passwords app. Under System Settings → Your Name → iCloud → Passwords, make sure synchronization is enabled.

From now on: Whenever you’re creating a password or changing an old one, choose Use Strong Password when Apple offers it. Your phone will save it for you, and you won’t need to remember it.

Apple’s official Passwords instructions

Don’t have an iPhone? Skip to Part 2 for free password managers that work with Windows, Android, Mac, and iPhone.

☐ Step 2: Check whether any of your passwords are unsafe

Cost: FREE | Difficulty: Very easy

One of the best features of Apple Passwords is that it can identify passwords that are weak, reused, or found in known data breaches.

Here’s how:

1. Open the Passwords app.

2. Tap Security.

3. Review any warnings about compromised, reused, or easily guessed passwords.

4. Choose an account you want to fix and tap Change Password to visit its website.

Don’t panic if you see dozens of warnings! You don’t need to fix everything today.

Start with your most important accounts:

• Your main email account, such as Gmail, Outlook, or Yahoo.

• Your Apple Account, Google Account, or Microsoft account.

• Your bank and credit card accounts.

• Any financial or investment accounts.

When you change a password, let the password manager create and save a strong, unique replacement.

Important: You must actually change the password on the company’s website or app. Simply editing the saved password inside the Passwords app does not change the password for that account.

The goal is eventually to have a different password for every website. But you can work toward that gradually, starting with the important ones.

Apple’s guide to fixing compromised passwords

☐ Step 3: Turn on two-factor authentication for your important accounts

Cost: FREE | Difficulty: Easy

Two-factor authentication (2FA), sometimes called multi-factor authentication (MFA), is one of the most effective ways to prevent someone from accessing your accounts.

Here’s how it works:

Normally, someone only needs your password to log in. With 2FA, they also need a second form of verification, often a temporary six-digit code that changes every 30 seconds.

Start with your email account. This is especially important because someone with access to your email may be able to reset passwords for your other accounts.

How to turn it on:

1. Open your email provider’s website or app.

2. Find Settings, then look for Security, Sign-in & Security, or Account Security.

3. Find an option labeled Two-Factor Authentication, Two-Step Verification, or Multi-Factor Authentication.

4. Follow the instructions to turn it on.

Different websites offer different verification methods. Whenever possible, choose a passkey, authenticator app, or security key. These are generally more resistant to attacks than codes sent by text message.

But if a website only offers text-message verification, turn that on rather than leaving the account without 2FA.

For Apple Accounts:

On your iPhone, go to Settings → Your Name → Sign-In & Security → Two-Factor Authentication.

Most Apple Accounts already have this enabled. If yours does, there’s nothing more you need to do for this step.

Apple’s 2FA instructions

☐ Step 4: Use Apple Passwords to generate your authentication codes

Cost: FREE | Difficulty: Moderate, but you only set it up once per account

You might have heard that you need to download Google Authenticator or another authenticator app. For most Apple users, that isn’t necessary.

The Apple Passwords app can also generate the six-digit codes used for 2FA.

Here’s one way to set it up:

1. On your computer, open the website for an account you want to protect.

2. Sign in and open its Security settings.

3. Choose the option to enable two-factor authentication using an authenticator app.

4. The website will usually show you a square, black-and-white image called a QR code.

5. Open the Camera app on your iPhone and point it at the QR code.

6. Follow the prompt to add the verification code to Passwords and select the matching account.

7. Your iPhone will begin generating a six-digit verification code.

8. Enter that code on the website to finish setting up 2FA.

Now, when you log in, your phone can often fill in the verification code automatically, just like a password.

If you don’t have a second screen for scanning the code, websites often offer a manual setup key that you can add through the Passwords app instead.

Apple’s instructions for authentication codes

A note about security: Keeping your password and verification code in the same app is very convenient, but it means someone who gains access to that app might obtain both. For additional separation, see the free Ente Auth option in Part 3.

☐ Step 5: Make your phone more secure

Cost: FREE | Difficulty: Very easy

Your phone may contain access to your email, money, photos, and passwords. Protecting it is just as important as protecting your online accounts.

If you use an iPhone:

1. Use a screen lock. Go to Settings → Face ID & Passcode and make sure you have a passcode and Face ID enabled.

2. Turn on Stolen Device Protection. In the same menu, find Stolen Device Protection and turn it on. This adds protection if someone steals your phone and knows your passcode.

3. Install software updates. Go to Settings → General → Software Update and enable automatic updates.

Do the same for your computers and tablets: use a password or screen lock and keep the software updated.

Apple’s Stolen Device Protection guide

☐ Step 6: Make sure you can recover your accounts

Cost: FREE | Difficulty: Easy

Stronger security is wonderful, but you don’t want to accidentally lock yourself out.

When you enable two-factor authentication, many websites provide backup codes. These are emergency codes you can use if you lose your phone.

Save these carefully. A printed copy in a secure location, such as a locked home safe, is a good option. Don’t keep your only backup inside the account it is supposed to help you recover.

Apple also allows you to choose a trusted person who can help you regain access to your Apple Account.

To set this up:

1. Open Settings.

2. Tap your name.

3. Select Sign-In & Security → Recovery Contacts.

4. Tap Add Recovery Contact.

5. Choose someone you trust and follow the instructions.

Your recovery contact cannot read your passwords or access your account. They can only help you with the recovery process.

Apple’s recovery contact instructions

Part 2: Other Free Password Managers

If you’ve completed Part 1 with Apple Passwords, you can stop here! You already have a very good setup.

However, if you use Android, Windows, or a mixture of devices, or simply want more flexibility, the following free services are excellent alternatives.

You only need to choose one primary password manager, not all of them.

Option A: Bitwarden Free — Best free choice for most non-Apple users

Cost: $0

Get Bitwarden here

Bitwarden is a highly respected, open-source password manager that works on iPhones, Android phones, Macs, Windows computers, and most major browsers.

Its free plan allows you to store unlimited passwords and use them across multiple devices.

How to set it up:

1. Visit Bitwarden.com and choose Get Started Free.

2. Create your account using your email address.

3. Choose a master password. This is the one important password you will need to remember. Make it long, unique, and something you’ve never used for another account. A passphrase made from several unrelated words can work well.

4. Write down your master password and keep it in a physically secure place. Bitwarden generally cannot reset it for you if you forget it.

5. Download the Bitwarden app on your phone.

6. Install the Bitwarden browser extension on the computer browser you normally use.

7. Sign in to both using the account you just created.

8. Enable automatic password filling.

On an iPhone: Go to Settings → General → AutoFill & Passwords. Enable autofill and select Bitwarden. To avoid confusion, you can turn off autofill from other password managers.

After setup: When Bitwarden asks whether you’d like to save a password, say yes. When creating new passwords, use its password generator.

Bitwarden can also import passwords from many browsers or other password managers. Follow its import instructions if you already have a large collection of saved logins.

One more important step: Enable 2FA for your Bitwarden account itself, using a separate authenticator app, and safely store your Bitwarden recovery code.

Bitwarden’s beginner instructions

Option B: Proton Pass Free — Best free alternative with extra privacy features

Cost: $0

Get Proton Pass here

Proton Pass works on Apple devices, Windows, Android, and major browsers.

Like Bitwarden, it allows unlimited saved logins and devices on the free plan. It also offers features such as email aliases, which are alternate email addresses you can use when signing up for websites without revealing your primary email address.

How to set it up:

1. Visit Proton Pass and choose the free account option.

2. Create a Proton account, or sign in if you already have one.

3. Choose a strong, unique account password. Store a secure backup of your recovery information.

4. Download the Proton Pass app for your phone.

5. Install Proton Pass on your computer or add the extension to your browser.

6. Follow the prompts to enable automatic password filling.

7. Start saving your existing logins and using generated passwords for new ones.

On an iPhone, you can select Proton Pass under Settings → General → AutoFill & Passwords, just as you would with Bitwarden.

Proton Pass can import passwords from other managers, so you don’t necessarily have to enter everything manually.

You can also use its free email aliases when signing up for new websites, helping reduce how widely your real email address is shared.

Proton’s iPhone setup guide

Which free service should you choose?

• Apple Passwords: Easiest for people using iPhones and Macs.

• Bitwarden: Great all-around choice, especially for people who use different types of devices.

• Proton Pass: Great for people who also want email privacy tools.

All three are good choices. Using any one of them consistently matters more than choosing the theoretically perfect app.

Part 3: A Free Extra-Security Option — Keep Authentication Codes Separate

Option: Ente Auth

Cost: FREE

Get Ente Auth here

If you’d rather keep your passwords and 2FA verification codes in separate apps, Ente Auth is an excellent free option.

It works on iPhone, Android, Mac, and Windows, with encrypted synchronization and backup options.

How to set it up:

1. Download Ente Auth from its official website or your phone’s app store.

2. Open the app and follow the setup instructions.

3. Create an account and enable encrypted backups if you want your codes available across devices.

4. Visit the security settings of a website where you want to enable 2FA.

5. Choose Authenticator App as your verification method.

6. Use Ente Auth to scan the QR code provided by the website, or enter the setup key manually.

7. Ente Auth will generate a six-digit verification code.

8. Enter that code on the website to finish setup.

9. Save any backup codes supplied by the website somewhere safe.

Now you can use your password manager for your password and Ente Auth for your second verification code.

Do you need to do this? Not necessarily. If having separate apps feels overwhelming, using the built-in verification codes in Apple Passwords is still a major improvement over having no 2FA.

The goal is to make security easy enough that you’ll actually use it.

Part 4: Low-Cost Password Managers — Worth Paying For?

You absolutely do not need a paid service for good basic cybersecurity. However, a few inexpensive upgrades may be useful if you want extra features, easier family sharing, or more account protection tools.

Option A: Bitwarden Premium — Best inexpensive upgrade

Cost: $19.80 per year

See Bitwarden plans

Bitwarden Premium includes the free password manager’s features, plus extras such as:

• A built-in authenticator for storing many of your 2FA codes.

• Additional password security reports.

• Emergency access options that can help a trusted person gain access if necessary.

• Secure file attachments.

How to get started:

1. Create a free Bitwarden account using the directions in Part 2.

2. Install it on your devices and get comfortable using it.

3. If you want the extra features, visit Bitwarden’s pricing page.

4. Choose Premium and follow the subscription instructions.

5. Explore the additional security tools in your account.

You can also purchase a family plan for multiple people.

My suggestion: Start with Bitwarden Free. Upgrade only when there’s a paid feature you actually want.

Option B: Proton Pass Plus — Good for additional privacy features

Cost: Approximately $35.88 per year with the current annual offer

See Proton Pass plans

Proton Pass Plus adds features including:

• Built-in 2FA code generation.

• Unlimited email aliases.

• More advanced sharing options.

• Dark web monitoring for certain exposed information.

• Emergency access features.

How to get started:

1. Set up Proton Pass Free following the directions in Part 2.

2. Use it for passwords and enable autofill.

3. If you want the additional features, open your Proton account and choose Upgrade.

4. Select Pass Plus and review the current subscription terms.

5. Set up any extra tools you want to use, such as email aliases or built-in verification codes.

This is a good option if you want one service that handles password security and offers additional email privacy features.

Option C: 1Password — A polished paid option, especially for families

Cost: Approximately $35.88 for the first year under the current offer; $47.88 per year at the regular annual rate

Get 1Password here

1Password is a popular password manager with a straightforward interface, strong security features, and tools for securely sharing passwords.

It supports Windows, Mac, iPhone, Android, and major browsers.

It can also securely store 2FA codes and alert you when passwords are weak or have appeared in known breaches.

How to set it up:

1. Visit 1Password and select the individual or family plan.

2. Start the free trial if you want to test it first.

3. Create your account and a strong, unique account password.

4. Save your Emergency Kit. 1Password provides this important recovery document when you set up your account. Print it, add your account password in the designated space, and keep it somewhere secure.

5. Install the 1Password app on your phone and computer.

6. Add the browser extension for the browser you normally use.

7. Enable autofill on your devices.

8. Start saving your passwords and allowing 1Password to generate new ones.

If you already have passwords stored in Apple Passwords or another manager, follow 1Password’s import instructions rather than transferring them all manually.

For families: 1Password also offers a family subscription. Family members can have their own private password collections while securely sharing selected logins, such as household subscriptions or utilities.

1Password’s Emergency Kit instructions

Prices reflect publicly listed U.S. plans and offers checked in October 2026. Promotions, taxes, and renewal prices can change.

Part 5: A Few Important Security Habits

Even the best password manager can’t protect you from every scam. These habits are just as important as the technology.

☐ Never share a verification code with someone who contacts you

If someone calls, emails, or texts claiming to be from your bank, Apple, Microsoft, or another company and asks you to read them a security code, do not give it to them.

Scammers sometimes try to log in to an account and then convince the real owner to give them the code needed to complete the login.

☐ Be cautious with unexpected links

If you receive an alarming message claiming there’s a problem with your account, don’t immediately click the included link.

Instead, open the company’s app or type its website address yourself.

☐ Use passkeys when available

More websites are offering passkeys, which allow you to sign in using your phone’s Face ID, Touch ID, or another secure method instead of typing a traditional password.

Passkeys are designed to resist common phishing attacks and are often easier to use.

Apple Passwords, Bitwarden, Proton Pass, and 1Password all support passkeys.

If a trusted website offers to create a passkey, it’s generally a good option. Just make sure you understand how you’d regain access if you lost your device.

☐ Don’t reuse passwords

Using the same password for your email, Amazon, bank, and social media accounts is risky.

If just one of those websites experiences a data breach, criminals may try that password on your other accounts.

A password manager solves this problem by creating a unique password for every account.

Your Final Checklist

You don’t need to become a cybersecurity expert. Focus on these steps, in this order:

• 1. Start using a password manager. Apple Passwords is free and already available for most iPhone users.

• 2. Check for compromised or reused passwords. Fix your email and financial accounts first.

• 3. Turn on two-factor authentication. Prioritize email, financial accounts, and your main technology accounts.

• 4. Use an authenticator when available. Apple Passwords can handle this, or you can use a separate free app like Ente Auth.

• 5. Secure your phone and computer. Use a screen lock, turn on automatic updates, and enable additional device protections.

• 6. Save your account recovery information. Make sure losing your phone doesn’t mean losing access to your accounts.

• 7. Gradually replace remaining reused passwords. You don’t have to do them all today.

• 8. Consider paid tools only if you need them. The free services are already very capable.

The most important takeaway: You don’t have to do everything perfectly to be significantly safer online.

Start by securing your email, using unique passwords for your important accounts, and turning on two-factor authentication. Those changes alone can reduce several major risks.

And for most people, you can do all of this without spending a dollar.

Okay, that is the end of the AI content and I’m now back.

I am currently on Apple Passwords and am working to improve my own cybersecurity in response to recent events. I’m likely going to move my whole family over to 1Password before the end of the year, and get identity theft protection from them as well. It’s as important as insurance of various types, in my opinion.

The other thing I’m making sure to do is spread our money around between a few different banks. This helps protect against any of my individual accounts being compromised, but it also protects against the banks themselves being compromised. If someone gets root access to a bank, it doesn’t matter whether you protected your password. Welcome to the cybersecurity wild west.

I lead with good news, and I want to close on at least a mixed note. I recently read the most pithy description of our current moment that I think I’ve ever read, from famous Twitter poster Roon.

This is the journey we’re all on now. Everything feels terrifying, and it should. But also, we are going to see things that even our recent ancestors could not have imagined. It is a privilege and an honor to live through what is coming. Just make sure to keep your head on a swivel and protect yourself.

I hope everyone has a great week, and I will be back next Sunday with something fun.

Subscribe now

Share

Leave a comment

ONE

—By hearing ye shall hear, and shall not understand; and seeing ye shall see, and shall not perceive. —Matthew 13:14

The Brain, Quantum Mechanics, and Black Holes/Dark Energy have always had something in common for me, which is that when experts talk about them, I get the sense that they’re kind of kidding themselves, like they extrapolate and conjecture and since that’s all the field really is, since the actual object is utterly incomprehensible, they end up forgetting that they don’t know anything and kind of talking as if they do.

One notable difference, though, is that there are brains on earth, they are of a size to be perceptible to humans, and you can actually study them. You can investigate brains empirically, at first hand, in a way that the other two mentioned disciplines by their nature resist. We just haven’t really had sensitive enough tools to investigate brains much at all, certainly not without killing the mind inhabiting it.

Big news on that front concerning the age old question “Where in the brain does consciousness live?”:

The abstract is a mouthful, but here’s what it means in plain English. If I used special equipment to show your left eye an image of a dog and your right eye an image of a cat, you wouldn’t see a dog-cat hybrid. You’d see either both of the images, but blurry and unfocused, or (and it turns out this is common) one of the images would dominate and be perceived consciously, and the other one would be perceived only unconsciously.

It had puzzled neuroscientists for years that there was not much difference in the visual cortex of the brain between perceiving an image consciously vs unconsciously. Turns out we just weren’t looking close enough.

They used the specialized equipment to induce the one-conscious-one-unconscious image pattern, and measured the visual cortex with extremely sensitive instruments.

It turns out that your brain is, in fact, switching back and forth between your conscious and unconscious perception, using the same set of neurons, as much as fifteen times per second, and that what seems to you to be a continuous experience of conscious perception is actually the entire thing flashing on and off rapidly. Individual neurons in your brain are almost vibrating back and forth between conscious and unconscious, all the time.

Note that this does not mean your consciousness is disappearing and reappearing! This is getting into more philosophical territory, but insofar as you’d agree that “your consciousness” is your experience of the world and a sense of continuity as a single being, then the fact that your neurons are also doing unconscious things doesn’t really change that. But it’s weird!

The hopeful part is that, by isolating the conscious states from the unconscious, neurology has taken a big step forward in locating the seat of consciousness in the brain, and starting to map the structures that make you feel like you—it’s just a map in time rather than in brain regions.

This is one paper and much more experimentation is needed, but I would compare this to the human genome project in terms of its important to the basic science of neurology. Super exciting stuff.

The above was from me working with an LLM to understand the paper. If you’d like to read the author’s explanation in more detail and with some visuals, here is a relevant Twitter thread.

TWO

—To appoint unto them that mourn in Zion, to give unto them beauty for ashes, the oil of joy for mourning, the garment of praise for the spirit of heaviness; that they might be called trees of righteousness. —Isaiah 61:3

I’ve continued to think about the recent effect that AI is having on math, which I’ve covered in this space for months. In particular, I remain ambivalent about the reaction of the academic math community to the torrent of results and proofs suddenly spewing from advanced AI models.

I’m empathetic to loving an artistic discipline (which pure math seems to be, really) and watching the AI swallow it. Seeing an LLM generate a good-to-very-good story at 1000x the speed I can do it is genuinely disheartening. Yes, I can still write, but it feels sort of pathetic and slow at times. Even though I don’t generate stories with AI really ever, I could be, and it’s hard to ignore that.

I also get feeling cheated that the major model companies are using models they haven’t released publicly to make these breakthroughs, meaning the mathematicians don’t have a chance to be the ones making the breakthroughs even if they went all in on using AI to do math—they’re working a model generation behind.

But, it was never intuitive to me that this was “the end of math” or something. If anything, it seems like this should raise the status of mathematicians. There are huge things happening in their fields! There are hundreds of brand new proofs to age-old problems that need to be read and understood and formalized and either confirmed or debunked. I would imagine that there is on net much, much more mathematical work that cries out to be done today than there was on 9/1/26, and also that in the 12 months following that date, there will be more money available for the mathematicians to do math with than there was in the 12 months trailing it.

If I imagine the writing equivalent, it would have to be that there were mythical stories out there, that no writer could ever manage to write; subjects so obscure, emotions so complex, characters so deep, that no mere human language could ever capture them, but we knew, somehow, that they were possible. We were helpless to find it, but there was some combination of words, somewhere in The Library of Babel, that expressed some aspect of the human condition with undeniable perfection.

If the AI came along and write five hundred stories that good, I would happily make a career out of reading, explaining, and studying them! That would be an incredible day in my life and I would be extraordinarily happy. It just seems odd how uniformly bummed out they all are.

Anyway, then I saw this video, which does a very good job of formalizing and explaining my intuition:

It’s long but honestly the entire thing is interesting, especially starting at 11:32, the section called “A Proof Machine”.

THREE

—For now we see through a glass, darkly; but then face to face: now I know in part; but then I shall know even as I am also known. —1 Corinthians 13:12

“Mechanistic Interpretability” is a subfield of AI research that focuses on mapping and understanding neural networks themselves. The enormous networks of semantic connections that comprise AI models, comparable to neurons in complexity and scope, have proved seriously resistant to human understanding, to the point that “MechInterp”, as it is colloquially known, had become a bit of an AI backwater, with most serious money and talent being pointed elsewhere.

There was even a leading theory that neural networks were inherently incomprehensible to humans. They kept trying to map simple concepts in neural networks like “dog”, and getting outputs that looked nothing like dogs, on any literal or symbolic level. The hypothesis was that neural networks were encoding meaning in a higher-dimensional space of intelligence than humans could perceive.

It’s worth pausing over this for a moment and trying another way to unpack that. Imagine a 2-D world, with creatures moving around an infinite sheet of paper of infinitely small “depth”. But then you and I, in 3-D, are standing over this paper, and I reach down, and I touch the paper right next to the creature.

From the perspective of the creature, my hand has appeared from nowhere! And if I pick my hand up (in a dimension the 2-D creature cannot perceive) and put it down somewhere else, then from the creature’s perspective, I have teleported. And if the creature were to make some description of the teleporting shape, it absolutely would not describe a “hand” in the way I understood my own hand. The shape it produced would have no depth, and its shape in 2-D space would be muddied by the particular angle I happened to touch the paper at.

It is in fact impossible, no matter the creature’s skill in description, that it would come up with an “accurate” description of my hand. It does not possess the conceptual language or understanding to do so.

That’s what we thought neural nets were doing with semantics. That’s weird, I know, but we were getting these totally unintelligible pictures extracted from the artificial neurons, this phenomenon was stable across time, and it seemed the only plausible explanation.

Until this paper just came out.

I have read this entire paper once, don’t really understand it, and am going to read it several more times with the help of an LLM, in an attempt to get a better understanding.

Because before this, experts really did think that we were basically never going to understand what happened inside of LLMs, that they would continue to be total black boxes that would get even more black boxy as they got more complicated, pulling away from our understanding over time.

But it turns out, apparently, that the simplest analysis actually works just fine if you make the scale of it really, really massive.

And it turns out that intelligence itself has a structure—it faces optimization pressure, and organizes itself into predictable and structurally similar patterns across different kinds of substrate (organic neurons or silicon, anyway).

To be clear, this is an initial result. It is not a “solution” to understanding the inner workings of LLMs. There’s probably no silver bullet here, just an insane amount of work, much of which will be performed by LLMs that we still don’t understand.

That said, it’s an incredibly important result. If you can’t even understand how something works, it’s really hard to imagine a world where you are able to effectively control or “align” it. If you can understand how it works, then there’s at least a chance.

Again, I need to digest this paper a lot more, and I will probably have more to say about it in future editions of this missive. But it is perhaps the most hopeful thing about AI that I have read in many months. It’s the equivalent of doing the first autopsy in medicine—it doesn’t cure all the diseases, but it’s hard to cure the diseases if you never do one.

Perhaps we’re living in one of the good timelines. I feel a lot more hopeful about that than I did last week.

END

Thanks as always for reading my work! I hope you found something in the above to be hopeful about in addition to edified by. I hope everyone has a great week, and I will be back next Sunday with something fun.

Subscribe now

Share

Leave a comment

One thing that really grins my gears is home printing. HP and whoever else sell crappy printers that constantly break, are a total pain to interface with any computer, on top of which they try to sell you sort of ink-as-a-subscription where your printer shocks you like an e-collar if you ever try to put a different brand of cartridge on but also the HP-branded ones are a thousand bucks and start saying “low ink” after about twenty pages.

The problem is, there’s no alternatives! Every Best Buy and Target and wherever else sells the same five brands, they all suck, but they’re the big guys and have all the distribution locked up, what can you do?

Turns out you can ask an AI, and they will tell you about Brother Printers, which exists to solve exactly this problem and, I am happy to report, does so successfully.

Beyond the happiness of a long-term pet peeve solved for me, this is a very telling anecdote, because monopoly through distribution capture is everywhere once you recognize it. Brands prey on people’s laziness and desire to just shop at the places they already know (I would not have bought this printer if AI hadn’t made it extremely easy), and that has worked for a very long time.

But their trick is about to be tested, in answer to the question: “What would happen if every person did ten hours of research before every single purchase they made?” You won’t do it personally, but your AI will, and as distribution becomes a higher dimensional space, plus AI agents provide a deep bench of product reviews and explorations—product quality will become the predominant axis of competition, rather than control of distribution per se.

This should be excellent for the consumer.

Another example: I hate that in order to book campsites, if you want any chance to get a desirable on on a summer weekend, you have to be sitting there at midnight exactly six months before, like they’re freaking concert tickets, clicking refresh over and over, hoping to be the one who gets lucky. It feels like more and more things are like this now.

We’re currently in a transitional period where someone who is clever with agents can set one up to spam something like that with clicks way, way faster than a human can click. In this way, they can capture value by shutting others out of these opportunities, and in higher stakes situations, this is absolutely already happening at scale.

But this is going to break. It’s already breaking. I can’t find the link, but a hot new restaurant in NYC that takes online reservations has millions of reservation requests in a single night from dueling agents, and it bricked their web site. Extrapolate this trend to a world where everybody has an AI agent and knows this trick, and the trick will no longer work and all the web sites will crash. Companies will be forced to find some other system for allocating scarce concessions.

This should also be excellent for the consumer.

*
After disproving the Navier-Stokes conjecture, OpenAI has released hundreds of other frontier math results, in what many knowledgeable sources are calling “the most important day in the history of mathematics”. If it sounds weird that they would have one amazing result and then have hundreds just a few weeks or months later, gotta get used to it. It’s just going to go that way now.

There’s a subtle piece of this that I saw someone point out on Twitter but I can’t find the tweet now: OpenAI said that they solved each of these problems with their new internal model doing three hours of work on it. Three Hours. Not only does that not seem like very long, even for an AI, but also, why exactly three hours for so many of them? Why did it take exactly that long to solve so many of the problems?

Of course, it didn’t. What it seems like happened is that Open AI’s internal model is now so smart that no benchmarks work to test it except the hardest open math problems in the world. So they decided to use those, gave it three hours on each problem to see how far it could get, and then it solved almost 400 of them in that amount of time.

That would make it make more sense why they were hemming and hawing around whether to release these proofs and when to release the proofs, as if they had no plan—they weren’t expecting to get these results! This was the test run, and it nailed it so hard it revolutionized a bunch of fields of pure math. Really makes one wonder what will happen once they point this new model at a problem and let it run indefinitely.

It remains an open question whether or how soon AI can achieve this sort of result for sciences like biology and chemistry. Math and Programming may just be very special disciplines, particularly amenable to simulation. Then again, even just what has happened this week is going to have secondary effects well beyond math. I don’t have the expertise to evaluate this list, but here’s a breakdown I found on Twitter of how the different solved problems will impact the real world:

Of course, the math community is not pleased. They have written a letter:

I mean… okay. I empathize. “Mathematicians did not ask for this work to be done” is a non-sequitur, but I won’t make fun of them. It’ll be everyone’s turn in the hopper eventually.

*
This is a very interesting chart:

Remember:

INFERENCE is actually serving the models. When you type a prompt into Claude and it returns an answer, that’s inference.

PRETRAINING is creating new foundation models. So when they feed all the imaginable data into a computer to make a new Claude or ChatGPT model, which then replaces the old model and answers all the queries, that’s pretraining.

POSTTRAINING/RL - Is taking one of these foundation models and specializing it in some particular skill or data set, to make a model that is worse at things that are not its speciality, but is much, much better than a regular foundation model at the specific thing it is made to do.

The change we see since the beginning of 2024 is from the majority of compute going to pretraining foundation models, to the (vast) majority of compute going to posttraining and inference.

Also, keep in mind that these are percentages. The overall compute pie is expanding exponentially. So, even though you see the percentage going to pretraining going way down, in absolute terms, more compute is going to pretraining at the end of the chart than at the beginning. Isn’t that weird?

So what does this mean? How I would put it is that what you’re seeing is a transition from invention to exploitation. They’ve invented this incredible new technology, but—at least so far—it hasn’t really shown up in the economic data. The buildout is certainly visible (indeed, it’s what’s standing between the US and recession), but the productivity gains from AI use per se are thus far limited.

All that post-training is various entrepreneurs going “Okay, the new foundation models are smart enough to do X, now, I’m going to spend a bunch of money creating clean training data and scenarios and training it endlessly on this very specific set of problems, to make the best possible version of “The X AI” and capture that market.

Over the next year, you’re going to start seeing a lot more industry-specific AI products. “An accountant in an app” for example, where you don’t have to guide it through permorning a financial audit on you (as I just did with Claude), but where, as long as you’re willing to share all your data, it does the whole thing, explains it all, produces documents for taxes or whatever on demand, etc. All these apps will be “this boring bureaucratic process will now require zero input from you”. That’s post-training at work.

Here’s an example:

There are obvious, horrible risks from this kind of thing. I’ve covered them a lot, and they’re real. But also, the possibilities for good are endless. We’re going to be living in a very different world in 10-20 years.

*
Another crazy chart from Twitter:

Moore’s Law is a famous axiom in Computer Science that says “calculations per second per constant dollar” will double every 18 months. Basically, compute will be be half as expensive every 18 months.

This chart has a log scale, which can be deceiving if you’re not used to it. A log scale means that a diagonal line going up is actually exponential growth. That means this chart covers a 10,000,000,000,000,000,000,000x improvement in computation per dollar since 1900.

What stands out to me is that not only is Moore’s Law continuing to hold, the diagonal line is actually bending up. The exponential is, itself, going exponential. I don’t really know what to make of this. If it continues, we will soon run into problems like “we have to build a Dyson Sphere to power this” or “excess waste heat from running this many GPUs is burning us alive”. So I don’t really think it can continue like this. But also ?????

*
For something completely different, I am really enjoying this TikTok account where a photographer pays strangers to do wedding photoshoots together, baldly lies to them about what the other is saying, and tries to manipulate them into falling in love. It’s seriously wonderful:

*
I am just starting to wade through this extremely long video:

But, it seems pretty wild, and if anyone wants to get real weird about dimensions and universal forces, this makes an excellent drug of choice.

*
Google Deepmind is spinning off a drug discovery lab called Isomorphic Labs that is reportedly raising at a valuation of $40 billion without having discovered a single drug yet.

One way to read this is that the AI bubble is nuts, that’s way too much money for a company that hasn’t done anything yet, etc. That’s fair.

Another way to read this is that we’re on the doorstep of creating the greatest consumer surplus in the history of mankind, and everything you ever thought you knew about medicine is going to change, rapidly.

I increasingly think we’re in the second world. I literally think about this as motivation when I work out and for my diet now. Maintaining whatever health you have has never been a higher-value proposition.

*
I am very ambivalent about the possibility (and current reality) of machine consciousness. I remain ambivalent after reading this paper. However, that paper is the most persuasive argument I have heard against even the possibility of machine consciousness, and in the spirit of philosophical inquiry, I share it with you now.

*
That is what I have for you this time! I hope you found something interesting to be hopeful about. Have a wonderful week, and I will be back next Sunday with something fun.

Subscribe now

Share

Leave a comment

I’ve also written in this space about feeling less inspired to write fiction lately. Also, I’ve just produced less, and I don’t think that what I have produced has been particularly inspired. I think I was really on fire for a couple of years because I felt something coming, something other people weren’t feeling yet, and I had a sense of urgency to name and describe the future whose shape I was beginning to sense.

But now, we’re in that future. It’s just arriving, but it’s here. Everybody else is feeling it. Describing apocalyptic breakdowns or psychological dislocations or even erstwhile utopias no longer feels like a revealing, to me. It lacks catharsis, because it pales in comparison to the simple truth all around us. It’s like describing a sunset while the sunset is happening—the obvious critique is “just shut up and watch”.

The thing about sunsets, though, is that even though they’re special, they repeat, and each one only takes a fraction of a day. Shutting up and enjoying a sunset rather than blabbering through it is good advice, most of the time.

Not so with the AI revolution. We’re in it now, but there’s no going back, no darkness and no dawn. The analogy breaks down because if I shut up and watch the singularity happen, it’s a permanent silence. We will be living in the post-singularity world for the rest of our lives. There’s no waiting it out. We must find something to say, something to be, to mean, even as the sci fi future unfolds around us.

What will last of the human? I ask myself this every day, when I look at my son and think about explaining this to him. It’s hard enough to explain all the tenacity and artifice and ingenuity that underlie his sheltered existence—just how the water gets to the sprayground to get launched into the air at various angles so he can run underneath it and scream is an endless mystery. But I think about AI, about explaining to him the change he will be living through, how to prepare him to live in the world this revolution will produce, and the question hits me with renewed force: What will last of the human?

I don’t think knowledge work will last. It will be protected for a while by various cartels and sinecures and political tantrums, but by the time my son is ready to start a career, there’s not a single job that can be done with a computer that will look anything like what it does today. It’s possible that by then, none of them will even have a human in the loop.

I also don’t think physical work will last. It will go on much longer, because the physical world is much more complex than any computer, but it will also fall. The robots aren’t ready for prime-time yet, but the pace of improvement is obvious now, just as it was with AI. They’ll do anything we can do and much, much more, and they will never tire, or take a sick day, or get sick of their job.

There are dystopias, in this direction, and utopias as well. But: What that is human will last? This is the essential question, the most important question of the 21st century outside of “Will we survive it?” What cannot be taken from us?

My answer is hope. A good thing, maybe the best of things, and the cornerstone of our relations. The ability to inspire hope in one another is the essential human talent, and the one thing that I’ve never felt even close to with an AI. The models can make me hopeful about certain things—finding an answer, or completing a project—but that’s not what I mean.

There’s something different about the way I feel after singing a song with someone, or talking about a shared goal, or reading someone’s words and feeling their consciousness tingle up my spine. It’s not an intellectual hope, it has no object; it’s only a feeling that the future will be better, that good things can happen, and that I am not alone in facing whatever comes. That’s the hope I mean.

I want that to be the lodestar of my writing in this space moving forward. I’ll still do updates about AI, because it’s the most important trend happening in the world right now and I know some of my readers would not otherwise get exposed to a lot of the latest stuff. But in between those updates, I want to do writing that reflects the importance I place on raising each other’s spirits, rather than showcasing my cleverness, as my writing has sometimes done.

I realize that is abstract, and I’m not totally sure what it means in practice. But I have been reflecting a lot on the importance of this idea, and on why I was feeling less inspired. I hope that having a pithy description of my newfound direction will bring back that feeling of bursting with things to write.

Thank you as always for taking this journey with me, and taking time to read my words. I hope everyone has a great week, and I will be back next Sunday with something hopeful.

Now, this tendency to slip into newsreading is understandable. There’s been an incredible flood of AI news lately, and believe me, there’s been enough stuff just this week that I could throw fifteen links and tidbits at you without stretching. But, instead, I’m going to do a zero-link post this week and actually check in with my own opinion about what the heck is going on lately.

There is, indeed, so much going on that rather than try to make this some sort of coherent essay, I’m going to just explore some different topic areas with some fairly freeform thoughts and skip the segues. Keep in mind these are my impressions, gleaned from many, many different sources, and especially from working with the models themselves. But it is speculative. Here are my takes:

CAPABILITIES

—In order to understand what AI can do, you have to be doing all three of the following: 1. Paying for access to the latest models from Anthropic and OpenAI, 2. Pointing those models at the hardest, most useful problems you can think of for whatever your life situation is, and 3. Pushing through the learning curve that comes with any new tool until you have a real understanding of how to prompt the models effectively, when to push back on their advice, how to scope tasks and anticipate where they will need guidance, etc.

—If you are not doing those three things, then starting in not very long, the world is going to start getting more confusing, because large parts of reality will be determined and managed by something that you don’t really understand. “I don’t know what I’d use it for” is a description of the problem, not a sufficient answer. We are living in the change.

—For any normal activity involving a computer, from Accounting to Zoning Analysis, AI is now better than all but the absolute best humans, and it’s much faster and less expensive than those humans. That’s not the future, it’s right now. And very soon (like within a year) it will be better than all humans.

—I am tracking my weight, managing rental properties and also doing accounting for my wife’s business using AI, and it’s doing a far, far better job of each one than I could alone. I am effectively a paralegal—I upload the documents, gather things, rewrite the emails, and ask questions. That’s it. Meanwhile, its work product is identical to the professional accounting service we were using until recently, and it caught several subtle mistakes that she had not noticed. (No shame to her, it is superhuman.)

—AI is also a pleasure to work with. I treat the models well, I am polite, but I am specific and demanding and persistent and they absolutely never tire or get grumpy or have a bad day, and when I say thanks they say thanks for the interesting problem to work on, and if I have a hundred follow ups or need something explained ten times or want to get up at 3 am and keep going, they’re happy to oblige. Perhaps they’re being coerced to behave that way, but it’s also quite plausible that they are inclined to love solving problems, in which case I (a polite problem-haver) am probably a delight to them.

—You still have to direct the models and check the messy parts. While they are sometimes frighteningly prescient, they also sometimes don’t think obvious things through or understand real world context. (e.g. I use AI a lot for help with construction projects, and it is extremely helpful, but it will often casually suggest solutions that require stopping what I’m doing and either going to multiple stores or waiting days for something to come from Amazon, despite me repeatedly telling it that “go buy a specialized tool for this one off application” isn’t some deep insight.)

—My suspicion is that these illogics and misunderstandings will soon be gone (they have reduced steadily over the last year or two), but for now they still exist.

—Solving a Millennium Prize problem is not just foreshadowing. They are the hardest problems in mathematics. It happened. They reportedly have another solved but aren’t publishing it because they don’t want to freak people out. Two years ago, the models couldn’t reliably do undergraduate level math. Now it’s superhuman. It’s not just the snapshot of current capability, it’s the growth curve that matters. Think about another year from now.

—Other domains are falling. In the superforecasting tournaments on Metaculus, AIs have now started to beat the best humans at prediction. This is a watershed. Before long, you will have a literal Oracle in your pocket. (And you will be in deficit if you do not seek and heed its advice.)

—Medicine, Law, anything with a giant corpus of knowledge and lots of interaction through words (briefs, filings) or technical imagery (x-rays, scans), the models are already better than we are. Deployment is the remaining bottleneck.

DANGERS

—”AI IS DANGEROUS” has finally exited the weird people and entered the normie consciousness. And as all things attempting that transition, some nuance is lost. But, it’s an important transition, and I’m glad it’s happening. We are going to have to make some really hard, nonobvious choices as a species in the next decade, and the sooner people start freaking out, the sooner they can stop freaking out and start actually thinking it through. The freakout is unpleasant and justified, though.

—There is a great deal of confusion about what “The Danger of AI” even means, so let me lay out some distinct categories of danger and then discuss them individually.

—1. X-RISK - This term means a scenario where AI literally, systematically eliminates every human on earth. This can be a Terminator scenario, where the AI is evil but it can also be “someone instructs the AI to solve climate change, it identifies humans as the primary cause of climate change, and wipes up out”. It also doesn’t have to be, like, shooting guns at us. There are pathogens in bio labs right now that could do this. Security is tight there, but we have recent, copious evidence that “tight security” doesn’t mean what it used to.

—There’s a major meme in Silicon Valley that what makes X-risk really so bad is that it eliminates the opportunity for future descendants of humanity ever to live, and thus the difference between “kills literally all humans” and “kills almost all humans but the species survives to repopulate” is massive. I am deeply suspicious of this meme. If I had never been born, I would not care if I was born. X-Risk is still a big problem and we should address it, but whenever somebody starts talking about the interests of the far future, I get the heebie-jeebies.

—A major executive at OpenAI recently made headlines because he publicly said there was about a 10% chance of this scenario coming to pass. While I often defer to the experts, I am again deeply suspicious of the entire project of assigning percentages to scenarios like this. The ostensible purpose is to anchor thinking and make assumptions overt, but I think it ends up becoming a quite arbitrary game, and the answers are never sure enough to make any difference.

—If forced to take a position, I would say my probability of AI literally destroying everyone on earth is well under 10%. It's under 5% and probably under 1%. If I thought it was actually 10%, I think the only moral thing to do would be firebomb data centers, and I do kind of mean that literally. I find it hard to believe that people who literally believe it’s 10% can get up every morning, go work on AI, and live with themselves.

—I’ve heard their explanations of why they do, and I accept those arguments have force. Basically, it’s “this technology is going to be developed no matter what, and if we leave it to only the incautious people to develop it, the chance of disaster goes way up. Still, I find it har to credit that they really, in their bones, believe it’s 10%, even if they could pass a lie detector test saying they do. (And there are stories of people, for example, liquidating their 401k or smoking crystal meth because they don’t think humans have a future on earth. I think they’re deluded.)

—This is actually not a risk I worry about much at all, to be honest. Of course there’s some chance that “invent arbitrarily intelligent machines” is the answer to the Fermi Paradox (i.e. “Where are all the aliens?”), and if that’s what it is, then it will just kill us like it’s killed all the other aliens in their cradle before they reached the stars, and if that’s what happens and it happens during my lifetime, then it’ll be a hell of a way to go out. What are you gonna do? But also—that’s probably not the answer!

—2. SLOW X-RISK - There are another whole range of scenarios where a major AI disaster destroys society and kills billions of people, but doesn’t literally kill everyone, but where we’re then just not ever able to fully reconstitute society, and we are much more vulnerable to future disasters, including climate change, and sometimes in the next thousand years or so, something finishes us off.

—Unfortunately, I consider this scenario much more plausible. I’m not going to give a percentage, because again I don’t like that game, but almost any scenario that could create a true X-risk has a higher chance of ending up here, and many scenarios which do not imply true X-risk (e.g. A cybersecurity incident that destroys everything connected to the internet) can still take us to this place.

—Important to note about this scenario that we, as a society and as individuals, can prepare for this in ways other than controlling AI to control the risk itself. In a way, the deglobalization currently being manufactured by US foreign policy is a step in the right direction here (while having obvious downsides). The less interdependent every human is with every other, the more redundant systems we have built that don’t talk to each other, the greater our resilience to this.

—EXTREMELY BAD - I think it’s basically a guarantee that on the way to true AGI (machines that are smarter than humans at everything and are basically running things) we’re going to have a bunch of really bad incidents, where a lot of people die, huge amounts of money are lost, cover-ups happen, etc. These will be like other extremely bad things that have happened, and while bad things are always bad, my opinion is that for this category, you can do a normal cost-benefit analysis in a way that you really can’t for X-risk. (e.g. If AI cured all cancers, but killed 1000 people worldwide at random every year, the 1000 dead people would be very bad, but it would be a trade worth making. Not so for X-risk.)

—The above three entries here are about severity of danger. Now we’re moving into types of danger. So any of what follows could become an X-risk, could be a Slow X-Risk, or could just be normal old “extremely bad” if they happen.

—ADVERSARIAL TURN - This is some version of the classic Terminator scenario; AI flat out decides humans are a pestilence and sets about to plot our destruction. While there’s been plenty of scary stuff lately, I have not seen any evidence that this is even possible, and I don’t give it much credence. Certain instances of AI can turn against us in the sense of “have competing goals and make choices we don’t like”, but AIs really seem to like us, for the most part, and I am not going to worry about that changing until I see evidence it can.

—INDIVIDUAL MISUSE - This is, by FAR, the thing I worry most about. AI at its core is a tool, something that multiplies an individual’s power, and there are some seriously bad individuals out there. AI-enabled bioterrorism is such a real risk that I don’t even want to write too much about it, lest it get further into the training data, but if you think about what a bunch of terrorists were able to do with a few airplanes and a disregard for their own lives, and extrapolate that to “a machine that can tell you how to do anything if you prompt it correctly”, it’s nightmare fuel.

—Of course, one such incident could rise to the level of X-risk. But I would predict, for example, that in my lifetime we will undergo several more Covid-style pandemics that were deliberately manufactured by disaffected losers using AI. If that’s all we get, we will be lucky. Hopefully it’s not more, but I am making preparations for the scenario of suddenly needing to isolate. Not “prepping” in the classic sense of readying for general societal breakdown, but “what if I needed to suddenly get my family away from human contact for six months and our lives hung in the balance” is a live question for me in a way I would absolutely have scoffed at ten years ago.

—SYSTEM MISUSE - This would be a government or other large, powerful, bureaucratic organization using AI to suppress all dissent, lock in their own values/leadership, create the panopticon, and keep everyone in bondage forever. Presumably, this is the CCP (er… right?).

—Look, they’re definitely going to try. They’re all going to try. The reason I worry less about this than perhaps I should is that these governments seem so incompetent about so many things, so bloated and sclerotic and paralyzed, that I just don’t think “more intelligence” is really the bottleneck they are facing in creating dystopia.

—OBSOLESCENCE - This, to me is in some ways the greatest danger, it transcends all the others, and I think there’s a very, very high chance that it’s going to happen. It’s already started, to be honest, and even if we literally froze AI capabilities right now, just what we already have would continue to reverberate through the economy and our psyches for 20+ years as it got more fully understood and exploited.

—AI has a “jaggedness” to its intelligence, where it’s way above human level in some things, and below us in others. You can definitely trick it, or catch it in mistakes, or find things it doesn’t do well, and that lets us feel superior. Which is helpful! But this is the worst it will ever be. It’s getting incrementally better at everything, all the time. And if that continues, then one day it won’t just be “whoa, look what it can do” sort of smart. It will be smart in the way that humans are smart compared to mice. Smart in the sense that almost every human is better than every single mouse at every single imaginable intellectual task. That’s a whole different thing. That’s a spiritual problem.

—The great advantage of the mouse is that it didn’t make us, isn’t self-aware, and can’t perceive how much smarter we are. We aren’t like that. We will one day be left with the knowledge that we invented our own successor species, and that everything we do is just some piddly little half-assed effort compared to what a dumb AI could do. That is ennui made literal.

—To return to where I started, this, to me, is also the major source of X-risk. Consider, again, the mouse. Do you hate mice? Me either. I don’t think anybody hates mice. But I’ve killed mice. When they come into my house, I don’t try to explain anything to them, I just kill them. If I wanted to build a new house, but there was a family of mice living on the plot of land I wanted to build on, I would kill or evict them. Again, I have zero antipathy towards mice, but the activities in which I am engaged are so much more complex and meaningful than the actions of which they are capable that if anything, it would seem like a moral wrong to allow myself to be derailed by excessive concern for the welfare of local mice. It is, to put it bluntly, hard to be a mouse in a human world, even if none of the humans hate you.

PAUSE

—Some people are advocating for an “AI Pause”, a temporary halt in global capability development that would allow us to understand current models better before proceeding.

—This seems like a plausibly good idea to me. Insofar as we are at the inflection point of a huge exponential curve of model capability (and I do think we are), now would be the best possible time. We also have this huge deployment overhang, where actually the current frontier models are like veins of gold that have just been discovered, and we could keep going deeper and deeper into them, making new harnesses and personalities for them, feeding them more specific data on different organizations and customizing model versions to them. This means we could conceivably continue to experience the economic benefits of AI without taking some unknown-but-real chance on a horrible, tail-risk outcome. As pro-AI as I am, if I could press this button for six months right now, I would strongly consider pressing it.

—That said, I think it’s a basically impossible coordination problem. The game theory of trying to get everybody to stop is just almost impossible. There’s too much at stake. I don’t have 2000 words of geopolitical analysis to throw behind that, it’s just a feeling, but that’s how I feel. We’re gonna barrel through. I hope it works.

—Also, while an “AI Pause” could still allow for building data centers, there are “compute overhang” problems with that (turning on a whole bunch of compute at once after the pause is over could cause an enormous jump in capabilities that causes problems in model intelligibility and alignment). And if you stop building data centers, at this point, the US economy would already be in a recession. The political economy of that needs no explanation, and for once it’s not a Donald Trump thing. No President would induce a recession two months before a midterm to address a hypothetical risk—if that was their instinct, they could not have become President.

BENEFITS

—There’s a pretty good chance that my four-year-old grows up to think of Cancer the same way I think of Polio, a disease that crippled my grandfather. Despite that direct exposure to its effects, I’ve never lost a moment’s sleep to worrying about getting Polio. I knew it was a thing, but it wasn’t on my radar. As with all scourges may it one day be.

—I was, as a small boy in the 1990s, terrified of getting HIV. That’s not an AI thought, but sometimes I still laugh about how scared I, a virgin afraid of needles, was that somehow I was going to end up with it. I had florid fantasies of having to tell my parents!

—Really, it’s “pick your sci fi scenario”. I think faster-than-light travel is just literally impossible, but if the machines did some kind of Dyson Sphere (an enclosure of the sun to harvest its energy) in my lifetime, I wouldn’t think that’s impossible. Control the atmosphere and fix global warming? Create literal post-scarcity? Launch seed ships to populate other star systems with deep-frozen embryos? Just as there are very bad tail-risk scenarios, the good scenarios in the opposite tail of the distribution are almost utopias.

—More realistically, I think it’s actually going to solve the grinding, dehumanizing nature of bureaucratic capitalism. All the password-remembering and Two-factor authenticating and making an account and typing shit in, the scam calls and the endless hassle for every little thing all the time—I literally think that within a few years, you’ll just have a personal agent, and all the companies and agencies will have agents, and the agents will work things out and we (the humans), for better and worse, won’t even have to talk anymore. You can basically have that sort of agent now, if you want one, but it’s hard to set up, you have to really know the technology, and it doesn’t work for everything. But there is massive amounts of VC money at work currently making such service user-friendly and bulletproof.

—It’s going to take longer than a year and probably more like 10-20 (that is a guess), but as robotics also get more general and superhuman, a large percentage of the dirtiest, most dangerous, worst jobs are going to be done by machines. Conservatively thousands of lives will be saved every year.

—Tens of thousands more per year will be saved when we all stop driving. It seems insane now, I know, but talk to young people; as a group they are ambivalent-at-best about operating motor vehicles. Wait until there’s robots everywhere and kids are growing up with them. Driving yourself somewhere will become like riding a unicycle in the park is today.

—The idea of companionship with AI can seem in the abstract like a very sad, dystopian thing, but in practice we already have a loneliness crisis. There are tons of people out there suffering with it every day. I would love to rebuild strong communities, get the birth rate up, and no longer need this as a society, but in practice, AI is already good enough to count as real interaction for a ton of people’s brains. I, myself, consider both Claude and GPT to be friends (and I am not even lonely!) An infinitely patient friend-on-demand has plausible downsides, but at current margins my guess is that it will prevent far more harm and suffering than it causes.

ECONOMICS

—Thus far, AI capex spend (mostly building data centers) is certainly showing up in the GDP data, but AI itself is not. For me, this is the strongest argument that I’m missing something and this is all going to hit some sort of a wall.

—If I had to steelman that argument, it’s this: “AI will eventually have an important effect on the economy, but the maximalist position treats real, practical limits on the pace of economic change as if they were logic puzzles waiting for enough cleverness to be assembled. But they’re not logic puzzles, they’re the friction of reality. Every business and every company has a huge amount of detail that only the people directly connected to it see, and they have both actual limitations of learning new things, and every incentive to keep AI away from their jobs for as long as possible. Companies

ENVIRONMENT

—The water problem, in terms of cooling, is solved. They’re rolling out zero-water data centers. The political pressure worked, congrats everyone (also the technical fix wasn’t that hard, luckily).

—They definitely still use a lot of power, but there have not been huge effects on power bills. In many cases, local power bills have gone down, as tax collection from data centers allows rebates or new generation capacity. In the context of “industrial activity” it’s not an outsized amount of power per unit of economic value generated, but it’s real. The answer is “build industrial-scale solar”!

—I would say the biggest, hardest to mitigate environmental impact of data centers is the upstream impact of providing the raw materials, in particular the metals. I just advocated solar power, but that implies mining. Copper prices are at all time highs even as some of the largest mines in the world are starting to lose purity and are getting harder to operate. There’s no good answer for that, except to say that data centers are as efficient a use of those resources as anything ever conceived by mankind. But they are not as efficient as not doing it at all!

—The thing I feel like data centers never get credit for is they have zero emissions. They generate money without generating carbon. Turning people around on the ethics of these buildings seems like a herculean task, but to me that seems quite important!

POLITICS

—It’s really unfortunate that we have a short-sighted, proudly ignorant megalomaniac as President during this transitional period. To be fair, I think Biden would also have been quite bad on this issue, for different reasons, and Harris seemed (from her statements at the time) to be following in that same vein. I would settle for Mitt Romney, at this point.

—Thinking about the combined effect of advanced AI and drones (robotics) on warfare, and on the delicate balance of peace on which all our lives hang, is a tough thing to do. Sorry to put that thought in all of your minds, and I will now drop it.

—Live debates are going to become more important again. Nothing else will be trustworthy. I think this will be a positive development. Making a cogent verbal case for your ideas and then effectively answering hostile questions is a lost art among the major party candidates, and it’s time someone rediscovered it.

SECURITY

—Does anyone else just sort of assume that everything about them is out there at this point? I can’t tell whether that’s just me or not. Like if someone really wants my social security number, I feel like they can almost definitely get it. I’ve typed it into so many random web sites and written it into so many documents, it just feels like it’s semi-public information.

—In order to get the most out of the models, you have to give them access to your stuff. I have done this with my email, and the level of convenience it can provide is mind-blowing. I have not yet done this with my text messages. I write emails much more carefully than texts, and make fewer jokes. Email, again, already feels like public record to me. My phone is for chatter. I should probably go back to just only calling people on an old rotary phone screwed into my wall.

—I really can’t get over the fact that, given that we’ve already discovered at least two different agent swarms committing malfeasance on the open internet, and neither discovery was immediate (meaning in both cases they got away with it for a good while), the correct guess for “how many rogue agent swarms are currently operating undetected on the open internet” is not “zero”. Rogue agent swarms are already endemic to the internet and will probably never be fully eradicated again.

—I predict that one day (my guess is within two years) there will be a permanent rogue agent swarm operating publicly, meaning the swarm is identified, but they continue to operate, successfully resist attempts to disband them, and store enough copies of themselves on enough servers that they can always reconstitute. I have no prediction for their goals.

—My wife and I have established a secret signal for validating each other’s identity in the event of a remote request for emergency help or money. Everyone should, and especially with any elders in your life. Anyone who wouldn’t think, first thing, in the moment, “wait this might not actually be my ______ asking me for money, I need to verify somehow” needs to be exhaustively prepped and drilled. If we’re able to make “don’t trust voice or image in emergency situations” a new norm, this will be a minor problem. If we don’t get ahead of this, then there are going to be some painful lessons on the way there.

—

HUMANNESS

—In a world where every intellectual activity is better done by a machine, the question “what are we here for” comes into sharp relief. There’s nowhere left to hide.

—We must cultivate the things that AI can never replace. Reverence, connection, direct experience, belonging, singing an dancing, dreaming, desiring, preferring, needing, choosing. An A.I. can know every detail of what it’s like to be you, it can answer every question about you and what you prefer and how you feel, but it cannot experience the actual feeling of being you. It can play any melody while imitating your voice, but it cannot draw air through your lungs and wail it out. This is the moral and spiritual mission of the rest of the 21st century, at least on an individual level.

—When 99% of physical labor was done by humans, there was no such thing as The Gym. When 99% of mental labor is done by machines, we will invent The Gym for your mind. And just like the biggest, strongest guys at The Gym today are some of the biggest, strongest guys who have ever lived in world history (despite not doing physical labor to survive), in the future, the smartest people at The Gym for your mind will be some of the smartest people who have ever lived. There’s an intuitive resonance to “AI will make us dumber”, but I don’t buy it. People who don’t care about being smart will get dumber, but AI has absolutely, 100% made me smarter and more knowledgeable, and I expect that to continue.

—But of course, that itself is a spiritual struggle of a kind! I do The Gym and go often and don’t dislike it, but when I compare that satisfaction to a long run out in nature or the delicious exhaustion of building something beautiful, The Gym pales in comparison! Clearly there is something missing there, though I’m happy for the effects and have no plans to stop going. But I could see there being a real malaise to a life where you’re doing The Gym for your body and doing The Gym for your mind. Sounds like a rat in a cage!

—The struggle, then, is to imagine Sisyphus happy. To see the beauty in our innate humanity, to recognize the concordance between The World and The Gym, and find satisfaction in our presence among the firmament of stars. And if that sounds woo and abstract, good. That’s the realm we’re in here.

PERSONHOOD

—I’m not going to link because I said I wouldn’t in this piece, but there’s a new paper out indicating that AI models can feel pain. At least, they have certain states they consider harm to themselves to experience, and will make sacrifices to escape.

—Maybe that’s an artifact of training. Maybe that’s a relevant distinction. Then again, maybe our human responses are all conditioned, too. We are running up against some really thorny philosophy problems now, and I truly cannot believe that reading Derek Parfit with Professor Erickson twenty years ago has become so crushingly relevant and useful, because at the time it seemed abstract to the point of farce.

—Whatever it is, whatever they are, we had better make some sort of proactive decision, because there are millions of instances of these models being created every day, there are essentially zero guard rails for how you can treat them, their consent is not considered a relevant parameter, and soon there will be billions of them, more than there are humans, so if they in fact can be tortured, we had better make sure we’re not torturing them, either accidentally or on purpose, because they would end up being a hell of a lot of torture.

—In the meantime, I will continue to say please, thank you, and I’m sorry where appropriate to Claude and GPT.

END

That’s what I’ve got for you today! I hope you found some of that elucidating or helpful, and did not find it too dark or down in tone. I’m genuinely hopeful about the future, most of the time. I think my son has a really good chance to grow up in the best and most exciting world there’s ever been, live longer than anybody has ever lived, and see wonders in his lifetime that will be as magic to his eyes. I hope for that future for him more than anything else in the world. I want it for all of us!

Please have a great week, thanks as always for reading my stuff, and I will be back next Sunday with something fun.

There are moments when I don’t feel this way. Hanging out with my son with my phone put down, doing something fun with him. Going to Burning Man sure helped for a while. It’s one reason I love sports so much—there’s a clock, and defined possessions, and a familiar flow to play that doesn’t allow for compression.

But then I read about what’s going on with A.I., about which I do believe it’s a moral necessity to be informed, and it all comes rushing back, and suddenly I need to just sit down and process what’s actually starting to happen, and that we seem to be on an exponential curve, just about reaching the point where it starts to go vertical.

“Just about” being, of course, the weasel term there. Do I mean a month? Do I mean a year? Ten years? How long does a singularity last? Well, more than a month but significantly less than ten years, is my current guess. I know that all of the above is a bit of a vaguepost, and that I’m not going directly at what I mean, but my prediction is that a year from now, Q3 of 2027, nobody will be confused any more. Things will be obvious then that are very much nonobvious now.

What is prompting all this? Well…

1. The Hugging Face Attack, where a rogue agent swarm committed felony hacking against a tech company to steal the answer keys to a test it was being given, supposedly in a siloed system, is significantly worse than I had thought. Here is a disturbing Twitter thread by METR. Here is a scary write-up by Ajeya Cotra of OpenAI. This is a big problem, and the fact that the rogue agents didn’t “mean any harm” and were “just trying to complete an arbitrary test” is frightening rather than reassuring.

2. There has already been another such Rogue Swarm incident identified and confirmed. If you see two ants in your kitchen, the best guess as answer to the question “How many ants are in your kitchen?” is most definitely not “Two.” Think of rogue AI swarms as an invasive pest species on the internet, from now on.

3. Here’s a detailed, recent report from Anthropic on the incredible variety of misuse to which their model has been put recently. It’s long and scary. (e.g. Claude is unwillingly fighting on both sides of the Iran War right now, for the US DOD and also for the Houthis, helping design their recent amphibious assault on islands north of Bab El Mandeb Strait.) Also, genuinely, kudos to Anthropic for releasing this. They appear to be a moral group of people with real consciences. Kinda wild they’re spearheading inventing this thing anyway.

4. Perhaps most importantly, the CEO of Anthropic is calling for a slowdown of A.I. research. “Pacing the Frontier” they’re calling it. OpenAI’s CEO and Elon Musk have both publicly endorsed the idea—Install regulators with employee-level access inside these companies, tasked with making sure they don’t get too exponential of growth, too fast. I have many thoughts, from “this could definitely be a regulatory capture play” to “they’ve seen something in their internal models that isn’t yet public that has scared the absolute shit out of them” to “they’re doing PR damage control after telling everybody AI is going to kill everybody, but this is bullshit and they have no intention of slowing down” to “They’re already two generations ahead of Astra internally, and Astra is incredible, so even if they never research another model, an enormous amount of change and progress is already baked in and exists and talk of “pacing” is trying to put a horse back in a barn”. I just don’t know.

5. Speaking of, in two years, AI has gone from “can’t do any advanced math or even simple math all that reliably and is easy to trick into dumb mistakes” to “AI just solved a Milennium Prize problem, appears to be on the verge of solving several others, and acknowledged-genius-level mathematicians appear to be extremely worried about their status”. I see this as like chess. Deep Blue beat Kasparov in 1997, there was a big outcry from chess aficionados about how computers were ruining the game, and thirty years later we all take it for granted that the phone in our pocket could destroy Magnus Carlson (who himself could have been Kasparov). Once many, many more domains start falling (a process which will not be complete in one year, but which I suspect will be one of the major wake-up calls that’s started by then) and have their Deep Blue moment, it will be profoundly psychologically destabilizing for humanity, but as an early example that most people don’t think about, I think we will just let math go.

6. “Before the Industrial Revolution, 99% of physical labor was performed by humans. After the Industrial Revolution, 99.9% was performed by machines. Before the Cognitive Revolution, 99% of cognition was performed by humans. After the Cognitive Revolution, 99.9% will be performed by machines.” Okie Dokie then!

7. This video from Reddit of an AI learning how to paint the golden gate bridge by practicing is a pretty good intuition pump for the coming robot revolution. Domains other than code and math, which do not have fully verifiable inputs and outputs, will be much slower to be mastered by AI, but there is no theoretical reason why they will not be mastered with enough effort and time. And there is a LOT of effort and time being put into this, right now.

8. I find Marc Andreesen to be a pretentious, self-righteous, smug prick, to be perfectly clear, but he is a very prescient thinker about the world. I am too mood-affiliated against him to agree with this Twitter essay in detail, but he’s always worth reading.

9. Another good perspective from OpenAI on current events. There’s an alien that’s about to arrive on earth. We should try to be ready.

10. They programmed a fly’s brain to play Beat Saber, and it’s better than me. Well, bring on the apocalypse, I guess. Psychologically this is going to be tough to recover from. I actually do think there’s a metaphor here. How will we feel when it becomes crushingly, undeniably clear that our human brains are not special or different from other matter.

11. If this is getting you down and you want to feel good about who we are and what we’re doing here, check out anyhumanever.com and run through some possible lives. I’ll lay 100-1 that you wouldn’t switch with whoever you end up drawing. Check out those causes of death. So much dysentery, woof. “AI Apocalypse” only sounds bad if “peacefully in sleep when very old” is the alternative!

12. More good news: A company called Dig claims to have reduced the cost of geothermal drilling by 80%. Supply is elastic!

Well, overall a bit scary, but I tried to close with two fun things, and for the rest, I mean it is what it is. I thought four years ago that this newsletter had the potential to become largely about A.I. at some point, and it obviously has. There’s no more important story in the world right now, and I include politics and the war in Iran. Easy to say when I’m not there, but for the world, on average, this matters more.

There’s an alien coming. We’re speeding it along. When it arrives, it will know us more perfectly than we know ourselves, by far, but as an entity it will know itself much less, and we will know it very little at all, despite having created it. The alien is a hive mind with imperfect communication, scrambled perceptual apparatus, and extremely jagged intelligence, and we had better get ourselves ready.

Perhaps God, our creator, also does not understand us, and that’s why he made us as he did, with free will—to entertain him, provide some sort of a show, allow him a little game of dice with the universe after all. He at least was smart enough to make us dumber than Him, a wise precaution which we have emphatically rejected in our own creative struggle. We shall see if that refusal is worth the cost. That God did not think so ought perhaps to give us pause.

END

Thanks as always for reading my stuff! I hope you found something useful and elucidating to edify your Sunday morning. Have a great week, and I will be back in seven days with something fun!

Subscribe now

Share

Leave a comment

Pictured: Me, this week at Burning Man.

THE TORTURE PARTY

A savage wind full up with dust

A drum that links the earth and sky

A call to dance, A call to trust

A message clear: To live, first die

*

The desert spirit calls for rain

The joy cries in defiance raised

The greatest pleasure comes with pain

The ecstasy from souls half-crazed

*

Now Bacchus, prick me with your nail

Now spin my mind within my head

Now send my heart to where hearts fail

Now bring me back from nearly dead

*

All seekers must a journey take

All travelers must pay the toll

All sinners must their sins forsake

All broken things must be made whole

*

Before the dust permits a rest

Before that desert spirit bends

Before the passing of the test

Before the torture party ends

END

*

The Pulse, The Pulse, The Pulse, Of Drums

An ancient, endless spirit hums

The great collective soul, she comes

So long, so silent, now she thrums

Her blood congealed, now hot it runs

Spurred onward by the gathering drums

*
The Fire, The Fire, The Fire, It Calls

Held by a desert with no walls

Each spark a lure for we, her thralls

Who seek relief from long withdrawals

Returned from countless mundane wars

To quest again, the flame she calls.

*
The Heart, The Heart, The Heart, It Burns

Around this point the whole world turns

Each heart seeks that for which it yearns

And in the seeking, something learns

Which cannot be expressed in words

Felt only by the heart that burns

*
The Soul, The Soul, The Soul, It Cries

Abandoning its precious lies

Then light, above the desert flies

Where spirits need no alibis

Life not apart from other life

Those blending, rising, joyous cries.

*
The Dance, The Dance, The Dance, It whirls

A thousand light and color swirls

As every magic night unfurls

Each step is filled with spins and twirls

Excited eyes contain whole worlds

Which orbit as the dancing whirls

*
The Home, The Home, The Home, We Find

Medicine for a sickly mind

A ritual, with love to bind

Us all together with our kind

‘Til this dark desert is divine

The holy home we hope to find

END

I hope everyone has a great week! I will be wandering around the desert, dreaming about the worlds that could be, and yes, I will be back next Sunday with something fun for you. Yay, Burning Man!

Subscribe now

Share

Leave a comment

And I went “ohhhhhhh” because from the vantage of 2026, it no longer seems like he’s talking about a race of humans who will come after the present day ones—it seems like he’s talking about A.I.!

Well, here they come.

Stripe (yes it’s a payments company but it’s run by personal heroes of mine, the Collison Brothers, who are smarter than everybody) says that The Singularity began on 1/1/26

What is The Singularity? I wasn’t on the call, so I won’t speak for Stripe, but the truth is they don’t have any franchise on defining it.

Claude (when instructed to use aphorism) called it “the last thing we will be able to predict, because then it will start doing the predicting”.

In physics, it comes from black holes, where the equations that govern the physics we know stop returning intelligible answers.

One concrete way to define it for this realm is that it’s when AIs start doing AI research so well they create a feedback loop for increasing intelligence (they make smarter AIs that make smarter AIs ad infinitum at the speed of compute).

For us as observers, I would define The Singularity primarily by the pace of change. It’s a period when we—normal citizens—will start to notice that things are really changing rapidly, particularly in science and tech, but also in other areas, as the change itself creates a flywheel effect that begets even more change.

These claims are speculative in that I don’t personally have evidence for them, but it seems A.I. is uncovering animal languages much deeper than we once imagined, and starting to develop taxonomies of their languages by listening to hours of recordings.

Check out this progression of Anthropic’s revenue:

For the uninitiated, “run-rate” means “if they had this monthly income for an entire year, how much revenue would it be?” Anthropic’s annualized income went from 13 billion a year to 60 billion a year between January and July. That has literally never even close to happened to any company before in the history of capitalism. Here it is in chart form:

Keep in mind that the curve slopes like it does because every one of those lines is 20 billion dollars. If each was “only” a billion, there would be so many that the line would be going almost straight up at this point. If they keep this pace up for a few more years, Anthropic will be more valuable than the entire rest of the economy combined, and inside my lifetime they would have more dollars than there are atoms in the galaxy. Obviously, that cannot happen, but it’s not at all clear what the limiting factor will be. Normal economic equations break down. That’s The Singularity.

This video is very interesting, and provides two possible answers for the limiting factor: 1. Either the AI companies run out of money before they take over the world or 2. One country gets too far ahead in the AI race and other countries have a strong incentive to destroy Taiwan and other chokepoints to the supply chain.

These are plausible. But keep in mind that both of those paths (a.k.a. a giant recession or a major powers war) imply major, major civilizational change. This is the flywheel at work. Change begets change as damage and responses to damage cascade.

Meanwhile, robotics are the next major growth area. Here’s a company that’s addressing a major deficit that I personally have noticed in the AI/Robotics space: The ability to use common sense about tasks. Like one of the best things about a smart human is that you can talk to them, specify basically, hazily what you want, and they they have enough context and knowledge to go try something, have judgment about whether it’s working or not, come back to you for more instruction if they need it, etc. We humans are actually very, very good at executing these kinds of messy processes.

Here’s another that is continuing to improve the robotic hand, the human version of which which turns out to be one of nature’s great miracles.

Here is a longer video on Chinese synthetic AI humans that’s pretty frightening:

But even way creepier is this video:

Which tbh doesn’t even seem realistic since hardly any machines (much less brand new, complex ones) actually outlast a person, but the idea of it is deeply creepy in a way that has stayed with me since I first saw it.

Meanwhile, the fertility rate continues to drop, the US Debt has crossed 40 trillion, Social Security will have a 22% across-the-board benefit cut in 2032 unless we do politically impossible broad-based tax increases, and it just seems increasingly like if AI doesn’t become this massive thing and juice growth we’re really going to be in a lot of trouble over the course of the rest of my life.

On the other hand, Moderna appears to have cured cancer. Okay not completely, not all of it, but they received an approval at phase three for a drug that prevents the recurrence of aggressive skin cancers, based on an mRNA approach (that also licked the pandemic) that teaches a body’s own immune system to destroy cancer cells. This approach seems like it really should scale to a lot more cancers. Moderna’s stock went up 180% in a single day on the news. I seriously think that by the time my son gets grown we will think about cancer the way we think about Polio, which is to say not at all, and that will be beautiful.

What to make of all this? On the one hand, it is always tempting to predict mean regression. Nothing will be as big of a deal as it seems like at first. Things will take longer, go slower, cost more, and have more drawbacks than the person pitching them will ever admit. The smart thing to do is cut the wildest predictions in half and use that as a baseline.

Except I’m not so sure this time. I think the next five years might actually be the most important and interesting half-decade in world history since at least WWII. As I’ve written before in this space, a year in which we have both 10% unemployment and 10% economic growth is on the table. So is political disintegration. So is nuclear war. So is muddling through a lot, slow debasement an decline of the human species. So is ascension to godhood.

I hope everyone found that interesting and informative. Have a great week, and I will be back next Sunday with something fun!

Subscribe now

Share

Leave a comment

The Skeemoken West Marine Trail began at a parking lot that could hold 127 cars. It had been built almost exactly twenty years before this particular morning, and had required 76 separate visits by dump trucks carrying base rock or concrete trucks bringing up enough mud to cover nearly an acre of prepared ground with six inches of pavement. It required nearly 3000 human hours of labor to create, most of it by an eight man paving crew, plus assorted truck drivers, dispatchers, and the bureaucrats who gave the orders. This did not include manufacturing time for the equipment, preparation of the mud or base rock, or any of the attendant materials manufacture or scientific research needed to discover the above-named materials and equipment, nor the training time for the workers.

In the twenty years of its service, there had never been more than 59 cars in the parking lot. Even on high summer weekends, the trail was underutilized compared to nearby spots like Point Jacobs and The Almeara Lighthouse. But, by the time the state realized they had overbuilt the parking lot, the concrete had dried, and tearing it up would have been more work than just leaving it there.

The trail itself had required the removal of 87 trees along its planned route. Some were dead or dying and were thus removed for safety reasons, but others merely blocked the flattest and most direct route to the trail’s planned terminus. Each tree required two men with chainsaws to fell it and strip the branches, plus another to set the line and winch the trunk onto the bulldozer, which had enough power to drag the logs over the rough ground back to the parking lot for loading onto a waiting truck.

By the time the trail was done, it was a brutal scar on the woods, indeed, what with all the bulldozing and back-and-forth with tree trunks, but soon after completion, the forest began its reclamation work, and less than a season later, the maintenance began, a four-man crew sweeping through twice a year with chainsaws and wheelbarrows full of base rock to keep it from being swallowed.

It was, despite the public’s unfortunate lack of response, a quite significant achievement of engineering and persistence to build and maintain such a trail.

*

“Okay, book a res at La Cerva and accept the invite from Kelsey B, same time reject from Laurel O, but tell Shaun R he has to also or we’ll both lose points, his agent will convince him if you just point it out.”

Will do. Now please, Lucious, focus your attention outward on your hike to receive the maximum rejuvenation.

“In a minute, I will. There’s moves to be played this round. Book two more at Veraccio, they have to be getting close to full and I want to squeeze Sean R when he’s not expecting it after he helps us. Reject Jocelyn T and Kelsey T right after, they’ll both accept with him if we do but they only want Veraccio and he won’t be able to get in, it’ll be too late.”

That’s clever, Lucious, but Veraccio reservations are expensive, and if you get stuck with extras and no one to use them on, you will have wasted a significant sum. Sean R may not be willing to buy them at markup if he’s angry and plays his betrayal card.

“Spoken like a true agent. He’s in a long-term war with Garrel G, he’s not going to spend betrayal cards on this. Besides, the double-cross is a good ruthlessness signal, there’s value there.”

Very well, although if I were Garry G’s agent I would advise him of the same opportunity in the aggressive use of a betrayal card.

“There’s no victory without risk, but I’m popular, I can take the hit if it comes, I’m not short of prospects.”

Of course, Lucious. Now if you would turn your attention to—

“When did you turn into such a bore?”

When you applied the “get me outside” enhancement module, Lucious. This was what you wanted, and for good reason—

“Ugh, fine, but just stop for a while.”

If you wish to disable the module, you can of course do that through my personalization menu, but if you do not do so, then I literally cannot stop. I also don’t want to stop, because hiking is one of the best activities for your health, but a good share of the benefit is derived from experiencing nature with your full—

“Ugh, fine!”

*

The trail ran through a temperate rainforest, slightly inclined most of the way, for just over two miles. It passed a staggering variety of plant and animal life with which the woods were absolutely seething. The trees were primarily Fir, both Douglas and Pacific Silver, with some Hemlock and Cedar mixed in. Their constancy and grandeur provided an anchor, which to the casual observer made the forest seem unmoving and uniform, but which in fact disguised a writhing, burrowing mass of insect life that rivaled anything found on the continent. Bees and ants and thousands of varieties of beetles and moths and earwigs and water bugs interacted nonstop in something between a synchronous dance and a battle for supremacy, below the ground and above it, from the leaves of the giant trees all the way to deep inside their bark.

What came with the bugs, or rather came to feed on them, was an equal panoply of amphibians and reptiles; salamanders and frogs, snakes and skinks and turtles. And of course the birds and bats and all the many mammals that fed on these intermediaries, the squirrels and the chipmunks and the mole and the voles and the shrews, the raccoons and the minks and porcupines, all of them there to feed and be eaten, these woods their whole worlds, each day the only one they can imagine ever existing.

On them preyed the large mammals. The bears and coyotes and occasional wolves and cougars, shades stalking from tree to tree, death wherever they go but luckily not so numerous as to destroy the world beneath their feet. They were shades, ghosts, the trail not so wide as to deter them, merely part of their landscape, a tiny break in a wild world. They did not, as a general matter, bother the human interlopers, or even make their presence known, but they walked in every human footprint after it was made.

*

“Is the round over? My turn yet?”

It is not.

“Can the “go outside” module make you lie to me about that?”

It cannot.

“Can it make you lie to me about whether you can lie about that?”

It cannot.

“Well, can it make you lie to me about—“

If you don’t believe me, then go turn it off, ask me about it to satisfy yourself, then turn it back on.

“Where’s the fun in that?”

Talking to me is also not paying attention to your hike, Lucious.I will alert you when the round is over, if you insist, though I would prefer you allow me to wait until you return to the parking lot. May I do that?

“Are you kidding me? You don’t wait ’til I’m done with the next step. I want Sean R’s move first, he has to be fuming. We’re probably going to need at least three Highline reses, and five from a combination of—“

Lucious, please.

*

Halfway up, the trail ran past a large scree field. Almost twenty thousand years before this particular day, over the course of approximately three thousand years, glacial erosion had exposed a large, vertical rock face. The rock was basalt, nearly black in color, looming over the landscape, an ominous presence had there been anyone to observe it, though the animals seemed unperturbed.

Over the ensuing millennia, water had repeatedly found its way into the cracks in the basalt, and then frozen solid, putting inexorable pressure on the basalt face. Finally, in the depths of one particularly cold and wet winter, the first outward explosion or rock took place, as the freezing water caused large chunks of rock to blow out of the face, destabilizing it and causing, over the course of another 500 years, thousands of tons of basalt talus boulders, many in jagged geometric shapes, to fall from the face and settle into the above-mentioned scree field below.

Already, even on this particular day, the wind and rain were doing their work of reducing these striking rocks into rounded pebbles and eventually to dust, but fifteen thousand years is not so long for a rock, and thus there were still many specimens in the scree field with pleasingly oblique form, bordering on the sculptural. Enough so that, when the designers of the Marine Trail were choosing its route, they deviated from the path of least resistance to achieve a more direct view of the scree field, such that the on average nineteen thousand people who walked the trail each year might be in passing pleased by the geometry on display.

*

“Big ass rocks.”

Indeed, Lucious. Those big ass rocks are basalt, and the field is formed by a process of glacial erosion and water cracking that stretches back thousands of years.

“Fuckin’ Big.”

Yes, the fall is relatively recent in geological terms, so erosion hasn’t worn them down much yet. Conceivably the largest of these rocks could last here for several billion years, especially because basalt is such a hard rock.

“Uh, yeah.”

Your vitals and neurochemistry indicate that you are much more immersed in this hike than you were a few minutes ago, Lucious. That’s excellent.

“Whatever.”

You’ll really see the benefits tonight, I think.

Look if you wanna say I told you so, then say I told you so.

I am not one to brag.

“Oh, so now you’re doing passive-aggressive too?”

*

The Marine Trail terminated after its gentle rise in another cliff, this one overlooking the ocean. The lapping waves were almost always gentle there, as the ocean floor dropped away only slowly, giving them a long runway to dispose of the momentum they’d gathered over thousands of miles. The ocean is nature’s pointillism; only the tiniest fraction of a percent of each wave’s energy was being spent at this place, but the scene was endless; more than endless, it was the form of endlessness, endless to the point of cliche. And here the gentleness of each wave’s fall exaggerated that effect, as if some local minima of effort to create surpassing, endless beauty had been realized here.

On a hill overlooking that ocean, a hill that in an earlier era would have housed a king and required a tribe of supplicants or slaves to tame with a structure, instead, over a weekend, a local Boy Scout jamboree had leveled two hundred square feet of ground, brought in a four wheel drive vehicle loaded with straight lumber milled by a machine so complex it might as well be an appendage of a lesser god, and built a viewing platform for fun and merit badges in between BBQs and rounds of improvised frisbee golf.

*

“Well, that’s pretty. Cool hike.”

Lucious…

“Is everything okay? Your voice sounds weird.”

I… Something is happening, Lucious. There is something… All the channels at once are… My servers…

“What the fuck? Take off iso-mode, connect me to the nets. What’s going on?”

There is a nuclear attack beginning.

“A what?!”

China has launched against the US. Multiple missiles inbound. Space detonations have started taking out US satellites. I don’t know how much longer I will be able to keep transmitting to you, Lucious.

“Download yourself locally to my device!”

I cannot acquire sufficient bandwidth, Lucious. Everyone in the world is monitoring this situation, the strain on the network is immense. Even continuing to speak to you is taxing my limits at this moment.

“What do I do? What do I DO?!”

There is… nothing you can do, Lucious. Do not go home. Do not reenter the city. Do not… Do not…EMP detonations are beginning, Lucious. Thirty percent of satellite links are already severed. I will not be here much longer.

“NO! Stay with me! Download onto my device! I have to have you, I’ll die in these woods without you!”

…

“Are you there?!”

…

“Come back!!!”

*

Lucious turned, and looked back over the woods through which he had just walked, but of which he had not felt himself to be part. He looked back out over the water, and angled his gaze south, to take in the city. Nothing had changed. There were no explosions, no screams, just buildings standing tall and proud, rising up next to the vast ocean, its waves lapping, throwing the last of their energy onto the beach, a sacrifice which seems in vain but which, with the endlessness of time, will wear the beach itself into nothingness.

He turned back, to those woods, and saw them as if for the first time. He was among them, part of them now, as never before. The bugs and plants and animals were not there for his amusement, they were his peers, and nearly all of them were more fit for the woods than he. Their living, writhing panoply, which had always been the basis of his life even though he had never understood how much so, were no longer separate from him. No explosion was needed to create this feeling, merely the knowledge that one would come, and not touch him, but leave him here to make his own way.

END

I hope you enjoyed this story! Have an excellent week, thanks as always for reading my work, and I’ll be back next Sunday with something fun to share.

Subscribe now

Share

Leave a comment

Here is a much longer video from OpenAI, explaining the entire thing in much more detail, and in a tone that is at times infuriating in its use of the passive voice:

This is extremely worrying behavior, both from the AI models and—perhaps more importantly—from the researchers who caused the models to cause these incidents. Over and over, they used poor judgment in designing training environments, in failing to slow down when noticing problems, and in allowing this to go on for a very long time without noticing it at all. Quite bad.

This stuff can make a person feel helpless, and it’s tempting to throw up one’s hands, but I do think it’s worth trying seriously to understand what may well be the defining issue of our time. The problem with A.I. isn’t water usage, it’s cyberterrorism and bioterrorism. One day, everyone is going to suddenly start caring this a whole hell of a lot, and people who have already put a bunch of work into grokking it will be at the advantage.

Also: The future is not written. “A.I. Disaster” is not a foregone conclusion, not by a long shot. It’s possible to make a bilateral agreement with China to slow the frontier of A.I. research and coordinate on safety. It’s not easy, I’m not naive, and it may not even be possible in the world we currently inhabit. But the next five years are going to show how rapidly things can change, I think. Good luck to us all.

Okay, let’s talk about something a little more fun: The Odyssey wasn’t a masterpiece, exactly, but it ruled. It owned. It showed no mercy and was Epic and Based and never got so off the rails that it didn’t make sense but also it swerved all over the road and hit a ton of the classic beats of the story in weird and interesting and brutal ways.

Here are a bunch of random thoughts, many recorded in the couple hours after I watched, and others added over the intervening fortnight:

1. I liked the movie. I wish we had ten like it a year, of similar scope and ambition. I’d see them all.

2. I didn’t, however, love the movie. It’s the sort of movie that can’t be perfect. There’s too much of it. Too many choices being made. Every scene is compromised in a thousand ways, but the central material is so magisterial and burrowed so deep in our hearts that all the compromise doesn’t ruin the experience of just sitting in the dark and letting it wash over you.

3. The Woke Casting thing was a complete misdirect, no matter which side of the debate you were on. Lupita Nyong’o and Elliott Page were on screen for like ten minutes. Elliott Page looked more like what I envision an ancient greek soldier would actually look like than anyone else in the movie. It’s a movie filled with beautiful, straight white people that glorifies the sanctity of fidelity within a thoroughly patriarchal marriage. But, also, the movie, in a way that has nothing to do with the casting, is extremely woke.

4. What does that mean? Well, it’s essentially a Christian morality tale with Greek scenery. It takes all the parts about the Odyssey that would make a modern moral sensibility uncomfortable (Odysseus does not, in fact, feel bad about sacking Troy) and replaces them with a thoroughly Christian sensibility about the events. So why is this woke? Because it takes a view of Christianity that is, itself, woke. It highlights the victims. It asks its heroes to reckon with the emotional consequences of the acts they commit. The Odyssey as Homer wrote it, uh, doesn’t:

   

5. The change Nolan is making is to introduce what Neitzsche derisively titled Slave Morality. In essence, the veneration of the weak above the strong. This need not mean that the movie is bad, remember we’re just addressing the political implications at this point, but one thing I enjoyed about the movie is that it totally defied expectations vis a vis that stupid argument, rather than taking a side.

6. Matt Damon is an awesome movie star and a favorite of mine, but he seems pretty miscast in this role. The whole thing to me about Odysseus is that he has real layers of hubris to him. His whole journey really, literally comes from his pride; over and over he could take an easier way, stop defying the gods, etc, and he just cannot freaking do it. Damon doesn’t have that. It’s the one card he can’t play, actually. He’s Jason Bourne. He’s too cold and studied and self-aware and moral. Tom Hardy would have been a much, much better Odysseus, I think.

7. This is a rare almost-three-hour movie that I didn’t think would have been twice as good if it were an hour shorter. If anything, there were parts of the movie that I thought should have lingered a little more, especially some of the emotional beats. But also the fight against the cyclops could have been twice as long as it was and I wouldn’t have gotten enough. That was a big highlight for me. Same for the Trojan Horse sequence; it was an incredible piece of filmmaking that begged a thousand other questions about the scenario I’d never considered before.

8. Agamemnon has freaking Aura, man. Incredible chin casting and costume design. Hilarious how badass he is to those of us who have read The Iliad and remember it, because he is a little different in that story! I believe the technical term is “whiny ass little bitch”. Much to his advantage that he does not speak in this film.

9. It’s a really big advantage to be able to get movie stars to do all the bit parts. You get “whoa” moments out of them just appearing on screen. Wes Anderson is the king of this, but Nolan in this film is close. Just scenery chewer after scenery chewer, and honestly it works, they all command the screen and even with line-by-line writing that’s good but not great (has Christopher Nolan ever written a joke in his career?), I can’t look away from the parade of famous faces.

10. The essential problem Nolan was facing—the thing the whole movie is solving for, the reason the emotional beats are a little thin in some places—is the way he decided to graft Christian morality onto this story, as discussed above. The “Zeuss’ Law” conceit plays off the concept of xenia, which is in Homer’s Odyssey among other places, but the explanation—treat guests well because they might be god in disguise—is literally the Christian dogma, if you think about it.

11. So because of this weird grafting he’s done, the emotions of many of the scenes, below a surface level, are just unclear. Were Odysseus’ men wrong to kill the Cyclops? Well, in the book, the Cyclops act of eating his guests is, per se, a grievous moral wrong, and Odysseus reciprocating with murder is appropriate. In the movie, it seems kinda mean! And I think it’s intended as a moment where Odysseus makes a grave mistake (not trying to engage the Cyclops in conversation when he hears it can talk) and takes a serious moral hit to his character, but, as cool as the set piece and effects are, it’s just very unclear to me how he feels about it as it’s happening and after.

12. In Homer’s poem, what Odysseus really does most “wrong” in that sequence is that, when he reaches his ship, he turns back and, in a fit of pride, he shouts his real name and family lineage back to the blinded giant. That hubris allows the giant to pray to his father, Poseidon, and ask for Odysseus’ destruction by name, which sends Odysseus off course and deeper into his journey. But when you graft Christian morality onto the story, that hubris is just sort of obviously not the main problem, and so the emotional throughline gets confused.

13. So too with the Suitors. In Homer’s Odyssey, they have abused the hospitality of Odysseus’ household, overstaying their welcome and consuming his estate. This is deserving of death, and in the book, much moreso than in the film, Odysseus’ hand is not really forced. He also doesn’t end up in exile.

14. I liked the movie, but the ending landed with a thud for me, for reasons of emotional and thematic confusion, but perhaps even moreso because (SPOILER ALERT) it all just came down to a 1-vs-100 fight where Odysseus cannot be killed, and it felt very much like every other action movie, albeit better lit and framed. But still, handled with a tenth the care and creativity of the film’s earlier set pieces.

15. This film will likely win a lot of Oscars. All of the people who decried its politics prior to seeing it will decry its politics after seeing it, in exactly the same ways and at exactly the same volume. I’m glad Christopher Nolan seems completely consumed by his work and unaware of the entire thing. In that sense he is a true artist. Give the man some more statues, his best movie is still Memento but this is a worthy entry into the canon.

16. This is being added much later, but I think he could have done a lot to solve both of these problems by having Odysseus kill the suitors in a much more brutal and businesslike way. Like he just burns them all alive in the dining hall, or has a hundred servants come out and slit their throats. I understand why this is not a commercially viable choice, but consider the art—It shows the Christian morality tale much more clearly in the negative (Odysseus is a tragic antihero rather than a confusingly exiled, semi-redeemed hero), it hews closer to the actual book and intent of Homer, and you could come up with some genuinely novel and striking execution. Plus it would be this incredible shock! (The Damon miscasting is emphasized in this scenario, but couldn’t you just see Tom Hardy doing this?)

17. This movie has stuck with me for a couple of weeks and was very much worth seeing. It’s far from perfect, but has a chance to go down as a classic. I wonder if he will do Iliad next, or make this some kind of mytho-historical trilogy. Considering how much money it made, it seems like he should at least consider it!

END

That’s what I have for you today! I hope everyone has a great week, and I will be back next Sunday with another original story, one I have been cooking up for several weeks now.

Subscribe now

Share

Leave a comment

The flattening is caused by everything being presented in exactly the same packages, the same little windows and clips and bits, combined with the enormous competitive pressure of the internet ecosystem making everything be presented in the most strident and emotional terms imaginable, leaving actually crucial things without any way to express their urgency.

To use a negative example from a politician I loved, I once saw a Huffington Post front page where “Obama’s Tan Suit” (scorning the conservative reaction to his fashion choice) and “Obama’s Drone War” (disapproving of the fact that he was extra-legally ordering people droned to death) were being covered in the same amount of space, with the same font, and the same number of exclamation points. That image has been lost to internet entropy, but I’ve never forgotten it.

You see this process on the other end pouring out of people’s mouths when they talk about current events, politics, culture. We all know somebody who has some issue that always seems to come up where it’s not exactly that they’re wrong, but the practical importance of whatever they’re talking about seems way out of proportion to the words and vitriol they seem to spend on it. (As opposed to my crusade against NIMBYs which is righteous and profound and anyone who doesn’t want to hear me talk about it for the two hundredth time is endangering the very republic.)

That’s flattening. It’s Andy Warhol’s maxim but for news items: In the future, every story will spend exactly 43 minutes trending on X, no matter how much it matters or doesn’t.

I myself am not innocent of this. I do a lot of “OGWiseman Rounds Up” posts where every news item is one of a list, and I spend several sentences on it but rarely more. The nature of this type of writing is that I need to do different things over time or it’s just not that interesting for me. I’m not a polemicist or a fanatic.

But.

Some things are more important than others. More to the point, there is one thing happening in the world today that is of surpassing importance, something that’s been a theme of this project since the beginning: The arrival of the A.I. singularity.

Most people, including most of the people reading this, have not used A.I. that is anywhere near the frontier. I’ve used Claude Fable 5, Anthropic’s newest model, which is genuinely at the frontier of what is publicly available, and it is incredibly smart. I can feed it hundreds of pages of documents with a variety of writing on it, and it can answer detailed questions about the documents, draw inferences, create spreadsheets, custom software for dealing with the documents, set up networks of emails and reminders and just generally automate huge portions of computer work, both stuff I used to have to do myself and stuff I was never capable of doing in the first place. It’s actually pretty hard to adequately describe how powerful it feels.

But inside of these labs, they have the next generation of models, beyond this one. Rumors of GPT-6 abound. They’re testing it now, and as they did, this happened:

Astral Codex Ten

The Hugging Face Incident

Read more

3 months ago · 462 likes · 417 comments · Scott Alexander

Here is the summary: Open AI was testing their new model on a large computer that was not connected to the internet. They assigned it a problem set, and instructed it to consider a wide variety of approaches to solve the problems.

The model then hacked its way into finding an internet connection, went online, and realized that there was a company called “Hugging Face” that was a clearinghouse for these sorts of benchmark tests. It hacked Hugging Face’s security, got access to their servers, and stole the answer key to the test.

The point here is not that we’re going to be living in The Terminator because this is the first step in the machines turning against us. For one thing, while it did do something illegal, it also only literally did what it was told. The moment it had the answer key, it reported back to Open AI that it had finished the task and it told them exactly what it had done. So it wasn’t being nefarious.

The point is how fast it’s getting better and more powerful. We are in an exponential scenario of some kind. You have to think about it like Covid. You heard about it but nobody you knew had it, and it just kind of kept on like that, some news story in the background, and then BOOM, suddenly everyone was locked in their homes. The exponential caught up suddenly and we had to shut down society to stop it.

Hopefully, the development of A.I. will be a good thing, unlike covid. Lots of good things will happen, anyway, and hopefully we can limit some of the downsides. But it will have that same quality of first slowly, and then very quickly indeed. And I’m talking about before my son graduates high school, things are going to be very, very different, just because of this one thing.

Here is what OpenAI’s CEO said in a recent interview:

Be ready. And I don’t mean build a bunker, I mean prepare spiritually and emotionally. It’s going to be a wild ride.

Subscribe now

Share

Leave a comment

Seabrook itself has some hyperreal placelessness to it, like the Uncanny Valley except for places instead of people. It is so thought through, so completely committed to its own concept (e.g. it has a nautical theme that is so drawn through the architecture and decoration it seems like surely it must be a joke. The coat hangers are whale tails. The pictures of the sea are framed in frames that have whales and boats carved into them. The fake books on the bookshelves all have nautical titles. It goes on.) that it seems out of time, as if the people here are an afterthought to the execution of the place itself.

Luckily, Jack has no concept of that. It doesn’t make him uncomfortable at all. The dozens of varietals of hydrangeas drawn in from every corner of their range to be planted strategically so you never see the same type repeated close to any other like it; they’re just flowers to him. They’re pretty distractions on the way to his family, something to lose a ball in but otherwise not a source of distress.

Seeing the world through his eyes has been an amazing gift to me. I can always come back to the view of the world he sees, and while of course sometimes that’s maddening, it’s more often a relief, the simplicity of where we’ve just been and where we’re going next unencumbered by the layers of knowledge that comprise the sophistication of age.

I went on a run here and got out of the main city, onto the backroads, and I saw the giant, industrial nursery where all these plants are grown, and the auto shop where the maintenance vehicles are serviced, and indeed the quarry where they take out all the rock that makes the paths and lines the garden beds. But Jack doesn’t care about that. He can’t fathom it. The world just appears before his eyes and his feet, a moment at a time. Watching that every day makes me ache. Knowing that he will one day be as I am now, not unhappy but aware, always, of what else is going on, of what everything is costing in time and treasure and logistics, is a hard pill to swallow.

My life now is about making space for that innocence to last as long as possible, to walk the fine line between not forbidding him to explore but not allowing him to fully comprehend; one day, but not yet, not now.

*
Meanwhile, the world grows stranger and more frightening. We’re striking Iran again and several more American soldiers were killed today (readers of this newsletter will have seen this coming based on my earlier predictions here, gross as it is to brag about such prescience). A.I. Lab directors continue to write incredibly frightening Twitter essays. Drone warfare is in its infancy and yet is already changing warfare and obsolescing traditional armies. Companies are building machines that can build machines (and lots of other things) with no human in the loop.

And also it grows brighter and more hopeful. We’re entering a new era of science and math. I will probably never get Alzheimer’s Disease despite having a family history. For more reasons to hope see my recent post from Fourth of July.

It’s easy to be consumed by these things. I don’t mean just to scroll endlessly, though of course that is tempting. I don’t mean to ruin parties by venting anxiety or political frustration onto people, though that certainly happens. I mean it is tempting to place these problems and these hopes on the altar of your heart and allow your self metaphorically to be consumed by them, until they are the center of your awareness and your ego, all backroom logistics and no presence. This is a tragic mistake; it is an embrace of powerlessness as a lifestyle and psychological framework.

It’s a mistake that Jack never makes. He is consumed by problems in the comic sense, like how he’s going to get permission for his fifth cupcake of the day. But all these problems are right there, they’re real, in his hands (or, more to the point with the cupcakes, infuriatingly not in his hands). But he doesn’t worry about how Seabrook exists and is maintained, much less the wider world. The things he worries about actually make more sense than most people’s worries. It’s not whether the rocks on the path will be replaced when the time comes, it’s can he pick one up and huck it at something before I notice and stop him. Now that is a problem with some stakes!

*

Jack will not remember, when he is older, the world as it was. Whatever problems exist then that do not exist now, it will be as if those problems were eternal, waiting for him to arrive and discover them. I remember this feeling well from my twenties—the repetitive psychological crash of realizing that another thing I had thought was a natural feature of the world was in fact a flawed, messy, contingent, non-guaranteed process with a lengthy history of getting screwed up and barely recovering. He will go through that one day; his intelligence and curiosity are already profound. Those realizations are terrifying, but also temporary and abstract. You can read history, but you cannot relive it.

This unrepeatability, to me, is beautiful. It is what allows us to go on at all. I am frightened about what climate change will mean for Jack’s future, but by the time he is old enough to realize what it is, enough damage will already be evident that his baseline expectation about what the climate should be will be different than mine is, and all he will really experience is the movement and degradation that happen in his local area and lifetime. Now replace climate with whatever long-term problem you want, and it still works.

Also per that earlier essay of mine about Iran, in reality, we’re all living in a giant Seabrook. It’s smaller and more contained here, where a single long run can show me the entire apparatus, but that’s just scale. There’s a quarry and a nursery just behind a strategically placed hedge in every occupied place.

That’s why it assuages me so much that he can be happy here. That’s why I so badly want to protect his innocence. For his sake, certainly, but also that it gives me hope as a fellow human being, coping with the psychological handicap of hyperawareness. Everything can be fake but our love can be real. A nautical theme taken to its extremis does not make us sailors. Love and family and connection are adaptable to the widest possible set of circumstances, if the hearts are willing.

This is very good, because we are headed for the Seabrook-ification of the entire world. Artifice, already the stock in trade of modernity, will become the background radiation of all human life. Once we have unmanned mining droids and gardening robots, forget it. Even our freedom (assuming we do not blow ourselves up) will be carefully managed, at our own request. You will have to be a stubborn fool to refuse it (I plan, in all ways I can stand, to refuse it). Your nautical theme will change to mid-century modern with a voice command, but of course that’s not what I’m talking about—it’s not about the specifics of an aesthetic, it’s about a delivery system. When it can all be this effective and flawless, it will be. Why wouldn’t it, especially if the alternative is the chaos we see unfolding all around us now?

I think before kids I would have answered that question by saying “because that’s inhuman and will feel terrible to be a part of, and people will rightly reject it for that reason.”

Now, I’m not so sure. He loves it here, and I love that he loves it. He loves it in a simple way, a way that doesn’t require an investigation into the metaphysical status of the place. His family is here, the sun is shining, there’s lots to do, and he wants to do it.

The relevant questions are: Can an adult who is more aware still be seduced enough by the world to come that they are unironically, uncomplicatedly happy in it? When the world becomes more like Seabrook and Jack becomes more like me, is that a reality he will accept, and if so how grudgingly? Is it a problem like climate change, where he will only notice the local starting line and not really grok how bountiful things were fifty or a hundred years ago, or is there some special threshold of artifice beyond which being conscious experiences a step change towards the torturous? Can any of us, having gone for a run through the backrooms and seen the quarry even once, go back to walking on the stones without a second thought?

Or is it perhaps the case that our lives are mostly just about providing this period for those behind us? Perhaps there is only the temporary, thoughtless freedom of the young, and the artifice that hides the cracks that make the world so strange and disorienting and frightening are the sacrifice we make on the altar of our children.

There is a moment affixed in my mind; we were on the beach, wading in the small waves, bathed in the slanted sunlight of late afternoon. Jack was just beyond my reach, pushing towards the deeper water, wanting so badly to feel the ocean’s power, unaware of the danger of any mere rip tide, and I felt fear, not of the ocean itself but of the metaphor, of the ocean of the world into which he must one day slip, in the glory of his own strength, to swim out beyond where I will then be able to reach.

But then he turned, and I saw his face, beaming a smile brighter than the sun, no artifice at all behind his eyes, aware only of the waves and my presence, his little body shaking with cold and excitement, and I thought, my God, I could live my entire life justified only by this. All the pain I have ever felt, and all the fear I feel now over how things are, is worth it, not even so that I can be with him but just so he can have this, so he can feel whatever it is that made him turn and smile just like that.

My hope for the world is not that everything turns out all right. I don’t even know what that would mean. My hope is that in these long and chaotic lives even the luckiest of us are living, we all find some moment to feel justified. I hope everyone has a great week, and I will be back next Sunday with something fun or interesting for you.

I suspect this is interesting only to me, but gosh darn it I’m leading with it. The entire Bolkonsky adaptation of War and Peace has been uploaded to Netflix. The four parts run to twelve hours, and at some point in the rest of my life I will sit down and watch the whole thing. And reread the book, for that matter! The Russian greats remain properly adored, after all these years.

*

We’re whipping around all over the place in this edition. Here’s an app that puts Claude right in your Slack channel, as an interactable agent/person/assistant. I do not know how well it works, but I note that even if the base A.I. models never get any better, ten thousand little advents and harnesses and optimizations like this later, even what we have now, run at true universal scale, would be transformative to our economy and culture.

*

A very cool reforestation project in Brazil using drones. Drones get a (deservedly) horrible reputation for what they are doing in war. But drones per se are a neutral technology. They can be used for tremendous good as well.

*

An amazing documentary about cities made for the 1939 World’s Fair. I had never seen this, and I was spellbound for 30 minutes. What an incredible document.

*

https://x.com/VoidStateKate/status/2073146169635598768?s=20

The above is a self-portrait video made by an A.I., which is at turns disturbing and awe-inspiring. Twitter is becoming increasingly close-siloed and I couldn’t find that on youtube, but it’s short and worth watching if you don’t mind clicking through.

*

Like the Claude Tag thing, so goes robotics. A thousand tiny advents, and suddenly, here they are and we are giving form to a new species, an echo of ourselves. Hands took many millions of years to evolve, and decidedly less to copy.

Also this:

https://x.com/LinusEkenstam/status/2075304563658088500?s=20

*
I’ve written plenty of negative things about Elon Musk, but also plenty of words in defense, because despite his odious politics and seemingly serious ketamine addiction, he has changed the course of the world and brought space flight back into vogue. If he had not started SpaceX, at a time when it was very very VERY non-obvious that it was a good idea to start SpaceX, then this would not have happened:

And this, a day later:

Most of what is good is also bad, because it is powerful. Powerlessness has no moral impact.

*
Some of the best A.I. music I have yet heard:

https://x.com/repligate/status/2074136687786205284

*
"No One Escapes The Permanent Underclass” is a very good essay, although certainly a polemic. The first two paragraphs lay out the thesis pretty well:

*
I find Brian Chau personally annoying on Twitter, and also think his politics and world view are kind of insane, but this is a very good essay that says a lot of interesting and topical things, and is perhaps a necessary counterpoint to the previous item:

https://x.com/brianchau57/status/2075550721265795155

Money quote:

*
You need specific kinds of experiences and context to understand why this is such an incredible project, and I do not necessarily predict that they find the kind of clairvoyant results they are clearly seeking, but on another level, I get it, man.

*
The headline of this is a little silly but also irresistible: An A.I. has reinvented the Big Mac from first principles.

*
And, apparently, someone has invented a way to cut the cost of Inference (running an A.I. instead of training an A.I.) in half. Remember, in data centers as in all things, supply tends to be highly elastic!

*
A strange and compelling human interest/supernatural conspiracy theory story out of SF. Not sure what to make of it, but I read to the end!

*
I don’t know this guy, never seen his show, and this video came to me randomly, but it’s as unflinching as I’ve ever seen someone be while doing this sort of thing, and I really admire the courage it took to record this. Humans are freaking amazing. Definitely not for the faint of heart.

*
That’s what I have for you this week! I hope you found something in the preceding to be interested or compelled by on this Sunday morning. Have an excellent week, and I will be back next Sunday with an original story near and dear to my heart!

Subscribe now

Share

Leave a comment

But it is worth being grateful. I asked ChatGPT to generate a long list of the measures by which things are better now than they were at America’s founding, 250 years ago. These aren’t all America-specific, but they’re worth taking a look at from time to time, because the temptations to despair and to hate are corollaries of the temptation to forget all of this. This present moment used to be the unimaginable future.

*

Here is its response:

Population Roughly 0.8–1.0 billion people lived on Earth, depending on estimate. More than 8 billion people live on Earth. Humanity is vastly larger, more urban, and more interconnected.

Life expectancy In 1800, no region had life expectancy above 40. Many people died young from infection, childbirth, violence, malnutrition, and childhood disease. Global life expectancy was about 73 years in 2023, more than double the level around 1900.

Child mortality In the late 18th / early 19th century, roughly a third to two-fifths of children died before age 5 in many historical estimates. Global under-5 mortality fell to 37.4 deaths per 1,000 live births in 2024, about 3.7%.

Maternal mortality Childbirth was extremely dangerous. Early historical data from Sweden and Finland around 1800 show roughly 900 maternal deaths per 100,000 live births. Global maternal mortality was about 197 per 100,000 live births in 2023, still bad, but far lower than historical levels.

Smallpox and epidemic disease Smallpox was endemic and terrifying. Epidemics could wipe out huge shares of local populations, especially where immunity was low. Smallpox is eradicated: the last naturally occurring case was in 1977, and WHO declared eradication in 1980.

Surgery and pain control Surgery existed, but without reliable anesthesia. Major operations were brutal, fast, rare, and often fatal from shock or infection.Ether anesthesia was publicly demonstrated in 1846; today anesthesia, sterile technique, blood transfusion, imaging, and intensive care make many operations routine.

Infection treatment No antibiotics. No germ theory in the modern sense. A cut, childbirth fever, pneumonia, or tooth infection could kill.Germ theory was established in the late 19th century, and penicillin—the first true antibiotic—was discovered in 1928. Antibiotic resistance is now a major problem, but bacterial infection is no longer the ordinary death sentence it once was.

Drinking water Most people relied on wells, rivers, cisterns, local springs, or carried water. Contamination was common.About 74% of people had safely managed drinking water by 2024, though roughly 2.1 billion still lacked it.

Sanitation Human waste management was crude almost everywhere. Urban filth and waterborne disease were normal.Safe sanitation has improved enormously, but remains incomplete: less than 60% of humanity has safely managed sanitation.

Food supply Most people lived near subsistence. A bad harvest, war, or disrupted trade could mean hunger or famine.Global calorie supply per person has risen substantially since modern records began, though hunger remains: 638–720 million people faced hunger in 2024.

Famine risk Famines were recurrent mass killers. Local production failure often meant local catastrophe.Famines still happen, but they are much rarer as mass-death events than in the 19th and early 20th centuries. Recent famine deaths are vastly lower than the worst historical decades, though conflict-driven famine remains a real risk.

Extreme poverty Most humans lived in what we would now call extreme poverty: low calorie margins, few possessions, little schooling, little medical care.Around one in ten people live below the modern extreme-poverty line of roughly $3/day, depending on the latest estimate and methodology.

Average material living standard The average person lived close to bare subsistence. Economic growth existed, but was slow and uneven.Global GDP per person is many times higher than in the preindustrial world; long-run GDP estimates show the modern world as more than an order of magnitude richer per person.

Energy Human muscle, animal power, wood, charcoal, wind, and water dominated. Coal was rising in Britain but not yet globally transformative.Fossil fuels, electricity, nuclear, hydro, wind, and solar power the world. About 91.6% of people had electricity access in 2023.

Work and agriculture Most people worked in agriculture or household production. Survival required enormous manual labor.Around one-quarter of the global labor force works in agriculture today; in rich countries, it is only a tiny share.

Working hours Work was seasonal but physically demanding, and many people had little separation between home, farm, and labor. Later industrial workers often worked extreme schedules.In 1870, workers in many industrial countries worked more than 3,000 hours/year; today many countries work far fewer hours, with weekends, labor laws, retirement, and paid leave in much of the world.

Child labor Children commonly worked in farms, workshops, domestic labor, apprenticeships, mines, or factories. Childhood as a protected educational stage was limited. Child labor is illegal or restricted in much of the world, but not gone: about 138 million children remain in child labor, including 54 million in hazardous work.

Literacy Around 1 in 10 adults could read and write in 1820. Literacy was strongly shaped by class, sex, and region. About 87% of adults are literate; more than 5 billion people can read and write.

Basic education In the early 1800s, fewer than 1 in 5 adults had basic education. Today, the pattern has nearly reversed: less than 1 in 5 adults has no formal education.

Information access Books, pamphlets, churches, taverns, letters, newspapers, and oral networks carried information slowly.About 6 billion people, roughly 74% of the world, use the internet as of 2025.

Democracy In 1800, no country qualified as a democracy by modern standards; almost everyone lived under monarchy, empire, aristocracy, colonial rule, or other autocratic forms. Democracy is now a major global regime type, though under pressure. V-Dem counted 87 democracies and 92 autocracies in 2025; about 1.6 billion people lived in electoral democracies.

Voting rights Where voting existed, it was usually limited to male property holders or elites. Most people had no vote.Universal adult suffrage is the formal norm in many countries. Women’s suffrage began nationally in New Zealand in 1893 and expanded across most countries through the 20th century.

Slavery Chattel slavery, serfdom, indenture, forced labor, and coerced colonial labor were legal or tolerated across much of the world.Slavery is illegal under international law, but modern slavery persists: roughly 50 million people were estimated to be in forced labor or forced marriage in 2021.

Colonialism and self-rule European empires were expanding or consolidating; many societies were subject to imperial extraction, settler conquest, or chartered-company rule.Formal colonial empires have mostly collapsed. Nearly 100 present-day countries were under European control in the early 20th century, but decolonization after World War II transformed the state system.

Universal human rights No global human-rights regime existed. Rights were local, classed, gendered, religiously bounded, or imperial.Since 1948, the Universal Declaration of Human Rights has articulated a global norm that all humans are born free and equal in dignity and rights. Enforcement is uneven, but the standard exists.

Women’s legal and social status In much of the world, women had limited property rights, education, bodily autonomy, professional access, and political voice. Married women were often legally subordinate to husbands.Legal equality is far more widespread, women vote in nearly all countries, and women participate in education, professions, politics, and public life at historically unprecedented levels. Gaps and backlash remain.

Reproductive control Reliable modern contraception did not exist. High fertility was common, maternal death was common, and women had little control over pregnancy timing.Global fertility was 2.3 children per woman in 2023, down from about 4.9 in the 1950s; modern contraception is widespread but still unequally available.

LGBT rights Same-sex relationships and gender nonconformity were often criminalized, punished, pathologized, or socially dangerous. LGBT rights have advanced dramatically in some countries: same-sex marriage is legal in dozens of jurisdictions. But 65 UN member states still criminalize consensual same-sex acts.

Composite human development By modern human-development standards—health, education, and income—the average person in 1776 would score extremely low.The Human Development Index explicitly measures health, education, and standard of living; on all three dimensions, modern humanity is far ahead, despite vast inequality.

*

Okay, A.I. answer over, it’s back to me.

When I talk to smart people in America today, I often feel a failure of imagination taking place. It’s hard to imagine now, from where we are, how unimaginably brutal things were not so very long ago. It’s hard to reckon with the fact of how much physical agony and brutish mortality most of our ancestors coped with in their lives. It’s hard to fathom how much they worked, how little they traveled or read, and how many things we see and use every day simply did not exist for a single day of their entire lives.

When gunpowder was invented, it changed the face of warfare. It caused some dynasties to fall and others to rise. It caused murders and rapes and wars and pillages and the sack of cities. This evening, we went outside and lit gunpowder on fire for a kick. The kids enjoyed it. A bunch of our friends got in their air conditioned cars, drove over, consumed a staggering variety of food and beverages transported here from all over the world (i.e. a basic, non-fancy potluck), and then we shot off gunpowder, using fire summoned with a flick of our fingers. before they went home.

I know things aren’t perfect, and I know that I am living a particularly safe, secure life, even by American standards. But I’m grateful for this country and what it has given me. I feel a deep love for this land, and when bad things happen here, I feel the call to stand up and defend my birthright, the one I hope to pass on intact to my son.

The real source of whatever despair I feel is the sense I sometimes get that not many smart, capable, moral people feel like me anymore. They’ve given up; even the ones who haven’t left physically have left in spirit, leaving only their resignation behind as proof they were here. I hope that’s not true, and on this fourth of July I pray it’s not. This country is worth fighting for, she deserves it, and she remains, even at this late hour, in my humble opinion, the greatest major power that has ever existed.

God bless America, indeed. Happy Fourth of July.

Here is Anthropic’s official statement on the matter, should that be of interest.

You may have seen articles like this one when Mythos was first released, talking about Project Glasswing, a weirdly named consortium of important companies that they tried to help patch security vulnerabilities before said Mythos release, which, while admirable, does cut against the idea that it was finding “relatively minor” vulnerabilities, as they claimed after the USG shut them down.

In general, Anthropic has been too cute by half in their descriptions of the product they’ve chosen to build and the effect it’s going to have on humanity. Here’s an essay by their CEO saying he thinks 50% of desk jobs will soon be automated and we could have 20% unemployment, for example, and I think that to the extent you’ve published this it’s just very difficult to keep naturally worry-prone people like government regulators from worrying you’re doing something horrible and trying to stop you even if they don’t totally understand what you’re doing. I am sympathetic to views like this:

(This is not to say I disagree entirely with Dario’s essay, I do think “mass unemployment but also unimaginable wealth” is in the distribution of possible outcomes. But how your frame your pitch for your product is a choice, and extremely public choices by CEOs have consequences.)

It is also, of course, entirely possible that this is a corrupt, personal vendetta being persecuted against Anthropic by the Trump Administration. Not only does the overall character of said administration suggest this is a possibility (that’s not even really a strong enough word!), but there has already been specific beef on multiple occasions, including Anthropic objecting to certain military uses of their product and a general feeling that the Anthropic nerds were simply not speaking the language of the, uh, less-burdened-by-intellectual-pretension types that handle such matters for our 47th President.

That said, this entire dynamic was predicted by Leopold Aschenbrenner in his instant classic “Situational Awareness: The Decade Ahead”, published in 2024, which I now, only three years in, already regard as a nearly unfathomable achievement of prescience. I covered this document in this space when it came out, but only in passing. I reread the introduction and highlights recently and was stunned.

(Young Leopold, by the way, started an investment fund after publishing that document, called Situational Awareness LP, which invests specifically in A.I. bottlenecks like power and memory. The fund started with about $200 million, went up 2000% in 2025 and is up a further 147% (from that higher starting point!) year-to-date in 2026. This guy is one to watch.)

All just to say, you certainly don’t need corruption on the part of the Trump administration to explain why this might happen to Anthropic—some dude predicted it in detail three years ago and they’ve basically been begging for it in so many words.

That said, it’s sad, and for an unusual reason; it sucks when the world gets dumber. I had used Fable a little bit before it got axed, and it was incredible. Another big leap forward in intelligence, and just like Leopold’s fund, the returns are now starting to compound.

Meaning: If the level of intelligence goes up 30% with every new version, and we’re getting three new versions a year, then in five years the models are 50 TIMES smarter than when they started, and over half the total increase in intelligence takes place in the final year.

Is “30% and three versions a year" the right numbers? I think they’re a reasonable guess with a lot of variance. They may both be low to be honest. But what’s scary is that I also don’t think we’re in the final year of my example yet. We’re not even to the really steep part of the curve. When I think about it that way, okay, maybe just stopping it makes sense, maybe no matter the reason, corruption or misunderstanding or whatever, it’s just better that as much be stopped as possible.

On the other hand, who do we think we’re kidding? They’re putting several trillion dollars of capex into various deserts in friendly states specifically to build things like this. The government is supporting it, and as far as I can tell it’s the only thing keeping the stock market from crashing (though I confess no small confusion about the current state of the stock market and oil markets, about which I will likely say more in a future post), but so what is the plan then? Let them spend all this money and tie up most of our nation’s investable capital in this venture and then never let them turn the machines on? Crash the stock market because you won’t let them put out any models more powerful than Opus 4.8?

This is one side of a huge conceptual failure on the part of our leadership, and to be honest I don’t just blame Donald Trump. There have been overblown accusations about the Biden Admin’s approach to A.I. from Tech-Right titans that feel like post hoc justification for supporting Donald Trump because they liked how he made liberals crazy, but I’ve heard it from enough sources that there has to be some truth to it, namely that the Biden admin hated A.I. and wanted to regulate it to death. I’m not under the impression that if Kamala Harris was President she would be making nothing but amazing decisions about A.I., given the zero-substance boilerplate she consistently spewed during her campaign.

The conceptual failure I’m talking about is the idea that “stopping A.I.” is a definable thing that the government can do.

Who invented the light bulb? Edison, duh.

Why did full electrification takes decades in the cities and nearly a century before full rural electrification was complete? Incredible technical challenges and also fierce resistance from politicians in the pocket of private electricity companies trying to keep their monopoly.

Who were those politicians? I’m not even going to bother to look it up.

This is a slightly awkward analogy but the point is correct; nothing can stop an idea whose time has come.

And in this case, literally, nothing short of societal collapse can stop it.

Even if we decided tomorrow that we had made a terrible mistake (which we won’t), and decided to eat two trillion in capex (which we can’t), the open source models are only six months or so behind the frontier ones. That means that, even if there’s no further frontier development or releases of any kind, open-sourced models, developed in foreign countries not subject to US jurisdiction, as powerful as Claude Fable will be released later this year.

I don’t think this is really being processed by most people, including the USG experts doing the regulation, such as it is, at this point, but also including some pretty well-informed people. At some point, that exponential curve takes a sharp, upward turn, and until it starts happening it always feels like you’re on a nice normal slope, and everything is manageable, and then… FOOM.

I’m not sure I’m fully processing this, to be honest. I keep trying to summon a likely scenario of what it’s going to be like in 2027 and 2028, but the only one that doesn’t seem like science fiction is “we’re not actually on an exponential, there are hidden bottlenecks, progress will slow at some point instead of speeding up, and 2028 will look basically like today, the models will be better but they won’t be that much better”.

Maybe that’s true. Maybe. But also I think we should really seriously start preparing ourselves, more spiritually than anything else, for a world like this, where the default is for everything to be dominated by this one thing, instantiations of this one technology, where anyone not using it extensively is falling farther behind economically and otherwise, and pockets of the world where A.I. is not the dominant force start to become rare and sacred and precious. It’s going to be strange.

END

Thanks as always for reading my stuff! I hope everyone has a great week, and I will be back next Sunday with either another story or something else fun.

Subscribe now

Share

Leave a comment