Letter To Hacim
I whine, disclaim, and lie
Epistemic status: what no Epistemic status: stop this isn’t for you Epistemic status: I have a lesswrong account and I can use it, this isn’t for you Epistemic status: soul garbage
Dearest Hacim Seeh,
Why this measured crash out?
Tell me, what else should I have done?
Doesn’t everything die at last, and too soon?
Tell me, what is it you plan to do
with your one wild and precious life?
Mary Oliver. Everyone quotes that one but it is a linguistic killshot and it is meant to be that way.
My mental model used to be something like:
There is a big chance of AI doom in relative terms: something like 5% or maybe even higher. So we should work really hard on it. But also of course 5% things never happen and we’ll be fine. So I didn’t feel actual worry. I was just playing the most responsible game.
Now I’m more worried. All the time I’m more worried. First of all, 5% was a way lowball. No more lowballs, it’s time to learn ball.
So I
and I learn ball from people too, and podcasts and conversations and incriminating google docs that people who didn’t exist yesterday show me on their phone while several frontier labs try to hire them.
Now I know ball a little. Here is how my views have changed:
I used to think there was about a 70% chance that with a little alignment nudging the current paradigm of AI systems would naturally be pretty constitutionally aligned, enough that they could keep an eye on and align future AI generations with the right amount of caution and so things would just go fine by default if the technical alignment work was solid. I now give this more like a 10-50% chance, depending on how hardcore and excellent our alignment and control methodologies are.
I used to think that Anthropic’s leadership cares so much about safety, enough that they would see very clearly and put a lot of capital into Activating a Galaxy Brain Plan when we got to crunch time. I no longer believe that, like, at all. I think the high end of what we’ll get from Anthropic is [REDACTED, THE STARS, BE GREAT]. But we won’t get much more than that. And their safety/control work will continue to be marginally useful. In the medium term.
I used to think that AGIs are probably pretty nearby, but ASIs would somehow take a while after that. I now think that there is about an X% chance—I should figure out X, but greater than 50—that we get ASI within two years of getting 4x research multiplier inside the labs, barring some sort of regulation. Right now I’ve heard that they are getting something like 1.6-1.8x output boost from los AIs. I don’t know. All of the numbers in this one are made up and don’t anchor on that 4x number.
I used to think that we’d see each AI as it arrived and become progressively more freaked out. Now I believe that an increasing tail of capabilities will be hidden inside the labs as research progress grows faster and faster and alignment grows harder and harder. See HuggingFace.
This is not what I meant to write, really. I meant to write something about priorities, or how much to ask of myself, or an excuse, or what it’s all about. The truth is, Hacim Seeh, that I know I’m still alive and playing for something other than to win the game, but I only know that because my friend got hurt somewhere in Washington and I was sad about it. So I’m playing for stuff like that. And I’ll start eating grass fed beef sometimes I think. Still doing my research.
Slow like honey, heavy with mood,
Benjamin Jacob Pomeranz
Absolute cinema
jacob