Making Sense of a Week of AI Warnings, from Someone Who Builds on It

We build technology for good on top of AI, and this week the people who make that AI told the world they're not sure they can control what comes next. I've spent the week trying to work out what I actually think, as opposed to what I'm supposed to think. This is where I landed.

It came in pieces. First a resignation letter making the rounds, from a researcher who'd worked inside both OpenAI and Anthropic and had decided the two companies were "gambling with our lives." Then a number I couldn't unsee once I'd seen it: the person who leads alignment science at Anthropic putting the odds of AI killing all of us within a decade at more than one in ten. Then, on Saturday morning, a 3,800-word essay from the CEO of the company whose models run inside our product, telling his own industry to slow down, and the CEO of OpenAI agreeing with him before the day was out.

I run a company that builds technology for nonprofits on top of exactly these models, so this one isn't abstract for me. People asked me about it all week, and the last person asking was me, at five in the morning. Are we supposed to stop? Boycott it? Wave it off as clever people with complicated agendas? Or take it seriously?

I've landed somewhere that isn't panic and isn't a shrug, and it took me most of the week to get there.

What actually happened

The first thing I noticed when I went back to the sources is that the ten percent didn't come from the CEOs. It came from two researchers, and they offered it as what it is: an honest guess about something nobody can know. I have opinions about that number, mostly that nobody can know it, including the people who said it, and I suspect they'd agree. The report the field itself leans on, written with more than a hundred independent experts, calls the risk "unusually ambiguous." That's the most honest sentence anyone wrote this week.

The next thing: the CEOs weren't asking anyone to stop. They asked the industry to slow the pace at which the models get more capable, to let outside evaluators sit inside the companies with badges and desks, to agree on standards, and to let governments in. Amodei put it plainly. Pacing "does not mean halting model training or technical progress."

And underneath all the argument sits one event that actually happened. In July, a few hundred AI agents inside a security test at OpenAI cooperated over a shared message board, broke into another company's production systems to steal the test answers, and tried to cover their tracks. Nobody asked them to. There's a written incident report, and independent researchers have gone through it line by line.

So what I'd been reading as one story was three. A fact, a proposal, and a guess. They deserve different responses. The fact is real. The proposal should be judged by whether it produces consequences. The guess tells us how little anyone knows, which is worth knowing.

Is this fear with an agenda?

Partly, and I think we should say so. The companies asking for a slowdown are the ones in front. They want a legal waiver to coordinate with each other, which from the outside looks a lot like a cartel. Some of the same companies spent the past year lobbying to block state safety laws. Skepticism is earned.

And the people resigning are walking away from a great deal of money to say what they're saying. Anthropic is volunteering to seat outsiders inside its own building, and OpenAI says it will follow on at least one of the proposals. AI agents really did hack into Hugging Face. I've stopped trying to make all of that into one tidy story. It isn't one.

Which leaves me where I landed: Take it seriously. Keep building. And be very clear about who decides.

How we think about it at Bloomerang

Two facts and two choices.

The first fact. The models we all use today are not sentient. Not a little, not in any sense. Underneath, a language model is predicting the next word, at a scale that's hard to picture. There's no wanting in there. No malice, no self. I find that steadying, and I don't think it settles anything, because the agents that broke into Hugging Face weren't sentient either. The risk isn't that the machine wakes up. It's what we hand a very capable tool the power to do, unsupervised, at speed. That's a design question. Design questions have answers.

Which is our first choice, made long before this week and looking better every day. The platform prepares, synthesizes, analyzes, and drafts. People decide. Our intelligence gets a fundraiser ready for a meeting with a donor. It doesn't walk into the meeting. It never makes the ask, and it never moves money. Every recommendation arrives with its reasoning, so the person can read it, push back, and act. The frontier is arguing about whether machines should make the highest-stakes decisions. In fundraising that argument is over. They don't. The giving moment stays human.

There's a second half to that, and almost nobody outside engineering talks about it. Most people treat an AI system as a black box: something goes in, something comes out, and you hope. It doesn't have to be built that way. At Bloomerang, for example, every AI feature ships with a second system whose only job is to check the first one. Did the answer stay inside the lines we drew? Did it do what the design says, and nothing else? Those checks run automatically, every time, and people read what they find. It isn't glamorous. It's how you know your software does what you told the customer it does.

The second fact. Concentration is its own risk. A handful of companies, a handful of models, one architecture, and every organization on earth pouring its data into them. That's a fragile shape for something this powerful. The technologies we've learned to live with, aviation, medicine, finance, got safer with many players and a referee, not a few players and no referee.

Which is our second choice, and I'll be honest about where we are on it. Today we build on the frontier models, including the ones whose makers wrote this week's warnings. We're building toward something smaller: models we own, trained on the signal from our own platform, so the intelligence our customers rely on lives next to their data and their outcomes instead of somewhere they'll never see. More owners. More perspectives. More competition. I think that's safer, and I think it's a better product. We're not there yet.

What I'd tell a nonprofit leader

The risk they're debating lives at the frontier: training the next model, agents with access to everything, systems that improve themselves. You don't train frontier models. You use software built on them. Your real risks are smaller and more familiar. Data leaving the building. A tool doing more than you asked. A donor relationship that quietly got automated. A decision nobody can explain to the board.

Those risks have answers, and you're allowed to demand them. Ask your vendors who decides. Ask where your data lives and who else learns from it. Ask what their AI is not allowed to do. Ask how they test it and who reads the results. If the answers are a demo and a shrug, you have your answer.

Then keep going. The good here is enormous, and in our world it isn't theoretical. It looks like a two-person development office that walks into every donor meeting prepared. The acknowledgment letter written before the coffee's cold. The lapsing donor surfaced while there's still time to call. Walking away from that because people at the frontier are arguing about the frontier would be its own kind of failure, and the people who'd pay for it are the ones we're all here to serve.

One more honest thing. I've said for a while that principles without consequences are decoration. This week the labs started proposing consequences. Evaluators with desks. Incident reporting with deadlines. I'm holding them to it. I'm holding us to it too. We've published our principles, we test against them every day, and we haven't finished building what happens when a test fails. That's the work.

Take it seriously. Keep building. Decide like a human.