The Trouble Isn't Seeing Inside the AI

People want to open up AI like a watch. They want to see the gears and point at one and say that's why it did that. When they can't, they get nervous. If you can't see the reasons, how can you trust the answer?

I used to think that was the important question. Now I think the real problem isn't about the AI at all. It's about me.

Here's what I actually do. When the AI helps me with something I already know, like code in a language I write, I trust it about as far as a fast but careless junior. I read what it gives me and I catch the mistakes, because I know what a mistake looks like there. But when it answers something I don't know, some domain where I couldn't tell you if the answer is brilliant or nonsense, I take it and hope.

That's backwards. I trust it least where I need it least, and most where being wrong costs the most. And I think the reason is that I was never really checking the AI. I was checking it against myself. When it missed something obvious, I caught the miss because I had my own answer to compare it to. The obviousness was in me, not in the output. Take away my knowledge of the domain and there's nothing left for the mistake to be obvious against.

So a better AI doesn't make this safer. A clumsy one sets off alarms. The code won't run, something feels off. A fluent one hands you a confident, plausible answer and never trips the alarm, because fluency is the signal we're worst at doubting. We read smoothness as understanding, and it isn't. A fluent answer only tells you the model is good at sounding sure.

What bothers me is what this did to me without my noticing. I didn't lose the ability to catch the obvious miss. I lost the pause where I used to catch it. There used to be a few seconds where I'd think about a problem before doing anything, and that pause is where the judgment lived. Now it's gone, because the AI is faster than the pause, and each time it's faster I skip it. Every single time, skipping is the efficient choice. It's only across all of them that I come out worse.

That's why the trap works. Nothing about it feels like a mistake while you're in it. You're not being lazy, you're being efficient, and the cost shows up later on a version of you that's worse at catching things than the one who started.

So I've stopped asking whether we can see inside the AI. We probably can't, the same way we can't really see inside each other. The question I ask instead is whether I'd catch it if it were wrong. When the answer is no, the thing I should distrust isn't the AI. It's that I stopped pausing long enough to notice I'm flying blind.