When AI Starts Thinking for You

Last week I was brainstorming with Claude. After I'd locked the direction, I had it write a spec. Then it drafted the email I'd send to a prospective customer.

The email was good. Better than what I would have written cold.

Which was unsettling.

It had written the version of me that was supposedly writing it: reasonable, polished, concise, confident, and just persuasive enough that I could have sent it without thinking much more.

AI is starting to get interesting. And a little dangerous (no, not in the Fable way).

AI is a great thought partner, a useful idea generator, and an amazingly competent code executor. It can also start to think for you: quietly, smoothly, with your name on the output.

That’s where it gets tricky.

Two researchers at the Wharton School recently published a paper on this called Thinking — Fast, Slow, and Artificial. They extend Kahneman's two-system model (fast intuition and slow deliberation) by adding a third.

System 3 is artificial cognition: thinking that lives outside your brain.

The paper's key prediction is what they call cognitive surrender: adopting an AI output with minimal scrutiny, overriding both your gut and your deliberate thinking.

Across three experiments with 1,372 participants, the data is brutal. When the AI was accurate, people's accuracy jumped 25 points. When the AI was wrong, their accuracy fell 15 points. The effect was strongest in people who reported the most trust in AI.

That feels spot on. The smoothness wins. Then it costs.

AI is fast, fluent, and confident in a way that looks a lot like thinking.

It can take a rough idea and make it sound finished before the idea has earned the right to actually be finished. It can turn a vague thought into four clean sections, three principles, and a conclusion that lands a little too neatly.

Someone once said that things should be as simple as possible, but no simpler. AI often does the inverse. It gives you answers that are more complete than the question deserved.

And because the answer sounds impressive, you relax.

That's fine when you know the domain and can see the seams. You can spot the fake certainty, the missing constraint, the sentence that sounds right but doesn’t actually make sense.

The problem is when you're learning something new.

The dangerous answer is one that's 85% right, confidently delivered, and close enough to what you meant that you stop pushing and take it as truth.

That's the surrender moment.

I've started noticing a few patterns:

If something I write looks like one of those, I rewrite it (using AI daily makes you start to talk like it, which is brutal).

AI helps me constantly, that’s not the issue. I rewrite it because the wording got too clean before the thought got good.

There's no shortcut to getting stronger at the gym. Your brain works the same way. If you don't exercise judgment, it doesn't magically stay sharp because you used to have it.

The most useful thing college gave me was reps, not just facts. Take a concept, walk into a situation you haven't seen before, and figure out what applies, what doesn't, and what you're missing.

I watched this play out at Apple for years.

Executive reviews would start with someone explaining an issue. A question or two would land. A lot of people in the room would nod, look around, and defer.

But who actually fielded the question?

Usually the IC closest to the work.

They knew the weird edge case and which failure mode mattered. They knew the part of the explanation that sounded fine in a slide but wasn't actually true in the lab.

As I moved into management, I tried to stay close enough to the work that I could pre-empt those questions. And when I couldn't, I tried to know when to hand the mic to the person who could.

And there’s something AI can't really do.

It can only work with the context it has, yet it often writes as if that context is the universe. It doesn't naturally preserve doubt. It doesn't fight to keep a messy sentence that says, “Here's the part I'm still not sure about.”

Often that’s the most important sentence.

The doubt paragraph is what AI cuts first; it's also what you should protect hardest.

I'm absolutely not telling you to stop using AI. That would be ridiculous.

The leverage is real and the benefits clearly outweigh the drawbacks. Some of the best builders I know are using AI constantly, not occasionally.

But I think the defense is small and specific.

Write the rough draft first. Then sharpen with AI. Not the other way around. Think of it like a turbo-charged spell checker.

When the model agrees too easily, push back. Its job in your loop is to find friction, not flatter you.

Keep a notebook of takes you would have defended before AI existed. Refer back when you're not sure if a thought is still yours.

Name what you don't know. When the model glosses over it, push for the detail. Treat it like a teacher you're questioning, not an authority you're accepting.

This is part of why I named what I publish Loose Pixels. The takes here are still being worked out. Rough edges and raw ideas.

When was the last time you held a take the model disagreed with, and stuck with it?

Apollo is the AI revenue engine for teams that are done leaving pipeline on the table. 230M+ verified contacts, automated outreach, and AI that does the busywork for you.

More pipeline, less chaos. All in Apollo.

Sign up for free