Too much thinking

I keep struggling with the “frontier” AI models. To be honest, I find them insufferable. I ask them for A and they will jump from B to C to D, E, F, and come back with G. They inject their own instructions into my prompts and drop strange ideas into our conversations. They constantly use the pushback sandwich form, often pushing back on ideas or interpretations that they themselves inserted into their own chain-of-thought. When I ask them to do research, they keep coming back with the same ideas. Mention the word “flow” and they will come back with a summary of Mihaly Csikszentmihalyi’s work on flow. Talk about constraints and they will come back with the story of how “Green Eggs and Ham” was written.

Recently I saw a Reddit thread that showed one of the Claude models mistaking part of its own system prompt for a prompt injection. The model was 100% certain, and 100% wrong. It was so wrong it was idiotic. But it couldn’t see that.

I blame this all on the drive to make autonomous agents. I’ve said it before, and I’ll say it many more times: AI engineers don’t know how to make intelligent systems. What they did was build a system to mechanize language. There’s a lot of value in doing that but it doesn’t mean that it will scale to superintelligence. Not until they learn how to build systems with a little bit of common sense.