🧠 Transformers Are Impressive – but Are They Really the Future?

The diagram below (from mechanistic interpretability research) is one of the best examples of why the Transformer architecture is hitting its limits.

👉 The Task: 36 + 59

What is trivial for us becomes a labyrinthine process inside a Transformer, with two parallel paths – one roughly estimates, the other tries to get the last digit correct.

📌 The Result? It Works – Somehow.

But not because the Transformer understands it. Rather, because we beat it into the right direction with billions of FLOPs.

What the Diagram Makes Clear

• ✅ Transformers can solve math problems
• ❌ But they don’t want to – the architecture practically resists it

💥 A Paradigm Shift Is Needed

This is massive evidence that we need a paradigm shift. Math is not optional for intelligent systems. And if an architecture requires this much energy to “simulate” simple arithmetic rules, that is a warning sign.

🔍 Mechanistic Interpretability

Mechanistic interpretability shows us not only how Transformers “think” – but also how little they truly comprehend. More on this can be found at mechinterp.com.

It is time to think about new architectures. Truly new ones. Not just bigger models.