Transformer Expressive Power & Turing Limit

The Expressive Power, Circuit Complexity & Turing Completeness of Transformers ($TC^0$ & RASP formalisms) establishes the mathematical boundaries of what self-attention can and cannot compute; the foundational theoretical landscape establishes: 1) Constant-Depth Transformer Circuit Limits: fixed-depth Transformers belong to the $TC^0$ complexity class (constant-depth threshold circuits), meaning they are theoretically incapable of solving non-$TC^0$ problems (e.g. Parity checks, graph connectivity, long-horizon dynamic programming) in a single feed-forward pass; 2) Turing Completeness via CoT: when augmented with Chain-of-Thought (autoregressively emitting unbounded reasoning tokens that act as external Turing tape memory), Transformer expressivity jumps to universal Turing Completeness; 3) RASP programming language formalizes attention as relational sequence primitives.

AI research scientist theoretical expressivity rounds, proving the mathematical necessity of Chain-of-Thought, and circuit complexity lower bounds.

Formally proves why single-pass next-token generation hits hard computational barriers and why Chain-of-Thought is mathematically necessary to unlock universal computation.

Prove why fixed-depth Transformers are contained in $TC^0$ and cannot compute $N$-bit Parity in a single forward pass without CoT?

Prove that autoregressive Transformers with variable-length Chain-of-Thought generation are Turing Complete?

Contrast Hard vs Soft Attention in formal language theory across Regular Languages (DFA) and Context-Free Grammars (PDA)?

Explain the theoretical equivalence between Self-Attention and fully connected Graph Neural Network Message Passing with dynamic adjacency kernels?

Compare State Space Models (Mamba) vs Transformers in theoretical associative recall capacity limits?