Einstein Notation: A New Lens on Transformers | by Dr. Christoph Mittendorf | Nov, 2024

Transforming the Math of the Transformer Model

In this article, we’ll embark on a playful journey through the world of transformers, unraveling the complexities of their architecture using the Einstein notation.

Introduction:

Transformer models have revolutionized the field of natural language processing (and beyond), achieving state-of-the-art results on a variety of tasks. They have impressive performance but the underlying mathematical operations can be complex and difficult to grasp — especially without breaking down the individual layers. In this article, I propose using the Einstein notation to express the mathematical operations within a transformer model.

Note that the Einstein notation is normally used in Physics and Mathematics such as in General Relativity, Electromagnetism, Quantum and Fluid Mechanics but also in Linear Algebra to represent matrix operations in a more compact form.

The goal is to write the mathematical operations of every layer in a concise and elegant way. By leveraging implicit summation over repeated indices, Einstein notation can simplify the representation of tensor operations, making it (potentially) easier to understand and therefore implement the individual layers of the transformer models…