“Attention is all you need” is the title of a paper¹ published in 2017, introducing the new deep neural network Transformer model architecture, which has been in fact “transformational” in the booming field of Generative AI. It is the architecture now used in natural language, computer vision, audio and multi-modal processing, and one of the pillars of Foundation Models.

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

The paper is considered the founding document for modern artificial intelligence, with presently more than 100.000 cites. It is one of those papers that you feel you are now obliged to cite in an academic paper, just to signal your referees you are “into AI”. The objective of this post, however, is not to talk about Artificial Intelligence, but just to emphasize an exceptional example of how innovation actually happens: knowledge spillovers and the diffusion of ideas.

The paper was signed by eight authors working all of them for Google (Google Brain and Google Research). Today, seven years later, all of them except one (Łukasz Kaiser, working for Open AI) are (co)founders and working on six new startups which has collectively raised no least than $2B².

The phenomenon has not escaped the attention of techno, financial and generalist media.

The tittle of the paper is a reference to the attention mechanisms used in machine learning resembling cognitive attention, but in a world more competitive than ever, and flooded with an information overload, it is at the same time a metaphor of a fundamental ingredient in the recipe for personal and entrepreneurial success.

Forget about love. Attention is for sure a necessary condition. One may wonder whether it is also sufficient. Surely not, but pay attention just in case.

____________________

(1) Ashish Vaswani et al., ‘Attention Is All You Need’, in Advances in Neural Information Processing Systems, vol. 30 (Curran Associates, Inc., 2017), https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.

(2) Data by Crunchbase, Feb. 18, 2024

Featured Images: Based on the LinkedIn (or X Twitter) Profiles of the eight paper’s authors