FUTURIS
MAPPULSELOCATORRADAR
RUEN
Telegram X
To the line
Capabilities on the main line · TRANSFORMER

Transfer across tasks

In one viewIn 2017, the Transformer introduced an architecture that connects data elements through attention and trains efficiently on parallel hardware.

The same principle proved useful for text, code, images, audio and video. Transformers became the shared foundation of nearly every modern large model.

StatusPASSED
TypeCapabilities on the main line
Marker2017
Events in dossier9
Development chronology

Researched

2017-06

Attention Is All You Need

About this eventThe Transformer replaced recurrence with attention and scaled better on parallel compute.

verifiedPermanent page

The Transformer removed the sequential pass used by recurrent networks and related input elements through attention. That made training far more parallel and created a common architecture later transferred from language to code, images, audio and video. The paper became the central fork in modern AI history: nearly every later large language and multimodal model descends from it.

Source: arXiv
2018-06

GPT

About this eventPretraining a transformer on a large corpus and then adapting it to a task became a repeatable transfer-learning recipe.

verifiedPermanent page
Source: OpenAI
2019-10

T5

About this eventTranslation, classification and question answering were reduced to one format: text in, text out.

verifiedPermanent page
Source: arXiv
2020-10

Vision Transformer

About this eventThe same architecture works on images — the transformer turns out to be universal.

verifiedPermanent page
Source: arXiv
2021-01

Mixture of experts

About this eventSparse models: more parameters without a proportional compute increase.

verifiedPermanent page
Source: arXiv

In progress

сейчас

Fighting the quadratic cost

About this eventAttention costs grow as the square of the input length. Linear and hybrid schemes are the main engineering work here.

NEWPermanent page

Planned

впереди

A successor architecture

About this eventThe design has held for nine years. Every major lab is looking for a replacement, but no alternative has yet won at scale.

NEWPermanent page
Sources and research

Primary material behind this dossier: papers, lab publications and official reports.

arXiv · 2017-06Attention Is All You NeedOpenAI · 2018-06GPTarXiv · 2018-10BERTOpenAI · 2019-02GPT-2arXiv · 2019-10T5arXiv · 2020-10Vision TransformerarXiv · 2021-01Mixture of experts
Capabilities on the main line
Machine perception · 2012 Learning from examples in context · 2020