Development became more engineerable: teams could estimate training budgets and expected results. This started the accelerator and data-centre race, but it does not guarantee every new capability.
About this eventIn-context learning: a couple of examples is enough.
GPT-3 showed that one large pretrained model could switch between tasks from an instruction and a handful of examples in its input. The weights do not change: the adaptation happens inside context, turning a general language interface into a practical way to program model behaviour.
Source: OpenAIAbout this eventPerformance was linked by power laws to model size, data and compute.
About this eventData beats size — every training budget gets rebuilt.
About this eventSome skills appear in a jump past a scale threshold rather than growing smoothly.
About this eventHigh-quality human text on the internet is close to exhausted. Next come synthetic data, video and companies' own corpora.
About this eventThe next growth axis is not model size but the volume of training on the model's own attempts and mistakes.
Primary material behind this dossier: papers, lab publications and official reports.