Predictable Scaling and Infrastructure

•What ∘Collect a dataset of text, code, images, audio, math. ∘Pre-train a model to predict the next word. ∘Post-train it to e.g. follow instructions, be conversational, use tools. •How ∘Synchronous SGD of a transformer on a large cluster of accelerators. ∘Combine many forms of data and model parallelism. ∘Reinforcement learning from human feedback.