Keep pulling the thread on Noam Shazeer.
Pathways Language Model (PaLM) is a 540-billion parameter, densely activated, Transformer language model.
The Pathways Language Model (PaLM) was trained on 6144 TPU v4 chips.
The Pathways Language Model (PaLM) achieves state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks.
The 540-billion parameter version of PaLM outperforms the finetuned state-of-the-art on a suite of multi-step reasoning tasks.
The 540-billion parameter version of PaLM outperforms average human performance on the BIG-bench benchmark.
A significant number of BIG-bench tasks showed discontinuous improvements from model scale, with performance steeply increasing at the largest model size.
The Pathways Language Model (PaLM) was trained using Pathways, a new ML system that enables highly efficient training across multiple TPU Pods.
The Pathways Language Model (PaLM) has strong capabilities in multilingual tasks and source code generation.