An open-source implementation for the pre-training architecture of Google's LaMDA in PyTorch. The research paper outlines an autoregressive, decoder-only, GPT-like transformer language model. The transformer uses T5 relative positional bias in the attention layers and gated-GELU activation function in the feed-forward layers.
The repository currently contains a script for basic training as well as Huggingface datasets and Weights & Biases integration.
LaMDA research paper: https://arxiv.org/abs/2201.08239
Github repository for the model: https://github.com/conceptofmind/LaMDA-pytorch
The pre-training architecture was peer-reviewed by Dr. Phil Wang. Please check out and support his work: https://github.com/lucidrains.
Updates: https://twitter.com/EnricoShippole