Build a Large Language Model (From Scratch)

← Back to Library

copyright
contents
Build a Large Language Model (From Scratch)
preface
acknowledgments
about this book
about the author
about the cover illustration
1 Understanding large language models
- 1.1 What is an LLM?
- 1.2 Applications of LLMs
- 1.3 Stages of building and using LLMs
- 1.4 Introducing the transformer architecture
- 1.5 Utilizing large datasets
- 1.6 A closer look at the GPT architecture
- 1.7 Building a large language model
2 Working with text data
- 2.1 Understanding word embeddings
- 2.2 Tokenizing text
- 2.3 Converting tokens into token IDs
- 2.4 Adding special context tokens
- 2.5 Byte pair encoding
- 2.6 Data sampling with a sliding window
- 2.7 Creating token embeddings
- 2.8 Encoding word positions
3 Coding attention mechanisms
- 3.1 The problem with modeling long sequences
- 3.2 Capturing data dependencies with attention mechanisms
- 3.3 Attending to different parts of the input with self-attention
  - 3.3.1 A simple self-attention mechanism without trainable weights
  - 3.3.2 Computing attention weights for all input tokens
- 3.4 Implementing self-attention with trainable weights
  - 3.4.1 Computing the attention weights step by step
  - 3.4.2 Implementing a compact self-attention Python class
- 3.5 Hiding future words with causal attention
- 3.6 Extending single-head attention to multi-head attention
  - 3.6.1 Stacking multiple single-head attention layers
  - 3.6.2 Implementing multi-head attention with weight splits
4 Implementing a GPT model from scratch to generate text
- 4.1 Coding an LLM architecture
- 4.2 Normalizing activations with layer normalization
- 4.3 Implementing a feed forward network with GELU activations
- 4.4 Adding shortcut connections
- 4.5 Connecting attention and linear layers in a transformer block
- 4.6 Coding the GPT model
- 4.7 Generating text
5 Pretraining on unlabeled data
- 5.1 Evaluating generative text models
- 5.2 Training an LLM
- 5.3 Decoding strategies to control randomness
- 5.4 Loading and saving model weights in PyTorch
- 5.5 Loading pretrained weights from OpenAI
6 Fine-tuning for classification
- 6.1 Different categories of fine-tuning
- 6.2 Preparing the dataset
- 6.3 Creating data loaders
- 6.4 Initializing a model with pretrained weights
- 6.5 Adding a classification head
- 6.6 Calculating the classification loss and accuracy
- 6.7 Fine-tuning the model on supervised data
- 6.8 Using the LLM as a spam classifier
7 Fine-tuning to follow instructions
- 7.1 Introduction to instruction fine-tuning
- 7.2 Preparing a dataset for supervised instruction fine-tuning
- 7.3 Organizing data into training batches
- 7.4 Creating data loaders for an instruction dataset
- 7.5 Loading a pretrained LLM
- 7.6 Fine-tuning the LLM on instruction data
- 7.7 Extracting and saving responses
- 7.8 Evaluating the fine-tuned LLM
- 7.9 Conclusions
appendix A Introduction to PyTorch
- A.1 What is PyTorch?
- A.2 Understanding tensors
- A.3 Seeing models as computation graphs
- A.4 Automatic differentiation made easy
- A.5 Implementing multilayer neural networks
- A.6 Setting up efficient data loaders
- A.7 A typical training loop
- A.8 Saving and loading models
- A.9 Optimizing training performance with GPUs
appendix B References and further reading
appendix C Exercise solutions
appendix D Adding bells and whistles to the training loop
- D.1 Learning rate warmup
- D.2 Cosine decay
- D.3 Gradient clipping
- D.4 The modified training function
appendix E Parameter-efficient fine-tuning with LoRA
- E.1 Introduction to LoRA
- E.2 Preparing the dataset
- E.3 Initializing the model
- E.4 Parameter-efficient fine-tuning with LoRA

copyright

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact

Special Sales Department

Manning Publications Co.

20 Baldwin Road

PO Box 761

Shelter Island, NY 11964

Email: orders@manning.com

©2025 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.

Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.

The authors and publisher have made every effort to ensure that the information in this book was correct at press time. The authors and publisher do not assume and hereby disclaim any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from negligence, accident, or any other cause, or from any usage of the information herein.

Manning Publications Co.
20 Baldwin Road
PO Box 761
Shelter Island, NY 11964

Development editor: Dustin Archibald
Technical editor: David Caswell
Review editor: Kishor Rit
Production editor: Aleksandar Dragosavljević
Copy editors: Kari Lucke and Alisa Larson
Proofreader: Mike Beady
Technical proofreader: Jerry Kuch
Typesetter: Dennis Dalinnik
Cover designer: Marija Tudor

ISBN: 9781633437166

Printed in the United States of America

← Previous Section 3 of 22 Next →