index

SYMBOLS

124M parameter

\[EOS] (end of sequence) token

.reshape method, 2nd

.to() method, 2nd

.weight attribute

.eval() mode

__getitem__ method

\[PAD] (padding) token

.T method

.backward() method, 2nd

%timeit command

.matmul method

04_preference-tuning-with-dpo folder

355M parameter

\[BOS] (beginning of sequence) token

\<|unk|> tokens, 2nd, 3rd, 4th, 5th

.view method, 2nd

__init__ constructor, 2nd, 3rd

.shape attribute

@ operator

__len__ method

\<|endoftext|> token

.pth extension

<i>Dolma\

An Open Corpus of Three Trillion Tokens for LLM Pretraining Research</> (Soldaini et al.)

== comparison operator

A

arXiv

Alpaca dataset, 2nd

argmax function, 2nd, 3rd, 4th, 5th, 6th, 7th

attention mechanisms

coding, 2nd

problem with modeling long sequences

attention scores

AI (artificial intelligence)

autograd engine

alpha scaling factor

autoregressive model

attention weights, computing step by step, 2nd

attn_scores

Axolotl

allowed_max_length

AdamW optimizer, 2nd

B

Bahdanau attention mechanism

backpropagation

BERT (bidirectional encoder representations from transformers)

BPE (byte pair encoding)

batch_size

C

compute_accuracy function, 2nd

causal attention mask

clip_grad_norm_ function

calc_loss_loader function

cross_entropy function, 2nd, 3rd

conversational performance

custom_collate_draft_1

custom_collate_draft_2

calc_accuracy_loader function

calc_loss_batch function, 2nd, 3rd

classification

tasks

custom_collate_fn function, 2nd

classify_review function

context_length

cfg dictionary

computing gradients

context vectors, 2nd, 3rd

CausalAttention class, 2nd

D

DistributedSampler

dim parameter, 2nd

Dataset class, 2nd, 3rd, 4th, 5th, 6th, 7th

DataLoader class, 2nd

datasets

downloading

download_and_load_gpt2 function, 2nd, 3rd

DummyGPTClass

DistributedDataParallel class

DummyLayerNorm, 2nd

placeholder

DummyGPTModel, 2nd, 3rd, 4th

deep learning

dot products

DDP (DistributedDataParallel) strategy

device variable

decode method, 2nd

data loaders, 2nd

code for

dropout

defined

drop_rate

drop_last parameter

DummyTransformerBlock

data list

ddp_setup function

d_out argument, 2nd

DataFrame

E

eps variable

evaluate_model function, 2nd, 3rd, 4th

embedding size

emergent behavior

encoder

encode method, 2nd, 3rd

emb_dim

eval_iter value

F

find_highest_gradient function

first_batch variable

FeedForward module, 2nd, 3rd, 4th

format_input function, 2nd, 3rd, 4th, 5th, 6th

fine-tuning

LLMs, to follow instructions

categories of

for classification

forward method, 2nd

G

generate_and_print_sample function

GELU (Gaussian error linear unit)

activation function, 2nd

GPTModel, 2nd, 3rd, 4th, 5th, 6th

class, 2nd

code

instance, 2nd, 3rd, 4th

GPT (Generative Pre-trained Transformer)

architecture

coding, 2nd

implementing from scratch to generate text

grad_fn value

gpt_download.py Python module

GPT_CONFIG_124M dictionary, 2nd, 3rd, 4th, 5th, 6th

generative text models, evaluating

GenAI (generative AI)

gpt2-medium355M-sft.pth file

GPTDatasetV1 class, 2nd, 3rd

generate_text_simple function, 2nd, 3rd, 4th, 5th, 6th

GPT-4

GPT-2

model

tokenizer

GPT-3

generate_model_scores function, 2nd

Google Colab

generate function, 2nd, 3rd, 4th, 5th, 6th, 7th

I

init_process_group function

instruction dataset

information leakage

input_embeddings

InstructionDataset class, 2nd

instruction fine-tuning

instruction following, creating data loaders for instruction dataset, 2nd

overview

’instruction’ object

K

keepdim parameter

L

logits tensor

LinearWithLoRA layer, 2nd

LoRALayer class, 2nd

loss metric

LLMs (large language models), 2nd

applications of

building and using, 2nd, 3rd

coding architecture

coding attention mechanisms, causal attention mechanism, 2nd

fine-tuning, 2nd, 3rd

fine-tuning for classification, 2nd, 3rd, 4th, 5th

instruction fine-tuning, loading pretrained LLMs, 2nd

overview of, 2nd, 3rd

utilizing large datasets

Linear layers, 2nd

LayerNorm, 2nd, 3rd

LIMA dataset

layer normalization, 2nd

load_state_dict method

load_weights_into_gpt function, 2nd, 3rd, 4th

loss.backward() function

Linear layer weights

Llama 3 model

LLama 2 model

LoRA (low-rank adaptation), 2nd, 3rd

parameter-efficient fine-tuning, 2nd

M

main function

max_length, 2nd, 3rd

model.eval() function

MultiHeadAttention class, 2nd, 3rd, 4th, 5th, 6th

model.train() setting

MultiHeadAttentionWrapper class, 2nd, 3rd, 4th, 5th, 6th, 7th

machine learning

multi-head attention, 2nd

Module base class

multiprocessing submodule

masked attention

multinomial function, 2nd, 3rd

macOS

model_response

minbpe repository

model_configs table

mps device

N

NEW_CONFIG dictionary

neural networks

implementing feed forward network with GELU activations, 2nd

nn.Linear layers

n_heads

numel() method

num_heads dimension

O

output layer nodes

ollama run llama3 command, 2nd, 3rd

ollama serve command, 2nd, 3rd, 4th

optimizer.zero_grad() method

Ollama application, 2nd

Ollama Llama 3 method

ollama run command

P

PyTorch

and Torch

automatic differentiation, 2nd

computation graphs

data loaders

dataset objects

efficient data loaders

implementing multilayer neural networks, 2nd

installing, 2nd

loading and saving model weights in

optimizing training performance with GPUs

overview, 2nd

with a NumPy-like API

pip installer

Phi-3 model

print_gradients function, 2nd

plot_values function

parameters

calculating

perplexity

partial derivatives

print statement

plot_losses function

Python version

Prometheus model

prompt styles

pretraining

calculating training and validation set losses

on unlabeled data

training LLMs, 2nd

print_sampled_tokens function, 2nd

pos_embeddings, 2nd

preference fine-tuning

Q

qkv_bias

query_llama function

query_model function

R

responses, extracting and saving, 2nd

re library

RMSNorm

ReLU (rectified linear unit), 2nd

re.split command

replace_linear_with_lora function

raw text

retrieval-augmented generation

RNNs (recurrent neural networks)

random_split function

S

shortcut connections, 2nd

saving and loading models

SimpleTokenizerV1 class, 2nd

spawn function

Sequential class

SelfAttention_v2 class, 2nd, 3rd

softmax_naive function, 2nd

sci_mode parameter

set_printoptions method

SGD (stochastic gradient descent)

SelfAttention_v1 class, 2nd

softmax function, 2nd, 3rd

self.register_buffer() call

state_dict, 2nd

SpamDataset class, 2nd, 3rd

special context tokens

stride setting

self.out_proj layer

supervised learning

supervised data, fine-tuning model on

strip() function

supervised instruction fine-tuning

preparing dataset for, 2nd

settings dictionary, 2nd

self-attention mechanism

computing attention weights for all input tokens, 2nd

implementing with trainable weights, 2nd

without trainable weights, 2nd

single-head attention, stacking multiple layers

SimpleTokenizerV2 class, 2nd

T

text generation function, modifying

train_ratio

text data

adding special context tokens, 2nd

creating token embeddings

sliding window, 2nd

tokenization, byte pair encoding, 2nd

torch.save function

token IDs, 2nd

tensor library

TransformerBlock class

token_embedding_layer, 2nd

token embeddings, 2nd

train_simple_function

ToyDataset class, 2nd

training function

enhancing

modified, 2nd

train_data subset

tril function

tokenizing text, 2nd

training, optimizing performance with GPUs

PyTorch computations on GPU devices

selecting available GPUs on multi-GPU machine, 2nd

single-GPU training

training with multiple GPUs

test_loader

train_loader

torch.sum method

training loops, 2nd

cosine decay

gradient clipping

learning rate warmup

train_classifier_simple function, 2nd

training batches, organizing data into, 2nd

text generation

using GPT to generate text

top-k sampling, 2nd

text_data

transformer architecture, 2nd, 3rd, 4th

temperature scaling, 2nd

train_model_simple function, 2nd, 3rd, 4th, 5th, 6th

tensor2d

tensor3d

torch.no_grad() context manager

test_set dictionary, 2nd

tensors

common tensor operations

tensor data types

torch.nn.Linear layers

transformer blocks, 2nd

connecting attention and linear layers in, 2nd

text generation loss

torchvision library

U

unbiased parameter

unlabeled data, decoding strategies to control randomness

V

variable-length inputs

vocab_size

v vector

vectors, 2nd

W

W<.Subscript>q</> matrix

weight parameters, 2nd

word embeddings, 2nd

weight_decay parameter

Word2Vec

weights

initializing model with pretrained weights

loading pretrained weights from OpenAI, 2nd

word positions, encoding

weight splits

X

X training example