祝融说。

FORM NOT VOID, MIND NO CORE

Chapter 5: The Door to Deep Learning: From Neural Networks to PyTorch in Practice

2026.08.10

Welcome to the second part of this book. In "Part 1: Foundation," we forged a solid suit of armor for ourselves: we mastered Python programming, wielded the tools of data science, and gained the ability to engineer and deploy applications. Now, fully equipped, we are about to embark on a more exciting journey -- to deeply explore the core engine driving the modern AI revolution: deep learning.

If traditional machine learning, as represented by Scikit-learn, is like a "handicraft workshop" built on exquisite mathematics and statistics, then deep learning is more like a "modern factory" capable of automatically learning and extracting features. By simulating the structure of neural networks in the human brain, it constructs deep neural networks (DNNs) composed of tens of thousands or even billions of interconnected "neurons," achieving unprecedented breakthroughs in fields such as image recognition, speech recognition, and natural language processing.

The large language models (LLMs) we marvel at today are the zenith of deep learning's development so far. To truly understand and harness LLMs, we must return to their source, understanding their most basic building blocks and working principles. This chapter is the key to opening the "door to deep learning."

We will start from the "first principles" of deep learning:

  • Neural Networks and Backpropagation: We will use the most intuitive approach to reveal how neural networks transmit information and learn. You will understand that the seemingly mysterious "learning" process is essentially an elegant "credit assignment" game based on calculus, called backpropagation.
  • Deep Dive into the PyTorch Framework: Theory is dry, but code is alive. We will deeply study PyTorch, the most popular deep learning framework in academia and industry today. You will master its three core pillars: Tensor (multidimensional arrays), Autograd (automatic differentiation engine), and nn.Module (the building block for neural networks).
  • Analysis of Classic Network Architectures: Before the Transformer architecture unified the field, CNNs (Convolutional Neural Networks) and RNNs (Recurrent Neural Networks) reigned supreme in their respective domains. Understanding their design ideas is crucial for grasping core concepts like feature extraction and sequence modeling, as well as the evolutionary context that gave rise to the Transformer.
  • The Art of Training: Building a network is only the first step; making it "learn well" is an art. We will explore the three core elements: loss functions (telling the model where it went wrong), optimizers (guiding the model on how to improve), and regularization (preventing the model from "rote memorization").

Finally, through a classic NLP hands-on project -- implementing a text sentiment classifier using PyTorch -- we will tie together all the theories and techniques in this chapter. We will start from scratch, define the dataset, build the model, write the training loop, and ultimately obtain an AI model that can determine whether a movie review is positive or negative.

This chapter is a crucial step in your transformation from an "AI user" to an "AI builder." It will lay the most solid and profound theoretical and practical foundation for your subsequent study of the Transformer architecture and mastery of large language models. Now, let us ignite the furnace of "computation" and embark on this journey of "alchemy" filled with challenges and creation.

5.1 The Core Idea of Neural Networks and Backpropagation

5.1.1 From Biology to Mathematics: The Abstraction of a Neuron

Deep learning draws inspiration from the biological brain. A biological neuron receives electrical signals (inputs) from other neurons. When the cumulative strength of these signals exceeds a certain threshold, the neuron is "activated" and sends signals (outputs) to other neurons.

Mathematicians and computer scientists abstracted this process into a simple mathematical model -- the artificial neuron (or perceptron):

  1. Inputs and Weights: The neuron receives multiple input values x1, x2, ..., xn. Each input is associated with a weight w1, w2, ..., wn, which represents the importance of that input.
  2. Weighted Sum: The neuron multiplies each input by its corresponding weight, sums them, and adds a bias b. The result z = (w1*x1 + w2*x2 + ... + wn*xn) + b can be concisely expressed mathematically as a vector dot product: z = w . x + b.
  3. Activation Function: The weighted sum z is passed through a non-linear activation function f to produce the final output a = f(z).

Why do we need an activation function?

The non-linearity of the activation function is the key to the entire neural network's ability to learn complex patterns. Without an activation function (or if the activation function is linear), no matter how many layers of neurons you stack together, the entire network is essentially just a simple linear model, incapable of learning complex non-linear relationships like image recognition or language understanding.

Common activation functions:

Sigmoid: f(z) = 1 / (1 + e^(-z)). Compresses the input to a range between 0 and 1. Commonly used in the output layer of binary classification problems to represent probabilities.

ReLU (Rectified Linear Unit): f(z) = max(0, z). Simple to compute and effectively mitigates the vanishing gradient problem. One of the most commonly used activation functions today.

5.1.2 Neural Networks: From Single Neurons to Interconnected Layers

A single neuron has limited capability. But when we organize a large number of neurons into layers and connect these layers together, we form a neural network.

Input Layer: Receives the most raw data, such as pixel values of an image or word vectors of a sentence.

Hidden Layers: Located between the input layer and the output layer, responsible for most of the computation and feature extraction. A neural network can have zero or more hidden layers. When there is one or more hidden layers, we call it a deep neural network.

Output Layer: Produces the final prediction result. For example, in a cat vs. dog classification task, the output layer might have two neurons, representing the "probability of being a cat" and the "probability of being a dog."

The process of information flowing from the input layer through the network to the output layer is called forward propagation. The output of each layer serves as the input to the next layer.

5.1.3 The Essence of "Learning": Backpropagation and Gradient Descent

We have built a network, but its weights w and biases b are initially randomized. Such an untrained network gives random, incorrect predictions for any input. So, how does a network "learn"?

The learning process is essentially a parameter optimization problem. We seek to find the optimal set of w and b such that the network's output for any given input is as close as possible to the true label.

This process can be divided into three steps:

Step 1: Define the Loss

We need a quantitative metric to measure "how bad" the model's prediction is. This metric is the loss function. For example, in classification tasks, a commonly used loss function is cross-entropy loss. The larger the loss value, the worse the model's prediction. Our goal is to adjust the parameters to make the loss value as small as possible.

Step 2: Compute the Gradient

The loss L is a function of all the weights w and biases b. Calculus tells us that the gradient of a function at a point points in the direction of the steepest increase of that function. Therefore, the opposite direction of the gradient is the direction of the steepest decrease.

We want to make the loss L smaller, so we need to compute the partial derivative of L with respect to each parameter (e.g., w_ij, representing a specific weight of the j-th neuron in the i-th layer). The vector containing all these partial derivatives is the gradient of the loss function with respect to the parameters.

How do we efficiently compute this gradient? This is the core of the backpropagation algorithm. It uses the chain rule from calculus to compute gradients layer by layer, starting from the output layer.

  1. First, compute the gradient of the loss with respect to the output layer's activations.
  2. Then, use this gradient to compute the gradient with respect to the output layer's weighted sum.
  3. Then, use this gradient to compute the gradients of the weights and biases connected to the output layer, as well as the gradient of the previous layer's activations.
  4. ... And so on, until the gradients of all parameters have been computed.

Backpropagation is an extremely clever algorithm that avoids a large amount of redundant computation, making it feasible to compute gradients in deep networks. Fortunately, in modern deep learning frameworks, we do not need to implement it manually.

Step 3: Update the Parameters (Gradient Descent)

Once we have the gradients, we can update the parameters. The simplest update rule is gradient descent:

new_w = old_w - learning_rate * L/w

learning_rate is a hyperparameter that controls the "step size" of each parameter update.

We take a step in the opposite direction of the gradient (indicated by the - sign) to update the weights.

This cycle of "forward propagation -> compute loss -> backpropagation -> update parameters" is the core of neural network training. We repeatedly feed the entire dataset (or a small batch of data) into this cycle for iterations. With each iteration, the model's parameters are finely adjusted in the direction that makes the loss smaller. After thousands or tens of thousands of iterations, the model gradually "learns" how to make accurate predictions.

5.2 Deep Dive into PyTorch: Tensor, Autograd, and nn.Module

PyTorch is an open-source deep learning framework introduced by Facebook AI Research (FAIR). It is beloved by academia and researchers for its Pythonic design, flexibility, and powerful dynamic computational graph mechanism, and is gradually dominating the industrial landscape.

5.2.1 Tensor: A GPU-Accelerated Multidimensional Array

PyTorch's Tensor is conceptually very similar to NumPy's ndarray, but can be thought of as a "super-powered version."

It is a multidimensional array, the basic unit of data flow in PyTorch.

Core advantage: Tensor can be seamlessly moved to the GPU for computation, leveraging the GPU's powerful parallel computing capabilities to accelerate training.

import torch

# --- Creating Tensors ---
# From a list
x_list = [[1, 2], [3, 4]]
x_tensor = torch.tensor(x_list)
print(x_tensor)

# Similar to NumPy creation methods
x_zeros = torch.zeros(2, 3)
x_rand = torch.rand(2, 3)
print(x_zeros)
print(x_rand)

# --- Tensor Properties ---
print(f"Shape of tensor: {x_tensor.shape}")
print(f"Datatype of tensor: {x_tensor.dtype}")
print(f"Device tensor is stored on: {x_tensor.device}")

# --- GPU Acceleration ---
# Check if a GPU is available
if torch.cuda.is_available():
    device = torch.device("cuda")
    print(f"GPU is available! Using device: {device}")
  
    # Move the Tensor to the GPU
    x_gpu = x_tensor.to(device)
    print(f"x_gpu is on device: {x_gpu.device}")
  
    # Perform computation on the GPU
    y_gpu = x_gpu + x_gpu
  
    # Move the result back to the CPU (e.g., for printing or interacting with NumPy)
    y_cpu = y_gpu.to("cpu")
else:
    device = torch.device("cpu")
    print("GPU not available, using CPU.")

# --- Interoperability with NumPy ---
# Tensor -> NumPy
np_array = x_tensor.numpy()
print(f"Numpy array: \n {np_array}")

# NumPy -> Tensor
np_array_new = np.ones((2, 2))
tensor_from_np = torch.from_numpy(np_array_new)
print(f"Tensor from numpy: \n {tensor_from_np}")

This seamless conversion between NumPy and PyTorch allows us to easily switch between data preprocessing (commonly done with NumPy/Pandas) and model computation (using PyTorch Tensors).

5.2.2 Autograd: The Magic Automatic Differentiation Engine

This is PyTorch's most core and magical feature. It saves us the pain of manually implementing backpropagation. Autograd silently records all operations performed on Tensors, building a dynamic computational graph.

When a Tensor's .requires_grad attribute is set to True, Autograd starts tracking it.

# Create Tensors that need gradients
w = torch.tensor([[2.0], [3.0]], requires_grad=True)
b = torch.tensor(1.0, requires_grad=True)

# Define input
x = torch.tensor([[1.0, 2.0]])

# Forward propagation
# y = x @ w + b  (@ is matrix multiplication shorthand)
z = x @ w + b
loss = torch.sum(z) # Assume the loss is the sum of z

print(f"z: {z}")
print(f"loss: {loss}")

# --- Backpropagation ---
# Calling .backward(), Autograd automatically computes the gradient of loss
# with respect to all Tensors that have requires_grad=True
loss.backward()

# --- Viewing Gradients ---
# Gradients are accumulated in the .grad attribute
print(f"Gradient of w: \n {w.grad}")
print(f"Gradient of b: {b.grad}")

What happened?

  1. Forward propagation z = x @ w + b, where x is [[1, 2]], w is [[2], [3]]. x @ w = 1*2 + 2*3 = 8. z = 8 + 1 = 9. loss = 9.
  2. loss.backward() is called.
  3. Autograd starts computing gradients backward: loss/z = 1 z/w = x^T = [[1], [2]] (chain rule) loss/w = loss/z * z/w = 1 * [[1], [2]] = [[1], [2]] z/b = 1 loss/b = loss/z * z/b = 1 * 1 = 1
  4. The computed gradients match w.grad and b.grad exactly!

Important Notes:

Gradients accumulate. You need to manually zero the gradients before each parameter update: optimizer.zero_grad().

Only Tensors of floating-point types can compute gradients.

During model evaluation (inference), we do not need gradients. We should use the with torch.no_grad(): context manager to disable Autograd, which saves memory and speeds up computation.

5.2.3 nn.Module: The "Lego Bricks" for Building Neural Networks

torch.nn is the PyTorch module specifically designed for building neural networks. All network layers, loss functions, and activation functions reside here. nn.Module is the base class for all neural network modules.

To build your own neural network, you typically need to:

  1. Create a class that inherits from nn.Module.
  2. In the __init__ method, define the various layers the network needs (such as convolutional layers, linear layers). These layers are themselves subclasses of nn.Module.
  3. In the forward method, define how data flows through these layers during forward propagation.
import torch.nn as nn
import torch.nn.functional as F

class SimpleNet(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        # Must call the parent class's __init__ method
        super(SimpleNet, self).__init__()
      
        # --- Define Network Layers ---
        # Linear layer (y = Wx + b)
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        # --- Define Forward Propagation Logic ---
        # x -> fc1 -> relu -> fc2 -> output
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

# --- Using Our Defined Network ---
input_dim = 784 # e.g., a flattened 28x28 image
hidden_dim = 128
output_dim = 10 # e.g., 10 classes for digits 0-9

# Instantiate the model
model = SimpleNet(input_dim, hidden_dim, output_dim)
print(model)

# We can call the model instance like a function; it automatically executes the forward method
dummy_input = torch.randn(64, input_dim) # Simulate a batch of data
output = model(dummy_input)
print(f"Output shape: {output.shape}") # torch.Size([64, 10])

By inheriting from nn.Module, our SimpleNet class automatically gains many powerful features, such as:

model.parameters(): Provides easy access to all trainable parameters (weights and biases) in the model.

model.to(device): Moves the entire model and all its parameters to the GPU with a single command.

model.train() / model.eval(): Toggles between training and evaluation modes (important for layers like Dropout and BatchNorm).

5.3 Classic Network Architectures: CNNs and RNNs/LSTMs

Before the Transformer emerged, CNNs and RNNs were the two mainstays of deep learning's success in computer vision (CV) and natural language processing (NLP).

5.3.1 Convolutional Neural Networks (CNNs): Capturers of Image Features

Core Idea:

Traditional fully connected networks, when processing images, flatten the image into a long vector, losing the spatial structure information of pixels. CNNs, through convolution and pooling operations, are specifically designed to process data with a grid-like structure (such as images).

  1. Convolutional Layer (nn.Conv2d): It uses a small kernel (or filter) (e.g., 3x3 or 5x5) that slides across the input image. At each position, the kernel performs element-wise multiplication and summation with the image region it covers, producing an output value. This process can be understood as extracting local features. For instance, one kernel might be sensitive to vertical edges in the image, while another might be sensitive to a particular color or texture.

    Parameter Sharing: The same kernel shares the same set of weights across the entire image, which significantly reduces the number of model parameters and gives the network translation invariance (a cat in the top-left corner or the bottom-right corner of the image can still be recognized).

  2. Pooling Layer (nn.MaxPool2d): Also called a downsampling layer. It takes the maximum value (Max Pooling) or average value (Average Pooling) within a region (e.g., 2x2) as its output.

    Function: Reduces the spatial size of the feature map, thereby reducing the computation and parameters in subsequent layers. Provides a degree of translation and rotation invariance, making the model more robust.

A typical CNN structure is usually a repeating stack of convolutional layer -> activation function -> pooling layer, followed by a few fully connected layers for classification.

class SimpleCNN(nn.Module):
    def __init__(self):
        super(SimpleCNN, self).__init__()
        # Input: 1 channel (grayscale), Output: 16 channels, Kernel: 3x3, padding=1
        self.conv1 = nn.Conv2d(1, 16, kernel_size=3, stride=1, padding=1)
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=3, stride=1, padding=1)
        # Assuming input is a 28x28 image, after two poolings, the size becomes 7x7
        # 32 channels * 7 * 7 = 1568
        self.fc1 = nn.Linear(32 * 7 * 7, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        # Flatten the feature map
        x = x.view(-1, 32 * 7 * 7)
        x = self.fc1(x)
        return x

5.3.2 Recurrent Neural Networks (RNNs/LSTMs): The Memory of Sequence Data

Core Idea:

For data with sequential order, such as text, speech, and time series, CNNs and standard fully connected networks cannot handle them effectively. RNNs solve this problem by introducing a "memory" unit -- the hidden state.

Recurrent Structure:

When processing each time step t of a sequence, an RNN not only receives the current input x_t, but also receives the hidden state h_{t-1} from the previous time step.

It computes x_t and h_{t-1} together to generate the current output y_t and a new hidden state h_t.

This new h_t is passed to the next time step t+1.

In this way, h_t acts as a dynamic memory, encoding all historical information from the beginning of the sequence up to the current position.

The Predicament of RNNs: The Long-Term Dependency Problem

Standard RNNs suffer from vanishing/exploding gradients when processing long sequences. This means that during backpropagation, gradients decrease or increase exponentially with the number of time steps, making it difficult for the model to learn dependencies between distant elements in a sequence (e.g., the relationship between a subject at the beginning of a long paragraph and the verb at the end).

LSTM (Long Short-Term Memory)

LSTM is a special kind of RNN that solves the long-term dependency problem by introducing a more complex internal structure -- the cell state and three gates.

Cell State C_t: Like a conveyor belt, information flows along it with only minor linear interactions. This allows gradients to pass easily through long sequences.

Forget Gate: Decides which information to discard from the previous cell state C_{t-1}.

Input Gate: Decides which new information to store in the current cell state C_t.

Output Gate: Decides which information from the current cell state C_t to output as the hidden state h_t.

These gates are small neural networks controlled by Sigmoid activation functions. They can learn when to forget, when to remember, and when to output, thus more effectively capturing long-term dependencies. Before the Transformer, LSTMs and their variant GRUs were the absolute mainstays for NLP tasks.

# Using LSTM in PyTorch
# input_size: Feature dimension of the input at each time step (e.g., word vector dimension)
# hidden_size: Dimension of the hidden state
lstm_layer = nn.LSTM(input_size=100, hidden_size=256, num_layers=2, batch_first=True)

# Input shape: (batch_size, sequence_length, input_size)
dummy_input = torch.randn(32, 50, 100) 

# Output: output, (h_n, c_n)
# output: Hidden state at each time step (32, 50, 256)
# h_n: Hidden state of the last time step (num_layers, 32, 256)
# c_n: Cell state of the last time step (num_layers, 32, 256)
output, (hidden_state, cell_state) = lstm_layer(dummy_input)

5.4 The Art of Training: Loss Functions, Optimizers, and Regularization

5.4.1 Loss Functions: The Lighthouse Guiding the Way

Loss functions tell us how far the model is from the target. Choosing the right loss function is crucial.

Regression Tasks:

nn.MSELoss (Mean Squared Error): L = (y_pred - y_true)^2. The most commonly used regression loss, sensitive to outliers.

nn.L1Loss (Mean Absolute Error): L = |y_pred - y_true|. More robust to outliers.

Binary Classification Tasks:

nn.BCELoss (Binary Cross-Entropy Loss): Requires the model's output to be passed through a Sigmoid activation function to represent probabilities.

nn.BCEWithLogitsLoss: Combines Sigmoid and BCELoss, numerically more stable. The first choice for binary classification.

Multi-class Classification Tasks:

nn.CrossEntropyLoss: The most commonly used loss function for multi-class classification. It internally includes the Softmax operation and negative log-likelihood loss. Therefore, the model's raw output (logits) can be fed directly, without manually applying Softmax.

5.4.2 Optimizers: The Engine Driving Learning

Optimizers implement the gradient descent algorithm, updating the model's parameters based on the computed gradients.

torch.optim.SGD (Stochastic Gradient Descent): The most basic optimizer.

optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

momentum is an improvement that introduces an accumulation of past gradients, helping to accelerate convergence and overcome local optima.

torch.optim.Adam (Adaptive Moment Estimation): One of the most commonly used and versatile optimizers.

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

It combines the ideas of momentum and RMSProp, adaptively computing a learning rate for each parameter. In most cases, Adam achieves good and fast convergence, making it the first choice for beginners.

A Standard Training Loop

# Assume model, train_loader, loss_fn, optimizer are already defined
num_epochs = 10
for epoch in range(num_epochs):
    for inputs, labels in train_loader:
        # 1. Move data to GPU
        inputs, labels = inputs.to(device), labels.to(device)
      
        # 2. Forward propagation
        outputs = model(inputs)
      
        # 3. Compute loss
        loss = loss_fn(outputs, labels)
      
        # 4. Backpropagation
        # a. Zero old gradients
        optimizer.zero_grad()
        # b. Compute new gradients
        loss.backward()
      
        # 5. Update parameters
        optimizer.step()
      
    print(f"Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}")

5.4.3 Regularization: The Remedy Against "Rote Memorization"

When a model's capacity (complexity) far exceeds the complexity of the training data, it might "memorize" every sample in the training set, including noise. This leads to excellent performance on the training set but poor performance on unseen test data. This phenomenon is called overfitting. Regularization is a set of techniques used to combat overfitting.

L1/L2 Regularization:

Adds a penalty term related to the magnitude of the model's weights to the loss function.

L2 regularization (weight decay) tends to make weights smaller and more spread out. Implemented via the weight_decay parameter in PyTorch's optimizers: torch.optim.Adam(..., weight_decay=1e-4).

Dropout (nn.Dropout):

During each forward pass in training, randomly sets the output of a fraction p of neurons to zero.

This forces the network not to rely too heavily on any single neuron, but to learn more robust, redundant feature representations.

Important: Dropout only takes effect during training. Calling model.eval() automatically disables Dropout.

Early Stopping:

During training, continuously monitor the model's performance on the validation set.

If the loss on the validation set does not decrease for several consecutive epochs, or even starts to increase, terminate training early and save the best-performing model.

5.5 Hands-On Project: Implementing a Text Sentiment Classifier with PyTorch

Now, we will apply all the knowledge learned in this chapter to build a sentiment classifier from scratch that can determine whether IMDB movie reviews are positive or negative.

Project Workflow:

  1. Data Preparation: Load the IMDB dataset, perform text preprocessing (tokenization, vocabulary building, numericalization).
  2. Define Dataset and DataLoader: Use PyTorch's tools to efficiently load and batch data.
  3. Build the Model: We will build a model based on LSTM.
  4. Define Training and Evaluation Functions: Write the standard training loop and evaluation logic.
  5. Execute Training: Run the training process and observe changes in loss and accuracy.

Implementation Steps:

  1. Data Preparation

    We will use the torchtext library to conveniently process data.

    pip install torchtext spacy
python -m spacy download en_core_web_sm

    # 1_data_preparation.py
import torch
from torchtext.datasets import IMDB
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iterator
import spacy

# Load the English tokenizer
tokenizer = get_tokenizer('spacy', language='en_core_web_sm')

# Load the IMDB dataset
train_iter, test_iter = IMDB(split=('train', 'test'))

def yield_tokens(data_iter):
    for _, text in data_iter:
        yield tokenizer(text)

# Build the vocabulary
vocab = build_vocab_from_iterator(yield_tokens(train_iter), specials=["<unk>", "<pad>"])
vocab.set_default_index(vocab["<unk>"]) # Set default index for unknown words

# Define text and label processing pipelines
text_pipeline = lambda x: vocab(tokenizer(x))
label_pipeline = lambda x: 1 if x == 'pos' else 0

# Encapsulate processing logic
def process_data(data_iter):
    processed_data = []
    for label, text in data_iter:
        processed_text = torch.tensor(text_pipeline(text), dtype=torch.int64)
        processed_label = torch.tensor(label_pipeline(label), dtype=torch.int64)
        processed_data.append((processed_label, processed_text))
    return processed_data

train_data = process_data(IMDB(split='train'))
test_data = process_data(IMDB(split='test'))

print("Data preparation complete.")
# Can save the vocab and processed data for later use
# torch.save(vocab, 'vocab.pth')
# torch.save(train_data, 'train_data.pth')
# torch.save(test_data, 'test_data.pth')

  2. Dataset and DataLoader

    PyTorch's DataLoader requires a collate_fn to handle variable-length text sequences by padding them to the length of the longest sequence in a batch.

    # 2_dataloader.py
import torch
from torch.utils.data import DataLoader
from torch.nn.utils.rnn import pad_sequence

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Load previously processed data
# vocab = torch.load('vocab.pth')
# train_data = torch.load('train_data.pth')
# test_data = torch.load('test_data.pth')

PAD_IDX = vocab['<pad>']

def collate_batch(batch):
    label_list, text_list, lengths = [], [], []
    for (_label, _text) in batch:
        label_list.append(_label)
        text_list.append(_text)
        lengths.append(len(_text))

    labels = torch.tensor(label_list, dtype=torch.float32)
    texts = pad_sequence(text_list, batch_first=True, padding_value=PAD_IDX)
    lengths = torch.tensor(lengths, dtype=torch.int64)

    return labels.to(device), texts.to(device), lengths.to(device)

BATCH_SIZE = 64
train_dataloader = DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True, collate_fn=collate_batch)
test_dataloader = DataLoader(test_data, batch_size=BATCH_SIZE, shuffle=False, collate_fn=collate_batch)

  3. Build the Model

    # 3_model.py
import torch.nn as nn

class SentimentLSTM(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers, 
                bidirectional, dropout, pad_idx):
        super().__init__()

        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)
        self.lstm = nn.LSTM(embedding_dim, 
                            hidden_dim, 
                            num_layers=n_layers, 
                            bidirectional=bidirectional, 
                            dropout=dropout, 
                            batch_first=True)
        self.fc = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim, output_dim)
        self.dropout = nn.Dropout(dropout)

    def forward(self, text, text_lengths):
        # text = [batch size, sent len]
        embedded = self.dropout(self.embedding(text))
        # embedded = [batch size, sent len, emb dim]

        # Pack sequence
        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths.to('cpu'), batch_first=True, enforce_sorted=False)

        packed_output, (hidden, cell) = self.lstm(packed_embedded)

        # Unpack sequence
        # output, output_lengths = nn.utils.rnn.pad_packed_sequence(packed_output, batch_first=True)

        # Concat the final forward (hidden[-2,:,:]) and backward (hidden[-1,:,:]) hidden layers
        if self.lstm.bidirectional:
            hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim=1))
        else:
            hidden = self.dropout(hidden[-1,:,:])

        # hidden = [batch size, hid dim * num directions]
        return self.fc(hidden)

  4. Training and Evaluation

    # 4_train.py
import torch.optim as optim

# --- Model Hyperparameters ---
VOCAB_SIZE = len(vocab)
EMBEDDING_DIM = 100
HIDDEN_DIM = 256
OUTPUT_DIM = 1
N_LAYERS = 2
BIDIRECTIONAL = True
DROPOUT = 0.5

model = SentimentLSTM(VOCAB_SIZE, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM, 
                    N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)
model = model.to(device)

optimizer = optim.Adam(model.parameters())
criterion = nn.BCEWithLogitsLoss() # Suitable for binary classification
criterion = criterion.to(device)

def binary_accuracy(preds, y):
    """Returns the accuracy for a batch"""
    rounded_preds = torch.round(torch.sigmoid(preds))
    correct = (rounded_preds == y).float()
    acc = correct.sum() / len(correct)
    return acc

def train(model, iterator, optimizer, criterion):
    epoch_loss = 0
    epoch_acc = 0
    model.train()

    for labels, text, lengths in iterator:
        optimizer.zero_grad()
        predictions = model(text, lengths).squeeze(1)
        loss = criterion(predictions, labels)
        acc = binary_accuracy(predictions, labels)
        loss.backward()
        optimizer.step()

        epoch_loss += loss.item()
        epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

def evaluate(model, iterator, criterion):
    epoch_loss = 0
    epoch_acc = 0
    model.eval()

    with torch.no_grad():
        for labels, text, lengths in iterator:
            predictions = model(text, lengths).squeeze(1)
            loss = criterion(predictions, labels)
            acc = binary_accuracy(predictions, labels)

            epoch_loss += loss.item()
            epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

# --- Execute Training ---
N_EPOCHS = 5
for epoch in range(N_EPOCHS):
    train_loss, train_acc = train(model, train_dataloader, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, test_dataloader, criterion)

    print(f'Epoch: {epoch+1:02}')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

Chapter Summary

In this chapter, we successfully pushed open the heavy and captivating door to deep learning.

We started with the most basic neuron model, understanding how neural networks make predictions through forward propagation, and how they "learn" through the core mechanism of backpropagation and gradient descent. This laid a solid theoretical foundation for understanding all complex deep learning models that follow.

Then, we deeply mastered the PyTorch framework. We learned to use Tensor for GPU-accelerated computation, experienced the magic of Autograd's automatic differentiation, and mastered the use of nn.Module to build our own neural networks like assembling building blocks.

We also reviewed two classic neural network architectures -- CNNs and RNNs/LSTMs -- and understood their unique design ideas for handling spatial features and sequence data, respectively. This is crucial for broadening our horizons and understanding the evolution of more advanced architectures.

Finally, we explored the "art of training," learning how to choose the right loss functions and optimizers, and how to use regularization techniques to prevent overfitting. Through a complete text sentiment classification hands-on project, we condensed all the scattered knowledge points from this chapter into a practical deep learning project development workflow.

After completing this chapter, you are no longer a stranger to deep learning. You have the ability to understand, build, and train a moderately complex deep learning model. This solid "cornerstone" will powerfully support your next and most core goal in this book: to deeply understand and master the "crown jewel" -- the Transformer architecture and large language models.

writings