祝融说。

If the Python inner strength we cultivated in the previous chapter is the "mental discipline" of an AI engineer, then what we are about to forge in this chapter is the sharpest "weapon" in our hands — the core data science toolchain. In the world of artificial intelligence, data is the fuel that drives everything. Raw data, like unrefined ore, is mixed and chaotic. What a model needs as input, however, is refined, pure "gold bullion." The process from ore to gold bullion is the "alchemy" of data science, and its core techniques lie in the four great tools: NumPy, Pandas, Matplotlib, and Scikit-learn.

NumPy is the cornerstone of this toolchain. It brings high-performance multidimensional array objects and a rich mathematical function library to Python. More importantly, it teaches us a new way of thinking — vectorized thinking — which is the first step away from inefficient loops and toward high-performance computing.

Pandas is the Swiss Army knife of data processing and analysis. It provides the powerful and intuitive DataFrame data structure, allowing us to easily clean, transform, filter, aggregate, and perform other complex operations on structured data, much like manipulating an Excel spreadsheet.

Matplotlib and Seaborn are our "eyes." They transform dry numbers into vivid charts, helping us intuitively understand data distributions, discover relationships between variables, uncover hidden patterns, and ultimately present our findings as a "data story."

Scikit-learn is the comprehensive toolkit for traditional machine learning. With its unified and concise API, it packages the entire pipeline from data preprocessing to model training and evaluation, making it a powerful tool for quickly building and validating baseline models.

For AI engineers, whether your future focus is cutting-edge deep learning models or complex LLM applications, this toolchain is an unavoidable path. The success of any model begins with a deep understanding and meticulous processing of the data. A well-cleaned and feature-engineered dataset is often more valuable than an untuned complex model.

In this chapter, we will not simply list the APIs of these libraries. Instead, we will use a single hands-on project running throughout — the classic Kaggle competition "Titanic: Machine Learning from Disaster" — as a vehicle to simulate a complete data science project workflow. We will start from loading the raw data, step by step use Pandas for cleaning and exploratory data analysis (EDA), leverage Matplotlib and Seaborn to reveal the secrets hidden in the data, utilize NumPy for efficient numerical computation, and finally use Scikit-learn to build, train, and evaluate our prediction model.

This is not just a study of technology, but a training of thinking. You will learn how to think like a real data scientist: how to ask questions, how to find answers through data, how to validate hypotheses, and how to transform analytical results into valuable models.

Now, let us load these powerful tools and embark on this exciting journey of "alchemy" from raw data.

2.1 NumPy: The Cornerstone of Scientific Computing and Vectorized Thinking

NumPy (Numerical Python) is the absolute core of the Python scientific computing ecosystem. Virtually all higher-level data science libraries, including Pandas, Scikit-learn, TensorFlow, and PyTorch, rely on NumPy's powerful ndarray object at their foundation.

2.1.1 ndarray: More Than Just a Python List

Python's built-in list is flexible but inefficient. A list can store elements of different types, which means it stores pointers to various objects scattered throughout memory. When you perform calculations on numbers in a list, the Python interpreter must dereference each pointer and perform type checking on each element one by one — a very slow process.

NumPy's ndarray (n-dimensional array) is fundamentally different:

1. Homogeneity: All elements in an ndarray must be of the same data type (e.g., int32, float64).
2. Contiguous Memory: ndarray occupies a contiguous, compact region in memory.

These two characteristics bring enormous performance advantages. Because the type is uniform and the memory is contiguous, NumPy can leverage highly optimized code written in C or Fortran to perform mathematical operations on entire arrays without needing loops at the Python level. This operation is called vectorization.

import numpy as np
import time

# Create a large list and an ndarray
n = 10_000_000
py_list = list(range(n))
np_array = np.arange(n)

# Python list comprehension for squaring
start_time = time.time()
py_list_squared = [x**2 for x in py_list]
end_time = time.time()
print(f"Python list comprehension time: {end_time - start_time:.4f} s")

# NumPy vectorized squaring
start_time = time.time()
np_array_squared = np_array ** 2
end_time = time.time()
print(f"NumPy vectorization time: {end_time - start_time:.4f} s")

Output (a calculation example from one typical machine; exact values vary by machine, but the order-of-magnitude difference is stable):

Python list comprehension time: 2.6512 s
NumPy vectorization time: 0.0210 s

In this calculation example the performance difference exceeds 100x (it fluctuates across machines and data sizes, typically within one to two orders of magnitude). This is the power of vectorized thinking. When performing numerical computations, your first thought should be: "Can I use a single NumPy operation to replace this for loop?"

2.1.2 Core Operations: Creation, Indexing, and Broadcasting

Creating arrays:

# From a list
a = np.array([1, 2, 3])

# Create arrays with specific shapes and values
zeros = np.zeros((2, 3))      # 2x3 array of zeros
ones = np.ones((3, 2))        # 3x2 array of ones
full = np.full((2, 2), 7)     # 2x2 array of 7s
eye = np.eye(3)               # 3x3 identity matrix
rand = np.random.rand(2, 3)   # 2x3 array of [0,1) uniform random numbers

Indexing and Slicing:

NumPy's indexing is more powerful than Python lists, supporting multidimensional indexing and advanced indexing.

arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

# Basic indexing
print(arr[0, 1])  # Output: 2 (row 0, column 1)

# Slicing
print(arr[:2, 1:]) # Output: [[2, 3], [5, 6]] (first 2 rows, from column 1 to the end)

# Boolean indexing (extremely important!)
bool_idx = arr > 5
print(bool_idx)
# [[False False False]
#  [False False  True]
#  [ True  True  True]]

print(arr[bool_idx]) # Output: [6 7 8 9] (all elements greater than 5)

# Integer array indexing (fancy indexing)
print(arr[[0, 2], [1, 2]]) # Output: [2 9] (get elements at positions (0,1) and (2,2))

Boolean indexing is a core technique for conditional filtering in data analysis, and we will see it again in the Pandas section.

Broadcasting:

Broadcasting is one of NumPy's most powerful and also most confusing features. It describes how NumPy handles arithmetic operations between arrays of different shapes. Simply put, if the shapes of two arrays do not match, NumPy attempts to "stretch" (or "broadcast") the smaller array to make its shape compatible with the larger array.

a = np.array([[1, 2, 3], [4, 5, 6]]) # shape (2, 3)
b = np.array([10, 20, 30])           # shape (3,)

# Each row of a is added to b
c = a + b
print(c)
# [[11 22 33]
#  [14 25 36]]

Here, b (shape (3,)) is broadcast to [[10, 20, 30], [10, 20, 30]] (shape (2, 3)), and then added element-wise to a. Broadcasting eliminates the need to manually create duplicate rows or columns, resulting in cleaner code and higher memory efficiency.

2.1.3 Role in AI

Data Representation: Images can be represented as 3D ndarrays of shape (height, width, channels). Text data, after being processed through word embeddings, can be represented as 2D ndarrays of shape (num_tokens, embedding_dim).

Mathematical Operations: The underlying tensor operations of all deep learning frameworks are highly consistent with NumPy's API and philosophy. Mastering NumPy's matrix multiplication (@ or np.dot), summation (np.sum), mean (np.mean), and other operations is fundamental to understanding the internal calculations of models.

Interacting with Libraries: Pandas' DataFrame can be easily converted to and from NumPy arrays (via the .values attribute and the pd.DataFrame() constructor), serving as a bridge connecting data processing and model training.

2.2 Pandas: From Data Cleaning to Exploratory Data Analysis (EDA)

If NumPy is the tool for handling pure numbers, then Pandas was born for handling the messy, imperfect tabular data (structured data) of the real world.

2.2.1 Two Core Data Structures: Series and DataFrame

Series: A labeled one-dimensional array. Think of it as an enhanced version of a NumPy one-dimensional array, because it has an associated index.

import pandas as pd
s = pd.Series([10, 20, 30], index=['a', 'b', 'c'])
print(s['b']) # Output: 20

DataFrame: A two-dimensional, labeled data structure that can be thought of as a collection of Series sharing the same index. It is the most widely used data structure in Pandas, and intuitively resembles an Excel spreadsheet or SQL table.

Each column is a Series. It has a row index and column index.

2.2.2 Loading Data in Practice: The Titanic Dataset

Now, let us officially begin our Kaggle project. First, download the Titanic dataset (typically containing train.csv and test.csv) and load it using Pandas.

# Assume the data files are in the 'data/' directory
train_df = pd.read_csv('data/train.csv')
test_df = pd.read_csv('data/test.csv')

# Initial data exploration
print("Training set shape:", train_df.shape)
print("\nFirst 5 rows:")
print(train_df.head())
print("\nBasic data information:")
train_df.info()
print("\nNumerical feature descriptive statistics:")
print(train_df.describe())

• head(), info(), describe() are the "three essential tools" for exploring any new dataset.
• head() gives us an intuitive impression of what the data looks like.
• info() tells us the data type and missing value situation for each column — the starting point for data cleaning.
• describe() provides statistical information for numerical columns such as mean, standard deviation, and percentiles, which helps us discover outliers.

From the info() output, we immediately see that the Age, Cabin, and Embarked columns have missing values. The Cabin column is missing a significant number of values.

2.2.3 Data Cleaning: Handling Missing Values, Duplicates, and Outliers

Handling Missing Values (NaN)

# Check the number of missing values per column
print(train_df.isnull().sum())

# Strategy 1: Imputation
# Age: Fill with the median age, because the age distribution may be skewed
age_median = train_df['Age'].median()
train_df['Age'].fillna(age_median, inplace=True)

# Embarked: Fill with the most frequent port
embarked_mode = train_df['Embarked'].mode()[0]
train_df['Embarked'].fillna(embarked_mode, inplace=True)

# Strategy 2: Deletion
# Cabin: Too many missing values, simply drop the column
train_df.drop('Cabin', axis=1, inplace=True)

# The same operations need to be applied to test_df
# ...

inplace=True means modifying the original DataFrame directly. axis=1 means the operation targets columns.

Handling Duplicate Values

# Check for completely duplicated rows
print(f"Number of duplicate rows: {train_df.duplicated().sum()}")
# train_df.drop_duplicates(inplace=True) # If any exist, delete them

2.2.4 Data Filtering and Transformation: .loc, .iloc, and Feature Engineering

Data Filtering

• .loc: Label-based indexing.
• .iloc: Integer position-based indexing.

# Filter for male passengers older than 60
old_men = train_df.loc[(train_df['Age'] > 60) & (train_df['Sex'] == 'male')]

# Filter rows 1 to 3, columns 2 to 4
subset = train_df.iloc[1:4, 2:5]

Note that the conditional filtering (train_df['Age'] > 60) & (train_df['Sex'] == 'male') leverages the same kind of NumPy-style boolean indexing.

Feature Engineering

This is the most creative part of data science — creating new, more useful features from raw data.

# Create FamilySize feature
train_df['FamilySize'] = train_df['SibSp'] + train_df['Parch'] + 1

# Extract Title from Name (Mr, Mrs, Miss, etc.)
train_df['Title'] = train_df['Name'].apply(lambda name: name.split(',')[1].split('.')[0].strip())

# Convert the Sex text feature to a numerical feature
train_df['Sex_numeric'] = train_df['Sex'].map({'male': 0, 'female': 1})

# One-hot encode Embarked
embarked_dummies = pd.get_dummies(train_df['Embarked'], prefix='Embarked')
train_df = pd.concat([train_df, embarked_dummies], axis=1)

• .apply() applies a function to each element of a Series.
• .map() is used for value replacement based on a dictionary.
• pd.get_dummies() is a convenient method for one-hot encoding.

2.2.5 Data Aggregation: groupby

The groupby operation is core to data analysis, implementing the "split-apply-combine" pattern.

# Calculate survival rate by gender
print(train_df.groupby('Sex')['Survived'].mean())

# Calculate survival rate by passenger class and gender
print(train_df.groupby(['Pclass', 'Sex'])['Survived'].mean())

# Use agg for more complex aggregation
agg_funcs = {
    'Survived': 'mean',
    'Age': ['mean', 'max', 'min']
}
print(train_df.groupby('Pclass').agg(agg_funcs))

Through groupby, we can quickly validate hypotheses, such as "Is the survival rate for females higher than for males?" and "Is the survival rate for first class the highest?"

2.3 Matplotlib and Seaborn: Visual Storytelling with Data

Numbers are abstract, but graphs are intuitive. Visualization is the soul of exploratory data analysis (EDA).

Matplotlib: The foundational library for Python visualization. It is powerful and highly customizable, but its API can sometimes be a bit complex.

Seaborn: Built on top of Matplotlib, it provides higher-level APIs focused on statistical graphics. It can produce more beautiful charts with less code. We typically use the two together.

import matplotlib.pyplot as plt
import seaborn as sns

# Set the plotting style
sns.set_style('whitegrid')

2.3.1 Univariate Analysis: Understanding Data Distribution

Categorical Variables: Countplot

plt.figure(figsize=(8, 5))
sns.countplot(x='Survived', data=train_df)
plt.title('Survival Count Distribution (0 = No, 1 = Yes)')
plt.show()

plt.figure(figsize=(8, 5))
sns.countplot(x='Pclass', hue='Survived', data=train_df)
plt.title('Survival by Passenger Class')
plt.show()

The first plot shows the overall survival distribution. The second plot, using the hue parameter, clearly demonstrates the strong relationship between passenger class and survival: the higher the class, the higher the survival rate.

Continuous Variables: Histogram (histplot) and Kernel Density Estimate Plot (kdeplot)

plt.figure(figsize=(10, 6))
sns.histplot(data=train_df, x='Age', hue='Survived', kde=True, bins=30)
plt.title('Survival Distribution by Age')
plt.show()

This plot conveys a wealth of information: we can see that children (Age < 10) have a very high survival rate, while the mortality rate for young adults (Age 18-30) is higher.

2.3.2 Bivariate/Multivariate Analysis: Exploring Relationships

Scatterplot: Continuous vs Continuous

# The Titanic dataset does not have two good continuous variables for this, shown for demonstration only
# sns.scatterplot(x='Age', y='Fare', hue='Survived', data=train_df)

Boxplot: Categorical vs Continuous

plt.figure(figsize=(10, 6))
sns.boxplot(x='Pclass', y='Age', data=train_df)
plt.title('Age Distribution by Passenger Class')
plt.show()

The boxplot clearly shows that first-class passengers have the highest average age and a wider age distribution.

Heatmap: Displaying the Correlation Matrix

# Select only numerical columns for correlation calculation
numeric_cols = train_df.select_dtypes(include=np.number)
corr_matrix = numeric_cols.corr()

plt.figure(figsize=(12, 10))
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm', fmt='.2f')
plt.title('Feature Correlation Heatmap')
plt.show()

The heatmap helps us quickly identify linear relationships between features. For example, Pclass and Fare have a strong negative correlation, which aligns with common sense.

Through this series of visual explorations, we have developed a deep understanding of the data, laying a solid foundation for the subsequent model building.

2.4 Scikit-learn: Rapid Implementation and Evaluation of Traditional Machine Learning Algorithms

Scikit-learn is the bridge that takes us from data analysis to the model building phase of machine learning. Its design philosophy is unity and simplicity.

2.4.1 Scikit-learn's Core API Design

Objects in Scikit-learn follow a consistent interface:

Estimator: Any object that can learn from data.

estimator.fit(X, y): Used to train a model. X is the feature data, y is the label.

Transformer: A special type of Estimator used for data transformation.

transformer.transform(X): Transform the data.

transformer.fit_transform(X, y): Learn parameters and then transform, more efficient. For example, StandardScaler.

Model: An Estimator used for making predictions.

model.predict(X): Make predictions.

model.predict_proba(X): Predict probabilities (for classification models).

model.score(X, y): Evaluate model performance.

2.4.2 Data Preparation: Building the Input for the Model

Models cannot directly process raw DataFrames; we need to convert them into purely numerical NumPy arrays.

# Drop unnecessary, non-numerical columns
train_df_final = train_df.drop(['PassengerId', 'Name', 'Sex', 'Ticket', 'Embarked', 'Title'], axis=1)

# Ensure the test set has undergone the exact same processing steps
# ... (code for fully processing test_df omitted here, but it is critical in practice)

# Define features X and target y
X = train_df_final.drop('Survived', axis=1)
y = train_df_final['Survived']

# Split into training and validation sets
from sklearn.model_selection import train_test_split
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

train_test_split is a crucial function that helps us divide the data into training and validation sets to evaluate the model's generalization ability. The random_state parameter ensures the same split every time, enabling reproducibility.

2.4.3 Model Training and Evaluation

Let us try several classic classification models.

Logistic Regression

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report

# 1. Initialize the model
log_reg = LogisticRegression(max_iter=1000)

# 2. Train the model
log_reg.fit(X_train, y_train)

# 3. Make predictions on the validation set
y_pred_log_reg = log_reg.predict(X_val)

# 4. Evaluate the model
accuracy = accuracy_score(y_val, y_pred_log_reg)
print(f"Logistic Regression validation accuracy: {accuracy:.4f}")
print("\nClassification report:")
print(classification_report(y_val, y_pred_log_reg))

Random Forest

from sklearn.ensemble import RandomForestClassifier

# 1. Initialize the model
rf_clf = RandomForestClassifier(n_estimators=100, random_state=42)

# 2. Train the model
rf_clf.fit(X_train, y_train)

# 3. Predict
y_pred_rf = rf_clf.predict(X_val)

# 4. Evaluate
accuracy_rf = accuracy_score(y_val, y_pred_rf)
print(f"Random Forest validation accuracy: {accuracy_rf:.4f}")

Random Forest typically performs better than Logistic Regression because it is a more powerful ensemble model.

2.4.4 Model Optimization: Cross-Validation and Grid Search

A good model not only performs well but also has its parameters properly tuned.

Cross-Validation: A more robust evaluation method than a single train-validation split. It divides the training set into K folds, trains on K-1 folds and validates on 1 fold in turn, and finally takes the average score.

Grid Search: Automatically searches through a defined grid of parameters using cross-validation to find the best combination of hyperparameters.

from sklearn.model_selection import GridSearchCV

# Define the parameter grid
param_grid = {
    'n_estimators': [100, 200, 300],
    'max_depth': [None, 10, 20, 30],
    'min_samples_split': [2, 5, 10]
}

# Initialize GridSearchCV
grid_search = GridSearchCV(
    estimator=RandomForestClassifier(random_state=42),
    param_grid=param_grid,
    cv=5, # 5-fold cross-validation
    scoring='accuracy',
    verbose=1,
    n_jobs=-1 # Use all CPU cores
)

# Search on the entire training set (X, y)
grid_search.fit(X, y)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best cross-validation accuracy: {grid_search.best_score_:.4f}")

Through grid search, we can find an optimal set of hyperparameters, further improving model performance.

2.5 Hands-On Project Summary: End-to-End Kaggle Data Analysis and Modeling

Looking back at our journey through this chapter, we completed an end-to-end data science project, which is a microcosm of an industrial AI project.

Process Review:

1. Problem Definition: Predict the survival of Titanic passengers (a binary classification problem).
2. Data Acquisition: Load data using Pandas' read_csv.
3. Exploratory Data Analysis (EDA) and Data Cleaning:
   1. Used .info(), .describe(), .isnull().sum() to quickly understand the data overview.
   2. Used Matplotlib and Seaborn for in-depth visual analysis, discovering strong correlations between factors like age, gender, passenger class, and survival.
   3. Based on the analysis, reasonably imputed missing values (median, mode) and dropped useless columns.
4. Feature Engineering:
   1. Created new features like FamilySize, Title, etc.
   2. Converted categorical features like Sex, Embarked into numerical formats understandable by the model (value mapping, one-hot encoding).
5. Model Building and Training:
   1. Used Scikit-learn's train_test_split to split the dataset.
   2. Followed the unified fit/predict API to quickly implement two baseline models: Logistic Regression and Random Forest.
6. Model Evaluation and Optimization:
   1. Used accuracy_score and classification_report to evaluate model performance on the validation set.
   2. Learned to use GridSearchCV for hyperparameter tuning to find better model configurations.

Final Deliverable:

The final step of the project is typically to retrain the best model found (grid_search.best_estimator_) on the complete training data, then make predictions on the official test.csv and generate a submission file.

# Use the best model to make predictions on the test set
best_rf = grid_search.best_estimator_
# ... (perform the exact same preprocessing on test_df as on train_df) ...
test_predictions = best_rf.predict(X_test_final)

# Create the submission file
submission = pd.DataFrame({
    'PassengerId': test_df['PassengerId'],
    'Survived': test_predictions
})
submission.to_csv('submission.csv', index=False)

Chapter Summary

In this chapter, we not only learned the APIs of four libraries — NumPy, Pandas, Matplotlib, and Scikit-learn — but more importantly, through a real project, we linked them together into an effective workflow.

You should now have a deep understanding that:

Vectorized thinking is the key to improving numerical computation performance.

Data cleaning and feature engineering are the core steps that determine the upper bound of model performance, requiring you to combine business understanding with data insight.

Visualization is not an optional embellishment, but the engine that drives data analysis and hypothesis validation.

Scikit-learn provides a powerful and concise set of tools that allow you to rapidly turn ideas into evaluable models.

This data science toolchain is your "standard equipment" as an AI engineer. Mastering it will give you the confidence to face any data, and will lay the most solid foundation for the more complex deep learning and LLM projects to come. In the next chapter, we will begin building the backend foundation for AI services, learning how to package our trained model into an API that can serve external requests.