祝融说。

In the previous chapter, we thoroughly dissected the Transformer architecture — the magnificent cornerstone that supports the entire edifice of modern NLP and AI. We understood how the Self-Attention mechanism enables models to capture long-distance dependencies in parallel, and we built a simplified Transformer with our own hands. Now, standing on the shoulders of this Transformer giant, we will gaze upon an even more magnificent landscape — the era of Large Language Models (LLMs).

Since 2018, the AI field has witnessed a profound transformation driven by the "Scaling Law." Researchers found that as the parameter count, training data volume, and computational scale of Transformer models grow, performance on a wide range of tasks keeps improving; some capabilities — such as In-context Learning, Zero/Few-shot Reasoning, and Chain-of-Thought — only become clearly visible in models beyond a certain scale, and have been called "emergent abilities." It should be noted that whether this "emergence" reflects a genuine phase transition remains debated: one study (Stanford's Are Emergent Abilities of Large Language Models a Mirage?) argues that part of the phenomenon is a measurement artifact caused by nonlinear or discontinuous evaluation metrics, with capability curves becoming smooth once continuous metrics are used. Whichever explanation holds, the model is no longer merely an executor of tasks; it begins to exhibit a degree of generality across a broad range of tasks.

From BERT's "fill-in-the-blank," to GPT-3's "text continuation," to ChatGPT and GPT-4's "conversational AI," every iteration of LLMs redefines the boundaries of human-computer interaction. They are fundamentally changing the way we create content, write code, access information, and even conduct scientific research. A new paradigm of AI application — Generative AI — has arrived.

As AI engineers in this new era, we can no longer be satisfied with merely training a model for a specific task. We must learn how to understand, harness, and deploy these powerful "Foundation Models." This chapter is your first "panoramic map" into the world of LLMs. Together, we will explore:

• The Evolution of LLMs: We will review the magnificent technical journey from BERT to the GPT series. You will understand the differences among the three major architectural branches — Encoder-only, Decoder-only, and Encoder-Decoder — and the scenarios to which each is best suited.
• A Guide to Mainstream Open-Source LLMs: Beyond closed-source models such as GPT-4, a thriving open-source LLM ecosystem is rising. We will survey and compare mainstream open-source models like Llama, Gemma, and Qwen, providing you with a decision-making basis for technology selection in real projects.
• Prompt Engineering: Interacting with an LLM is like conversing with an "alien intelligence" that is knowledgeable yet requires precise instructions. We will systematically learn the emerging art of "prompt engineering," mastering how to use carefully crafted prompts to maximally unlock and channel the LLM's potential.
• Limitations and Challenges of LLMs: With great power comes great responsibility. We must also soberly recognize that LLMs are not omnipotent. We will delve into their inherent problems of hallucination, bias, and safety, and understand the current industry approaches and methods for meeting these challenges.

This chapter will build a macro-level cognitive framework around LLMs. It will help you understand "where we are now" and point the way toward "where we are going." With this map in hand, you will be better oriented, in the chapters ahead, to learn how to fine-tune LLMs and build advanced applications such as RAG and agents. Now, let us step together into this new era of intelligence forged from code, data, and immense computational power.

7.1 The Evolution of LLMs: From BERT to the GPT Series

The birth of the Transformer architecture paved the way for building larger and more powerful language models. Subsequent development proceeded mainly along three parallel yet mutually influential technical routes, corresponding to three architectural variants of the Transformer: Encoder-only, Decoder-only, and Encoder-Decoder.

7.1.1 The Establishment of the Pretraining Paradigm: From ELMo to BERT

Before LLMs, NLP tasks typically required designing a separate model for each task and training it from scratch. In 2018, a series of works established the new paradigm of "Pre-training, Fine-tuning," completely changing the rules of the game in NLP.

ELMo (Embeddings from Language Models): It used a bidirectional LSTM pretrained on large amounts of text, but its core contribution was proposing "contextualized word embeddings." The embedding of the same word differs across different sentences, solving the "polysemy" problem that plagued traditional word vectors (such as Word2Vec).

BERT (Bidirectional Encoder Representations from Transformers): BERT was the tipping point of this revolution. It uses only the Encoder part of the Transformer.

Architecture: Encoder-only.

Pretraining Tasks:

1. Masked Language Model (MLM): This is a "fill-in-the-blank" game. In the input sentence, 15% of the words are randomly replaced with a special [MASK] token, and the model is asked to predict the masked-out original words. Through this task, the model is compelled to learn deep bidirectional contextual representations.
2. Next Sentence Prediction (NSP): Given two sentences A and B, the model must determine whether B is the actual next sentence following A. This task is designed to teach the model relationships between sentences.

Core Advantage: Thanks to the MLM task, BERT can see both the left and right context of a word during pretraining, thereby achieving a deeper semantic understanding than any previous model.

Applicable Scenarios: BERT and its variants (such as RoBERTa, ALBERT) excel at understanding tasks (NLU), such as text classification, sentiment analysis, named entity recognition, and extractive question answering. They act like a powerful "feature extractor," providing high-quality sentence or word representations for downstream tasks.

7.1.2 The Rise of Generative Capabilities: The Decoder-only Path of the GPT Series

Developing in parallel with BERT was the GPT (Generative Pre-trained Transformer) series led by OpenAI. It chose a different technical route.

Architecture: Decoder-only. The GPT series uses only the Decoder part of the Transformer (while removing the Encoder-Decoder cross-attention layer).

Pretraining Task: The standard language model task (CLM), which predicts a word based on all the words preceding it. This is a "text continuation" game. For example, given the sentence "The weather today is," the model must predict that the next word is "fine."

Core Characteristics:

• Auto-regressive: GPT's generation process proceeds one word at a time; the currently generated word becomes the input for the next prediction.
• Unidirectional Context: When predicting the t-th word, the model can see only the information from the preceding t-1 words, not future information. This is achieved through "masking" in the Self-Attention layer.

The Evolution of the GPT Series:

• GPT-1 (2018): First demonstrated the effectiveness of generative pretraining on diverse, unlabeled text followed by fine-tuning on downstream tasks.
• GPT-2 (2019): Significantly increased model scale (up to 1.5 billion parameters) and used a larger, more diverse dataset. GPT-2 exhibited astonishing zero-shot generation capabilities, completing some simple tasks — such as article summarization and machine translation — without explicit fine-tuning. This gave the research community its first glimpse of the power of the "Scaling Law."
• GPT-3 (2020): A massive leap. Its parameter count reached 175 billion. GPT-3's in-context learning (ICL) capability at scale stunned the research community. You need only provide a few task examples (Few-shot) in the prompt, and the model can "grasp" your intent and complete new, similar tasks — without updating any model weights.
• InstructGPT / ChatGPT (2022): Building on GPT-3, it introduced Reinforcement Learning from Human Feedback (RLHF). By having humans rank the model's different outputs, training a reward model, and then using reinforcement learning to optimize the language model, its outputs became better aligned with human preferences, instructions, and values. This greatly improved the model's alignment capability, making it more "helpful" and "harmless," and giving rise to the phenomenon that was ChatGPT.
• GPT-4 (2023): Further scaling of the model, together with the introduction of multi-modality capabilities, enabling it to process both text and image inputs.

Applicable Scenarios for Decoder-only Architecture: Thanks to its powerful text generation capabilities, this class of model is highly suited to generation tasks (NLG), such as dialogue systems, content creation, code generation, summarization, and translation. Virtually all the chatbots we know today are built on this architecture.

7.1.3 The Best of Both Worlds: The Encoder-Decoder Architecture of T5 and BART

The third route attempts to combine the strengths of the previous two.

Architecture: Full Encoder-Decoder.

Representative Models: T5 (Text-to-Text Transfer Transformer), BART.

Core Idea: Unify all NLP tasks into a "Text-to-Text" format. For example, for translation, the input is "translate English to German: That is good." and the model should output "Das ist gut." For sentiment classification, the input is "sst2 sentence: This movie is great!" and the model should output "positive." For summarization, the input is "summarize: {long article text}" and the model should output the summary.

Pretraining Task: Typically a "denoising" objective. For example, T5 randomly corrupts the input text (by deleting or replacing segments), then has the model recover the original, uncorrupted text.

Applicable Scenarios: The Encoder-Decoder architecture excels at tasks that require both a deep understanding of the source text and complex generation, such as summarization, translation, and generative question answering.

Summary of the Three Architectural Branches:

7.2 A Guide to Mainstream Open-Source LLMs (Llama, Gemma, Qwen, etc.)

While closed-source models like GPT-4 are powerful, they are "black boxes," costly to call, and cannot be deeply customized. In recent years, a thriving open-source LLM ecosystem has been flourishing, giving enterprises and developers more choices.

When selecting an open-source LLM, you need to consider several key factors:

1. Model Scale (Parameter Count): e.g., 7B, 13B, 70B (B stands for Billion). The larger the model, the more capable it usually is, but the higher the demands on hardware (especially GPU memory).
2. Context Length: The maximum length of input text the model can process (measured in tokens). For tasks involving long documents, this parameter is crucial.
3. License: Determines whether you can use the model for commercial purposes. For example, Llama 2's license permits commercial use but with certain restrictions.
4. Multilingual/Multimodal Capabilities: Does the model support Chinese or other languages? Can it process image input?
5. Community Ecosystem and Support: Is there an active community, adequate fine-tuning scripts, quantized versions, and other resources?

7.2.1 Meta's Llama Series: The "De Facto Standard" of the Open-Source Community

Llama (2023.02): Meta's first release. Although it was not initially open for commercial use, its high quality and performance ignited the passion of the entire open-source community.

Llama 2 (2023.07): A milestone in the history of open-source LLMs.

• Sizes: 7B, 13B, 70B.
• Features: Compared with Llama 1, it was trained on a larger public dataset (2 trillion tokens), with the context length doubled to 4096. It also released a Llama-2-Chat version aligned through RLHF, with significantly improved conversational capabilities.
• License: Permits commercial use, but with restrictions on companies with over 700 million monthly active users.
• Ecosystem: It has the largest and most active community. Nearly all new fine-tuning techniques, quantization methods, and inference frameworks support Llama 2 first. It is the baseline model for both academic research and commercial exploration.

Llama 3 (2024.04): The latest iteration at the time of writing (open-source models iterate extremely fast; check for the newest releases when reading).

• Sizes: Currently released in 8B and 70B versions, with a larger 400B+ model expected in the future.
• Features: It was pretrained on a massive dataset of up to 15 trillion tokens, with extremely rigorous filtering of data quality. Its new tokenizer offers better multilingual support. At the same scale, Llama 3 comprehensively outperforms Llama 2, and its 8B model can even rival Llama 2 70B on some benchmarks. The context length has been increased to 8192.
• Selection Advice: If hardware permits, Llama 3 was the common first choice among open-source models at the time of writing; open-source models iterate extremely fast, so verify against the latest benchmarks and your own workload measurements.

7.2.2 Google's Gemma: A Technologically Advanced "Approachable" Option

Gemma (2024.02): Released by Google DeepMind, with technology inherited from the powerful closed-source model Gemini.

• Sizes: 2B, 7B.
• Features: Despite their modest size, their performance is highly competitive within their class; the 7B version outperforms Llama 2 7B/13B on multiple benchmarks. It uses the same tokenizer and architecture as Gemini.
• License: Very permissive, allowing commercial use.
• Selection Advice: For scenarios with limited resources that call for deployment on edge devices or small servers, Gemma 2B/7B is a highly cost-effective choice.

7.2.3 Alibaba's Qwen Series: A Leader in the Chinese Domain

Qwen (2023.08): A series of models from Alibaba DAMO Academy.

• Sizes: From 1.8B, 7B, 14B to 72B, covering a very wide range.
• Features: Its greatest advantage is support for Chinese. It has invested heavily in Chinese pretraining corpora, yielding excellent performance in Chinese understanding, generation, and alignment. It also offers various variants supporting long contexts (Qwen-7B-Chat-32k), code generation (CodeQwen), and multimodality (Qwen-VL).
• License: Early versions carried commercial restrictions, but later versions (such as Qwen1.5) have become quite open.
• Selection Advice: If your core business scenario is oriented toward Chinese, the Qwen series is undoubtedly one of the top choices.

7.2.4 Other Notable Models

The Mistral AI Series (Mistral 7B, Mixtral 8x7B): Released by Mistral AI, a French company founded by former DeepMind and Meta employees.

• Mistral 7B: Known for its compact size and performance surpassing Llama 2 13B, with outstanding efficiency.
• Mixtral 8x7B: A Sparse Mixture-of-Experts (SMoE) model. It contains 8 "experts" (each a 7B model) internally; during inference, each token is routed to only the 2 most relevant experts. This allows it to possess a huge effective parameter count (~47B) while maintaining an inference cost comparable to that of a 12B model, achieving an ideal combination of performance and efficiency.

The Yi Series (from 01.AI): Released by the company founded by Kai-Fu Lee, drawing attention for its strong foundational model capabilities and long-context support.

Simplified Selection Decision Tree:

1. Primarily Chinese-oriented?
   • Yes -> Qwen Series
   • No -> Continue
2. Need the strongest overall performance, with sufficient hardware resources (e.g., A100/H100)?
   • Yes -> Llama 3 70B or Mixtral 8x7B
   • No -> Continue
3. Limited hardware resources, seeking the best value for money?
   • Yes -> Llama 3 8B (overall best choice), Gemma 7B (newer technology), or Mistral 7B (high efficiency)
4. Need deployment on edge devices or in extremely low-resource environments?
   • Yes -> Gemma 2B or Qwen 1.8B

7.3 Prompt Engineering: The Art of Effective Communication with LLMs

Having a powerful LLM is like possessing a superbrain that knows everything but is a bit "bookish." How you ask your questions directly determines the quality of the answers you receive. Prompt Engineering is the discipline of designing and optimizing prompts to better guide an LLM in completing specific tasks.

7.3.1 Basic Components of a Prompt

A good prompt typically contains one or more of the following elements:

• Role: Assign an identity to the LLM. "You are a senior software architect..."
• Instruction: Explicitly tell the model what to do. "Please summarize the following article..."
• Context: Provide relevant background information. "Background: The company is planning an e-commerce project. Question:..."
• Examples / Shots: Supply input/output exemplars, i.e., in-context learning.
  • Zero-shot: Ask directly without giving any examples.
  • Few-shot: Provide a small number (typically 1-5) of examples.
• Output Format: Specify the output form you want. "Please return the result in JSON format, containing 'title' and 'summary' keys."

7.3.2 Core Principles and Techniques

1. Be Clear and Specific, Not Brief: Do not be afraid to write long prompts. Providing ample detail and context yields far better results than vague, terse sentences.

   • Poor: "Summarize this article."
   • Good: "Write a summary of no more than 100 words for the following article, aimed at general readers unfamiliar with AI. Highlight its core conclusions and practical significance. Article: {...}"

2. Assign a Role: Having the model play an expert role can effectively steer the style and depth of its output.

   • "You are an experienced travel planner. Please design a 7-day family itinerary for the Kansai region of Japan, with particular attention to traveling with the elderly and children."

3. Use Delimiters: Use clear delimiters such as triple quotes """, XML tags <tag>, or Markdown ### to separate instructions, context, and input data, avoiding confusion.

   ### INSTRUCTION ###
   Based on the customer review provided below, extract the product name and the customer's sentiment (positive/negative/neutral).
   
   ### REVIEW ###
   """
   The battery life of the SuperPhone Pro I just bought is amazing!
   """
   

4. Provide Examples (Few-shot Learning): For complex or novel tasks, providing examples is one of the most effective ways to guide the model.

   Rewrite the sentences in a more formal business email style.
   
   Sentence: Hey buddy, is that report done?
   Rewrite: Hello, may I ask about the current progress of the report on Project XX?
   
   Sentence: The boss said it must be submitted by next Wednesday.
   Rewrite:
   

   The model will follow your format and output something like "Per the Director's request, please ensure the relevant report is submitted by next Wednesday (MM/DD)."

5. Chain-of-Thought (CoT): This is a revolutionary technique for improving LLM performance on complex reasoning tasks (such as math problems and logic puzzles). The core idea is to have the model write out its problem-solving steps one by one before giving the final answer.

   • Zero-shot CoT: Simply append the magical phrase "Let's think step by step." to the end of your prompt.
   • Few-shot CoT: Also include detailed reasoning processes in the examples you provide.

   Question: A grocery store has 5 apples. They buy 2 boxes of apples, each box containing 6 apples. How many apples do they have now?
   Answer: Let's think step by step.
   1. The store initially has 5 apples.
   2. They buy 2 boxes, each with 6 apples, so they bought 2 * 6 = 12 new apples.
   3. They now have a total of 5 + 12 = 17 apples.
   So the final answer is 17.
   
   Question: Roger has 5 tennis balls. He buys 2 cans of tennis balls, each can containing 3 balls. How many tennis balls does he have now?
   Answer:
   

   The model will mimic your reasoning process and solve the problem more accurately.

6. Specify the Output Format: To make parsing easier for programs, force the model to output in a specific format (such as JSON, XML, or a Markdown table).

   "Please return the result as a JSON array, where each object contains 'name' and 'capital' fields."

7.3.3 Iteration and Optimization

Prompt Engineering is not something you get right in one go; it is a process of continual experimentation and iteration. When results fall short, you can try:

Analyzing the model's erroneous outputs and considering where the misunderstanding arose.

Adding more explicit instructions or constraints.

Providing higher-quality or more diverse examples.

Attempting to reframe the question or change the angle of approach.

7.4 Limitations and Challenges of LLMs: Hallucination, Bias, and Safety

Although LLMs are powerful, they are far from perfect. As responsible AI engineers, we must soberly recognize and properly handle their inherent limitations.

7.4.1 Hallucination

Definition: Hallucination refers to an LLM generating information that appears plausible but is in fact false, inconsistent with the facts, or unrelated to the source text. It is one of the most notorious problems with LLMs.

Causes:

• Knowledge Cutoff: The LLM's knowledge is "frozen" at the cutoff point of its training data. It knows nothing about new events after that point, yet it may "fabricate" answers.
• Training Objective: The LLM's training objective is to predict the next most likely word, not to "tell the truth." If false information frequently appears in a fluent form in the training data, the model may learn to generate it.
• Errors in Source Data: The training data itself contains a large amount of error, contradiction, and outdated information.

Coping Strategies:

• Retrieval-Augmented Generation (RAG): This is currently the most effective method for countering hallucination. Before asking the LLM a question, first retrieve relevant text segments from a reliable, updatable knowledge base (such as company documents or databases), inject these segments into the prompt as context, and require the LLM to answer based on the provided information. We will study RAG in detail in Chapter 9.
• Explicit Constraints in the Prompt: "Please answer only on the basis of the provided context. If the context contains no relevant information, simply say 'I don't know.'"
• Fact-Checking: For key generated information, cross-verify through search engines, databases, or other reliable sources.

7.4.2 Bias

Definition: LLMs are trained on vast amounts of internet text, which inevitably reflects the various biases present in human society, such as gender, racial, regional, and occupational stereotypes. The model will learn and amplify these biases.

Manifestations:

When asked to describe "a programmer," the model may be more likely to use masculine pronouns.

It may give culturally biased answers to questions asked from different cultural perspectives.

It may generate harmful or discriminatory content.

Coping Strategies:

• Data Cleaning and Debiasing: During the pretraining stage, carefully filter and process the data to reduce biased content. This is an extremely difficult challenge.
• Instruction Tuning and RLHF: In the instruction tuning and RLHF stages, "correct" the model's behavior by using data created by human annotators from diverse backgrounds, aimed at countering bias.
• Guidance in the Prompt: "Please describe the topic in a neutral, unbiased tone..."
• Output Screening: Before presenting model outputs to users, filter out potentially biased and harmful content using a content-review model or rule-based system.

7.4.3 Safety

Definition: This encompasses both the risk that LLMs will be used for malicious purposes and the harmful outputs they may themselves produce.

Malicious Use: Generating phishing emails, spreading disinformation, writing malicious code, and the like.

Harmful Outputs: Generating violent, pornographic, or hateful content, or providing dangerous instructions (such as how to manufacture weapons).

Jailbreaking: Users bypass the model's safety restrictions through carefully crafted "jailbreak prompts," luring it into generating content it should not produce.

Coping Strategies:

• Red Teaming: Before a model is released, a dedicated team plays the role of attackers, trying various methods to "attack" the model and uncover its security vulnerabilities.
• Multi-Layer Safety Filters: Install safety filters at both the input and output ends.
• Continuous RLHF: Use discovered "jailbreak" cases and harmful outputs as new training data, progressively reinforcing the model's safety guardrails through RLHF.
• Usage Restrictions and Monitoring: Enforce API usage policies to restrict high-risk applications and monitor suspicious calling patterns.

Chapter Summary

In this chapter, we sketched a grand panorama of large language models, officially stepping into the LLM era.

We reviewed the magnificent history of evolution from BERT to the GPT series, understanding the design philosophies and applicable scenarios of the three major architectural branches — Encoder-only, Decoder-only, and Encoder-Decoder — providing us with a genealogy through which to understand the "origins" of all modern LLMs.

We surveyed the flourishing open-source LLM ecosystem, learning how to make sound technology selection decisions among mainstream models like Llama, Gemma, and Qwen, in light of project requirements and resource constraints.

We delved into the art of communicating effectively with LLMs — Prompt Engineering. From basic components to advanced techniques such as Chain-of-Thought (CoT), we mastered how to use carefully designed prompts to bring the LLM's potential to full expression.

Finally, with a critical and responsible gaze, we confronted the formidable challenges facing LLMs: hallucination, bias, and safety. We understood the roots of these problems and explored the core strategies the industry currently employs to meet them.

After completing this chapter, your understanding of LLMs has grown from scattered concepts into a systematic framework. You are no longer an outsider to LLMs, but an insider who understands their history, present, and future. This macro-level understanding will serve as a solid intellectual foundation for your subsequent advanced practice in LLM fine-tuning, RAG, and agent development. In the next chapter, we will move from "using" LLMs to "customizing" them, learning how to turn a general-purpose LLM into an expert in your specific domain through fine-tuning techniques.