Large Language Models (LLMs)

What are Large Language Models?

Core definition: LLMs are AI systems optimized for predicting the next token in a sequence.

This seemingly simple objective enables remarkable capabilities:

• Generating coherent essays and articles
• Answering questions across domains
• Translating between languages
• Writing functional code
• Engaging in dialogue

Training Process

Phase 1: Data ingestion

• Web pages, books, code repositories, conversations
• Hundreds of billions to trillions of tokens
• Requires months of computation and significant resources

Phase 2: Pattern extraction

• "After 'The cat sat on the...' likely follows 'mat' or 'floor'"
• "Questions beginning with 'How to...' typically yield procedural answers"
• "Code starting with 'function...' usually includes '' syntax"

Phase 3: Text generation

• Input: "Write a poem about dogs"
• Model: Accesses learned patterns about poetry structure
• Output: "Golden fur in morning light..."
• Process: Predicts each subsequent token probabilistically

The Appearance of Understanding

LLMs are sufficiently skilled at pattern prediction that their outputs appear to reflect understanding. However, they lack genuine comprehension.

A useful analogy: an extremely sophisticated autocomplete system that has processed virtually all human text and can recombine patterns convincingly. Impressive capability? Certainly. Conscious understanding? No.

Scale and Parameters

Parameters: The "knobs" the AI adjusts while learning

• Small model: 1 million parameters (can barely form sentences)
• Medium model: 1 billion parameters (can chat reasonably)
• Large model: 100+ billion parameters (can fool you into thinking it's human)

More parameters = better predictions (usually)