Large Language Models (LLMs)
What the model is, what the product adds, and what we actually know
What is a large language model?
A large language model is a machine-learning model trained on large collections of sequences so that it can model relationships among tokens and generate or score sequences.
Many modern text-generating LLMs are trained with an objective closely related to predicting the next token from previous context. That training objective is simple to state, but the learned model can support many tasks: drafting, summarizing, translating, coding, question answering, classification, extraction, and more.
Tokens, not words
Models usually process tokens rather than whole words. A token may be a word, part of a word, punctuation, or another encoded unit.
Tokenizer Playground
See how LLMs break text into tokens - the fundamental units of language processing
Token IDs:
Why Tokenize?
LLMs can't process raw text - they need numbers. Tokenization converts text into integers that neural networks can understand. Good tokenization balances vocabulary size with semantic meaning.
BPE (Byte-Pair Encoding)
Most modern LLMs (GPT, LLaMA) use BPE - it merges frequently occurring character pairs to create subword tokens. This handles rare words gracefully while keeping common words as single tokens.
⚠ Token Limits
LLMs have context windows measured in tokens (e.g., 8k, 32k, 128k). Efficient tokenization means more text fits in context. GPT-4 uses ~100k vocabulary with ~0.75 tokens per word on average.
A model is not the whole product
This distinction matters:
Base or post-trained model
- Neural-network weights
- Tokenizer and model architecture
- Learned statistical structure from training and post-training
Deployed AI system
- Model
- System instructions
- Search or retrieval
- Tools and code execution
- Memory or saved preferences
- Safety controls
- User/account permissions
- External databases and APIs
- Product interface
A statement such as “LLMs have no current information” may be true of a model used by itself, but false of a product that can search the web or query a live database. The same applies to memory, calculation, file access, and tool use.
What training does
Pretraining adjusts model parameters so that the model becomes better at its training objective across many examples. Modern training corpora can include text, code, and—in multimodal systems—other modalities represented in model-compatible forms.
Post-training can then change model behavior. Depending on the system, this may include supervised fine-tuning, preference optimization, reinforcement learning, safety training, distillation, or other techniques. RLHF is one important technique, not a definition of fine-tuning or the only post-training method.
What LLMs are good at
Depending on the model, prompt, tools, and evaluation setting, LLM-based systems can be useful for:
- Drafting and rewriting
- Summarization and extraction
- Translation
- Code generation and explanation
- Question answering
- Classification
- Structured data transformation
- Multi-step tasks when paired with tools and verification
Performance is not uniform. A model can be strong on one benchmark or workflow and weak on another.
What LLMs can get wrong
Hallucination and unsupported claims
LLMs can generate fluent statements that are false, unsupported, or inconsistent with a source. There is no single universal hallucination rate. Error rates depend on the model, task, benchmark, prompt, available context, tools, and how “hallucination” is defined.
Reasoning
It is inaccurate to reduce every model behavior to “no reasoning.” Modern systems can solve some multi-step reasoning tasks and fail badly on others. Researchers continue to debate what mechanisms produce these behaviors and how reliably they generalize.
For practical use, treat reasoning ability as empirical and task-specific: test the system on the work you care about.
Understanding and consciousness
Whether model behavior should be described as “understanding” depends partly on the definition being used. Behavioral competence does not, by itself, establish subjective experience or consciousness. AIAIMate does not present philosophical questions about machine understanding or consciousness as settled scientific facts.
Knowledge, memory, and tools
A model's learned parameters are not a live database. But a deployed system may add:
- Web search
- Retrieval-augmented generation (RAG)
- Databases
- Calculators or code execution
- Persistent memory
- User-provided files
Always ask whether a claim refers to the model alone or the full system.
References
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS).
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems (NeurIPS).
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., et al. (2022). Training Compute-Optimal Large Language Models. Advances in Neural Information Processing Systems (NeurIPS).
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS).
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. FAccT 2021.
Citation Note: Some citations are open access (arXiv or DOI, where linked). Others are books or journal articles that may sit behind a publisher paywall. Use the linked DOI or arXiv when available. If you notice any citation errors, please let us know.