Tokens, Embeddings, and Context
Words are not tokens. Tokens are not meaning. A context window is not memory. Model parameters are not the chat.
Five inequalities
Words are not tokens. Tokens are not meaning. Embeddings are not database rows. A context window is not memory. Model parameters are not the chat.
That is the public mental model. Keep it. The rest of this page is why those sentences are true.
AI is a category of machine-based systems. [nist-ai-glossary] Tokens are how some sequence models slice input. They are not the definition of AI. A search engine, a rule list, and a planner can all be AI systems without this pipeline. See What is AI? and AI Is More Than Machine Learning.
Everyday example: type "I don't know"
Type "I don't know" into a chat box. Or type an emoji and a name. You typed characters you can read. The model does not receive those characters as English words.
A tokenizer chops the string into tokens. A token might be a word, a piece of a word, punctuation, a space-marked fragment, or a short byte sequence. The contraction, the apostrophe, and a following name can split in ways that surprise a reader. A rare name or a code snippet can cost more tokens than it looks.
Different tokenizers chop the same sentence differently. Byte-pair encoding, WordPiece, SentencePiece, and byte-level schemes exist. They are not interchangeable, and none of them is "the" tokenizer. [sennrich2016] [kudo2018]
Tokens are not meaning
Tokens are units for the model, not units of meaning.
The model then maps tokens to embeddings, which are vectors, lists of numbers used as a representation. It uses context, the finite window of tokenized input in this pass, to score possible next tokens. That is not a mind. It is not a database lookup of "the meaning of the word."
Nearby vectors, under a chosen metric such as cosine similarity, are a geometric fact about those vectors. Closeness is not proof of human meaning. [mikolov2013]
For how a transformer language model scores the next token, see Large Language Models. That lesson already has the tokenizer playground and the next-token math. This page is the distinction, not a second transformer chapter.
A context window is not memory
If the product shows a long chat, only a bounded slice of tokenized input, plus whatever the product retrieves or stores outside the model, is what this call can use.
The model can attend to a finite amount of tokenized input at once. Products state a limit. This page does not invent a window size, a vocabulary size, or a parameter count. If a number is unpublished, it stays unpublished. How to Check an AI Claim already taught that.
Chat history that scrolled out of the window is not "remembered" by the weights. Saving a thread, pinning a note, or searching your files is a product feature. A retrieval system (RAG) is search beside the model, not extra memory inside the weights. See RAG: Retrieval-Augmented Generation.
Parameters are not the conversation
The published model is a set of parameters from training. Training changes those weights. Your prompt is input at inference. A conversation does not, by default, rewrite the published model.
Saving a chat is a product feature, not the model updating itself. Some products log, store, or fine-tune. Data handling depends on the product, the account, the contract, and the settings. This lesson does not claim that no product ever stores logs. It also does not claim that every chat fine-tunes the model.
Where that model sits among hardware, software, and data is the subject of Hardware, Software, and Models.
What you can say after this lesson
- Words ≠ tokens
- Tokens ≠ meaning
- Embeddings ≠ database rows
- Context window ≠ memory or understanding
- Model parameters ≠ conversation history
- Related lessons: Large Language Models for transformers and next-token scoring. Data: What Machines Actually Learn From for "the dataset is a sample." RAG: Retrieval-Augmented Generation for retrieval as search. What is AI? for the category. AI Is More Than Machine Learning for methods that do not use this pipeline. How to Check an AI Claim if someone quotes a window size they cannot source.
References
National Institute of Standards and Technology (NIST) (2026). Artificial Intelligence — Glossary. NIST Computer Security Resource Center.
Organisation for Economic Co-operation and Development (OECD) (2024). Explanatory memorandum on the updated OECD definition of an AI system. OECD Artificial Intelligence Papers.
Sennrich, R., Haddow, B., and Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL).
Kudo, T., and Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations.
OpenAI (2022). tiktoken. GitHub repository.
Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv preprint.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS).
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems (NeurIPS).
Citation Note: Some citations are open access (arXiv or DOI, where linked). Others are books or journal articles that may sit behind a publisher paywall. Use the linked DOI or arXiv when available. If you notice any citation errors, please let us know.