Language Model Landscape
Updated 01.08.24The many pieces to pay attention to when it comes to deploying language models.
How LLMs work
The AI Insights & Reading Notes section is organized according to this structure. The corresponding chapter are linked below.
-
This remains the main interface through which to interact with LLMs, but the techniques have evolved - from few-shot learning to trees of thought etc.
-
The latest on the research on what LLMs are capable of in terms of step-by-step inferencing, planning, and solving more complex problems.
-
A lot of recent LLM performance improvement has simply come from scaling up. What do we know about the emergence of capabilities depending on how LLMs scale?
04: Scaling Laws & Capabilities: Which LLMs Perform How Well
-
If multimodality gives LLMs eyes, this gives them hands. You can teach LLMs to inject explicit external calls into their own output (like calls to APIs), and you can get them to write code. And then you can loop that onto itself, so LLMs can call LLMs and you’re teaching an LLM “tool use”.
-
Interpretability research is trying to understand how LLMs actually gain the capabilities they have. This comes in two flavors: a) which knowledge and which state of the world LLMs really store inside of them, and b) various techniques to look under the hood.
-
What do we know about limitations exhibited by LLMs? Which fundamental constraints on their performance exist?
-
GPT-4’s largest context window is 32K tokens, which is around 25 pages of text or so. That's not enough to really encode complex knowledge. Larger use cases require fine-tuning of open-source models.
-
This is the other way to get more area-specific knowledge into LLMs: don’t re-train them, but give them a way to look up external knowledge.
-
This is where most LLM applications will spend most of their time. Foundation models will become more and more commoditized, and cloud providers will enable better and better finetuning-as-a-service. Which means, to really make use of LLMs, you have to think hard about systems design: how do you prompt, store learnings, iterate on LLM output etc. This section includes clever examples for that.
13: Systems Design: How to Make LLMs Part of a Feedback Loop
-
Language models operate on text, so how can you get images and sound into them? You can get surprisingly far through embeddings, adapters and/or auto-formalization.