🖥️ AI InfrastructureID:
context-memory-serviceContext & Memory Services
上下文缓存与记忆服务
🎯Core Definition
context and memory services reuse prefix KV caches across requests (prefix KV caching service) and provide tiered memory — short-term working memory plus long-term storage/retrieval.
💡Use Cases
multi-turn chat, multi-tenant shared system prompts, long-horizon memory agents; prompt caching and memory tiers are classic high-concurrency AI system design questions.
⚡Key Problems Solved
full per-turn prefill makes TTFT and cost grow linearly with context length; prefix caching cuts prefill compute from to (TTFT down 80%+), and tiered memory (short-term context window + long-term external store with retrieval) compresses long-session cost to “window + cache hits” scale.
🎯5 High-Frequency Exam Points
1
How does prefix KV caching reuse work across requests, and how does hit rate affect TTFT and cost?
2
How are short-term and long-term memory tiered (working memory vs external vector store)?
3
How do retrieval services (RAG) integrate with memory services, trading off relevance vs latency?
4
How to control long-session cost (window trimming, summarization, cached prefixes)?
5
How are prefix caches invalidated and rebuilt when system prompts or history change?
📖 In-depth Guide:📄 high-concurrency-ai-system →
Updated 2026-08-12🎯
🚀 Start Card Practice ➔Test Your Knowledge: Practice Questions for "Context & Memory Services"
Single choice pitfall questions with instant feedback and mistake tracking.