Context & Memory Services

context and memory services reuse prefix KV caches across requests (prefix KV caching service) and provide tiered memory — short-term working memory plus long-term storage/retrieval.

multi-turn chat, multi-tenant shared system prompts, long-horizon memory agents; prompt caching and memory tiers are classic high-concurrency AI system design questions.

full per-turn prefill makes TTFT and cost grow linearly with context length; prefix caching cuts prefill compute from $O(L_{\text{total}})$ to $O(L_{\text{new}})$ (TTFT down 80%+), and tiered memory (short-term context window + long-term external store with retrieval) compresses long-session cost to “window + cache hits” scale.

How does prefix KV caching reuse work across requests, and how does hit rate affect TTFT and cost?

How are short-term and long-term memory tiered (working memory vs external vector store)?

How do retrieval services (RAG) integrate with memory services, trading off relevance vs latency?

How to control long-session cost (window trimming, summarization, cached prefixes)?

How are prefix caches invalidated and rebuilt when system prompts or history change?