Wiring Android's MediaPipe LLM Inference API to a Multi-Turn Conversation Engine
--- title: "Android MediaPipe LLM: Session State, KV-Cache, and the Context Budget That Keeps Latency Flat" published: true description: "Deep dive into MediaPipe LLM session management, KV-cache reuse, and context budgeting to keep P99 latency flat on mid-range Android devices." tags: android, kotlin, mobile, architecture canonical_url: https://mvpfactory.co/blog/android-mediapipe-llm-session-state-kvcache --- ## What You Will Build By the end of this tutorial you will have a multi-turn `ConversationEngine` backed by MediaPipe's `LlmInference` API that holds its session open across turns, enforces a token budget with a sliding window, and handles LoRA adapter switching without destroying your latency profile. Building a single-turn prompt wrapper is easy. Building one that stays responsive across 20 exchanges on a Pixel 6a — that is what we are here for. --- ## Prerequisites - Android project targeting API 26+ - MediaPipe Tasks dependency added to your `build.gradle` - A quantised on-device model (Gemma 2B INT4 is the reference throughout) - Basic familiarity with Kotlin coroutines --- ## The Core Problem: Session Lifecycle Here is the gotcha that will save you hours. Most teams treat `LlmInference` like a stateless HTTP endpoint and re-create it on every turn. That nukes your KV-cache on every call and forces the model to re-process the entire conversation prefix from scratch — every single time. The `LlmInference` API maintains an internal KV-cache storing key-value attention pairs for all previously processed tokens. MediaPipe does not expose a stateful "add turn" API — you pass the full accumulated prompt on each call. But as long as you hold the session open, the runtime uses prefix cache matching: tokens the model has already seen at the same position are served from cache. Only net-new tokens at the tail require full attention computation. **You pass full history. The session only computes the delta.**
kotlin
// WRONG: session torn down each turn — full re-inference on every call
fun respondToUser(context: Context, options: LlmInference.Options, input: String): String {
val session = LlmInference.createFromOptions(context, options)
return session.generateResponse(input).also { session.close() }
}
// CORRECT: session held open across turns
class ConversationEngine(context: Context, options: LlmInference.Options) {
private val session: LlmInference = LlmInference.createFromOptions(context, options)
private val history = mutableListOf()
fun respond(userInput: String): String { history.add("User: $userInput") val prompt = history.joinToString("\n") val response = session.generateResponse(prompt) history.add("Model: $response") return response } fun close() = session.close() }
--- ## Context Window Budget: The Latency Cliff Context growth is not a linear problem — it is quadratic in attention cost. These are illustrative order-of-magnitude estimates based on profiling patterns for a Gemma 2B INT4 model on a Snapdragon 778G versus a Snapdragon 8 Gen 2: | Context Tokens | TTFT (Flagship) | TTFT (Mid-range) | P99 Risk | |---|---|---|---| | 256 | ~180ms | ~320ms | Low | | 512 | ~260ms | ~610ms | Moderate | | 1024 | ~440ms | ~1,400ms | High | | 2048 | ~850ms | ~3,200ms+ | Budget exceeded | If you target devices outside the flagship tier, your effective usable context budget is roughly **512–768 tokens** before P99 crosses the 1-second user-perception threshold. Budget by tokens, not turns. Use `sizeInTokens()`. Here is the minimal setup to get this working:
kotlin
fun buildPromptWithBudget(
session: LlmInference,
history: List,
budget: Int = 600
): String {
val builder = ArrayDeque(history)
fun tokenCount(text: String) = try {
session.sizeInTokens(text)
} catch (_: UnsupportedOperationException) {
text.length / 4
}
var tokenEstimate = tokenCount(builder.joinToString("\n"))
while (tokenEstimate > budget && builder.size > 2) {
tokenEstimate -= tokenCount(builder.removeFirst())
}
return builder.joinToString("\n")
}
Keep your system prompt pinned outside the sliding window and account for its token cost separately. The 4-chars-per-token heuristic is a reasonable fallback for Latin-script prose but breaks down for code-heavy or multilingual content. --- ## LoRA Switching Is a Session Reset Let me show you a pattern I use in every project that handles LoRA adapters. Switching adapters always invalidates the KV-cache — there is no partial cache preservation across adapter boundaries in the current MediaPipe contract. Design your conversation flow to group turns by mode before transitioning:
kotlin
sealed class ConversationMode {
object General : ConversationMode()
object CodeAssist : ConversationMode()
}
class AdaptiveSession(private val context: Context, private val baseOptions: LlmInference.Options) {
private var currentMode: ConversationMode? = null
private var session: LlmInference? = null
fun ensureMode(mode: ConversationMode): LlmInference { if (mode != currentMode) { session?.close() session = LlmInference.createFromOptions(context, baseOptions) currentMode = mode } return session!! } }
Batch all turns of a given mode together. Scattered adapter switches throughout a session are a P99 latency time bomb. --- ## Gotchas **KV-cache invalidation triggers to avoid:** - Calling `close()` and re-creating the session between turns - Swapping LoRA adapters mid-session without a full reset - Exceeding the model's `maxTokens` window (the cache is fully evicted) - Modifying the system prompt between turns **The docs do not mention this, but** heap pressure from NNAPI delegate allocation is the silent killer of session stability. Use Android Studio's Memory Profiler with the **NNAPI heap** category enabled. If that heap climbs without bounds across turns, you have a session lifecycle leak — not a context window problem. --- ## Conclusion Three things worth repeating: 1. **Hold your session open.** Session re-creation is the single highest-impact latency regression most teams ship without realising it. 2. **Budget by tokens, not turns.** Use `sizeInTokens()`, set a hard ceiling of 512–768 tokens for mid-range targets, and preserve your system prompt outside the sliding window. 3. **Treat LoRA switches as session resets.** Make them predictable and infrequent. On-device LLM work involves long debugging sessions — I keep [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) running in the background to catch myself before hour three without moving. Small thing, real difference. For further reading: [MediaPipe LLM Inference API docs](https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference/android).