KVCache Meetup Korea 2026: Event Recap

On the evening of Sep 30, 2026, PyTorchKR and XCENA co-hosted KVCache Meetup Korea. PyTorchKR is a major AI/ML community in Korea, and XCENA is a startup that focuses on CXL (Compute Express Link). Tensormesh, the company behind LMCache, also sponsored the event. Speakers from XCENA, SK hynix, FuriosaAI, and LMCache shared their work on large language model (LLM) inference and KV Cache optimization. LLM inference runs a language model to generate output. KV Cache stores attention data for reuse during inference.

 
Attendees at KVCache Meetup Korea 2026.

Talks covered CXL memory expansion, GPU and high-bandwidth memory (HBM) bottlenecks, request scheduling on RNGD, and LMCache MP Mode. The sessions showed how hardware and serving software handle KV Cache in different ways. Serving software processes requests that run a model.

Registration and Opening

The program combined technical talks with opportunities for attendees to connect. Attendees who wanted to join the networking chose up to two interest stickers at registration. The five categories were Model, Serving, Infra, Hardware, and Memory.

 
Attendees chose their interest stickers at registration.

PyTorchKR organizer Junghwan Park and XCENA CPO Juhyun Kim opened the event. The speakers were Jooho Lee from XCENA, Sejin Oh from SK hynix, Donghun Lee from FuriosaAI, and Baoloong Mao from LMCache. They shared their experiences and approaches.

   
Junghwan Park from PyTorchKR (left) and Juhyun Kim, CPO at XCENA (right), welcomed the attendees.

CXL and KV Cache Storage: Jooho Lee, XCENA

The session “CXL for KVCache” started with the basics of CXL and memory expansion. Jooho Lee explained how CXL expands the memory space that CPUs can access with load/store instructions. He also described a setup where multiple hosts use a memory pool and introduced Maru. Maru is a high-performance KV Cache storage engine that XCENA released as an open-source project. It helps LLM inference use shared CXL memory efficiently.

Jooho Lee from XCENA presented “CXL for KVCache.”

The KV Cache design combined DRAM and SSD storage in a CXL memory pool. It kept frequently used cache in DRAM and less frequently used cache on SSDs. Maru connects the inference engine to these storage layers. Optimizing LLM inference requires more than storing a large amount of cache. The path and cost of moving cache to the GPU also matter.

The talk also covered splitting computation for experts in mixture-of-experts (MoE) models between a GPU and MX1. Experts are specialized parts of a model. The design places some weights in memory near MX1 and processes them there. It transfers intermediate values, called activations, and computation results instead of the full weights. The session started with KV Cache storage and also covered weight placement and ways to reduce data movement.

GPU Compute and HBM Bandwidth: Sejin Oh, SK hynix

Sejin Oh presented “Analyzing Utilization of GPU and HBM in LLM Serving Systems.” The talk examined how prefill and decode use system resources. Prefill processes the prompt, and decode generates output tokens. The session presented Roofline analysis alongside measurements from a running system. Roofline analysis compares compute and memory bandwidth limits.

Sejin Oh from SK hynix presented the analysis of GPU compute and HBM bandwidth.

The experiments used two H100 GPUs, Llama 3.1 8B, and vLLM. GPU compute use was high during the main projection operations in prefill. During decode, HBM bandwidth use was high. The experiments also explained the difference between TTFT and TPOT. TTFT measures the time to the first output token. TPOT measures the time per output token.

Reusing KV Cache reduces computation during prefill. After that reduction, memory bandwidth can become the bottleneck. Whether a service needs more memory capacity or bandwidth depends on its input and output lengths, batch size, and other operating conditions.

Scheduling and KV Cache on RNGD: Donghun Lee, FuriosaAI

FuriosaAI is building Furiosa-LLM, its own inference engine for running LLMs on RNGD. In “LLM Inference Engine for RNGD,” Donghun Lee shared engineering problems from building the engine, including request scheduling and KV Cache management. He followed each request from arrival to execution. He explained PagedAttention for KV Cache blocks, Continuous batching for active requests, and Chunked prefill for smaller prompt segments. He also introduced buckets, or tensor shapes compiled before execution.

Donghun Lee from FuriosaAI explained how to share KV Cache outside a node.

Combining requests into one batch increases throughput, or the number of requests processed over time. Larger batches can need larger buckets and increase latency, or response delay. Furiosa-LLM measures the execution cost on the actual hardware. It merges two batches when the combined cost is lower than processing them separately. The decision also accounts for the context length of each request and the cost of padding. Padding fills unused positions in a fixed tensor shape.

The talk compared the benefit of cache reuse with the cost of transferring it. The slides used a 12,370-token prompt, with one request at a time and three measurements for each condition. Retrieving the host cache and generating the first token took about 92 ms. Recomputing the cache took about 900 ms. Under those test conditions, retrieving the host cache was faster. He also explained how to overlap cache transfer with computation at each layer.

The benefit depends on the amount of data transferred and the available bandwidth. Experimental features included a network-shared cache and running prefill and decode on separate resources. He also discussed plans for future expansion.

‘LMCache: A Unified KV Cache Management Layer with Multi-Platform Architecture’: Baoloong Mao, LMCache

LMCache maintainer and core architect Baoloong Mao presented “A Unified KV Cache Management Layer with Multi-Platform Architecture.” LMCache is an open-source KV Cache project created by the team that founded Tensormesh. He explained how LMCache connects different inference engines with storage systems and hardware. He focused on MP Mode, which moves cache management from inside the inference engine into a separate process.

Baoloong Mao from LMCache introduced the unified KV Cache layer and MP Mode.

In MP Mode, CUDA IPC allows processes to share GPU memory. IPC means communication between processes. The processes exchange control information such as block IDs. They use a shared CPU memory pool within a node to reduce duplicate storage. MP Mode also reads and loads data from several storage layers in parallel. This design separates inference computation from cache I/O, or data transfers, to reduce competition for resources.

The session also covered deployment and operations. Topics included deployment on Kubernetes and monitoring cache hit rates and cache lifetimes. A cache hit means that the requested data is available. The talk also covered isolating failures between inference and cache processes. External plugins allow connections to more storage systems and devices.

Q&A and Networking

In the Q&A session, the speakers returned to the stage to answer questions from attendees. Questions included separating prefill and decode, compared with mixed batching. Mixed batching processes prefill and decode work together. Attendees also asked about the roles of LMCache and the cache features built into vLLM. Another topic was the plan to extend MP Mode across multiple nodes.

         
The speakers answered live questions together after the talks.

After the Q&A, attendees from different fields gathered in small groups to talk. Their interest stickers helped them find people with similar interests. Speakers and attendees exchanged ideas and heard different perspectives.

Conversations also continued around the CXL server from XCENA at the venue. Attendees examined the hardware up close and asked questions. They linked the memory expansion techniques from the talks to the hardware on display.

 
Attendees talked during the networking session.

Feedback from 72 Survey Responses

The following summary draws on 72 survey responses. All numbers and percentages below use the survey responses as their base. Some questions allowed multiple selections.

Event satisfaction and work experience by role, based on 72 survey responses.

In the feedback questions, 48 respondents selected the program as a positive point (66.7%). The venue and facilities received 30 selections (41.7%). The atmosphere for conversations and the event timing each received 22 selections (30.6%). This question allowed multiple selections. Respondents gave positive feedback on the talks and the chance to meet other attendees.

Repeated comments mentioned microphone quality and volume, screen visibility from the back rows, and limited space. Some respondents also asked for interpretation or translation to help them follow the English talks. Some respondents also asked for a dedicated session where developers in the same field can talk. Requests for future events included real production examples, vLLM and SGLang internals, compilers for neural processing units (NPUs), recent inference optimization techniques, and demos. Respondents wanted details about problems, trial and error, and solutions from real deployments. They asked speakers to explain these experiences, as well as their products.

Group photo from KVCache Meetup Korea.

PyTorchKR will use this feedback to plan and run future events. Thank you to everyone who joined the event on a weekday evening. The organizers also thank the speakers who shared their knowledge and experience. Thank you to PyTorchKR and XCENA for organizing the event, and to Tensormesh for its support.

The post KVCache Meetup Korea 2026: Event Recap appeared first on PyTorch.