P2MediumPart P · MLE Production Engineering & Calibration
Dual-Budget Token & Item Batch Packer for Async Embeddings
Pack variable-length documents under simultaneous item-count and token-memory ceilings to maximize GPU throughput without OOM.
⏱️ Time:
O(N)💾 Space:
O(N)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
def pack_token_bounded_batches(doc_tokens: list[int], max_batch_items: int, max_batch_tokens: int) -> list[list[int]]:
batches: list[list[int]] = []
cur_batch: list[int] = []
cur_tokens = 0
for idx, tok in enumerate(doc_tokens):
t =
if t > max_batch_tokens:
if cur_batch:
batches.append(cur_batch)
cur_batch, cur_tokens = [], 0
batches.append([idx])
continue
if cur_batch and (len(cur_batch) >= max_batch_items or cur_tokens + t > max_batch_tokens):
batches.append(cur_batch)
cur_batch, cur_tokens = [], 0
cur_batch.append(idx)
cur_tokens += t
if cur_batch:
batches.append(cur_batch)
return batches
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why sort chunks by length before batching during massive RAG backfills?
Q2: How do you prevent host RAM spikes if Kafka consumption outpaces GPU encoding?
💡
Core Mental Anchor / Mnemonic
Flush on item count OR token budget; isolate oversized items cleanly
📐 Mathematical Derivation & Core Formula
$$\text{batches} = \left\lceil \frac{\sum_{i \in B} \text{tokens}_i \le T_{\max} \land |B| \le B_{\max}}{\text{flush}} \right\rceil$$
Transformer activation memory scales with batch tokens rather than document count. Greedily flushing whenever
len >= B_max or tokens + t > T_max (and isolating single oversized documents) bounds peak activation memory across heterogeneous ingestion queues.🔄 Tensor Dimensions & Shape Flow
doc_tokens: List[int] -> List[List[int]] micro-batch index groups
🛡️ Numerical Stability & Pitfalls Checklist
- Isolate single oversized items cleanly so they never stall or break loop invariants
- Pair with length sorting over a lookahead buffer to minimize intra-batch padding waste
- Enforce bounded async queue capacity to apply upstream backpressure
💻 Reference Code Implementation
def pack_token_bounded_batches(doc_tokens: list[int], max_batch_items: int, max_batch_tokens: int) -> list[list[int]]:
batches: list[list[int]] = []
cur_batch: list[int] = []
cur_tokens = 0
for idx, tok in enumerate(doc_tokens):
t = max(1, int(tok))
if t > max_batch_tokens:
if cur_batch:
batches.append(cur_batch)
cur_batch, cur_tokens = [], 0
batches.append([idx])
continue
if cur_batch and (len(cur_batch) >= max_batch_items or cur_tokens + t > max_batch_tokens):
batches.append(cur_batch)
cur_batch, cur_tokens = [], 0
cur_batch.append(idx)
cur_tokens += t
if cur_batch:
batches.append(cur_batch)
return batches🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
toks = [100, 120, 90, 500, 50, 50]
res = pack_token_bounded_batches(toks, max_batch_items=2, max_batch_tokens=250)
assert res == [[0, 1], [2], [3], [4, 5]], f'Got {res}'
print('P2 PASS')🎯 Core Follow-up Inquiries & Reference Answers
Q1:Why sort chunks by length before batching during massive RAG backfills?
Dynamic padding forces every item in a batch to match the longest sequence; bucketing similar lengths eliminates wasted compute on zero pads.
Q2:How do you prevent host RAM spikes if Kafka consumption outpaces GPU encoding?
Use a bounded semaphore/asyncio.Queue(maxsize) so the consumer pauses fetching new offsets when in-flight batches saturate.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.