O4EasyPart O · BA-to-Python Data Science & Stats Kernels

Vectorized Out-of-Order Event Stream Sessionization (SQL LAG to NumPy)

⚡Flashcard Drill➔

Reconstruct session IDs from unordered clickstream events using np.lexsort, lagged gap flags, cumsum, and inverse permutation scatter.

⏱️ Time: O(N log N)
💾 Space: O(N)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def assign_session_ids(user_ids: np.ndarray, timestamps: np.ndarray, gap_threshold: float) -> np.ndarray:
n = len(user_ids)
if n == 0:
return
order = np.lexsort((timestamps, user_ids))
u_sorted = user_ids[order]
t_sorted = timestamps[order]
is_new = np.zeros(n, dtype=np.int64)
is_new[0] = 1
if n > 1:
user_switch = u_sorted[1:] != u_sorted[:-1]
time_gap = (t_sorted[1:] - t_sorted[:-1]) > gap_threshold
is_new[1:] = (user_switch | time_gap).astype(np.int64)
sess_sorted =
out = np.empty(n, dtype=np.int64)
out[order] = sess_sorted
return out
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why must mobile clickstream logs be re-sorted before sessionization?
Q2: How would you distribute this over 5 billion events?
Next: K-Means++ D-Squared Initialization for RFM & Codebooks →
💡

Core Mental Anchor / Mnemonic

lexsort puts primary last, OR user/gap switch, cumsum assigns IDs

📐 Mathematical Derivation & Core Formula

$$\text{new\_session}_i = \mathbb{I}\left(u_i \neq u_{i-1} \lor (t_i - t_{i-1}) > \tau_{\text{gap}}\right)$$
Translates SQL LAG(ts) + SUM(...) OVER into multi-key np.lexsort((timestamps, user_ids)), contiguous boundary boolean flags user_switch | (dt > threshold), prefix-sum ID assignment, and inverse scatter back to original row indices.

🔄 Tensor Dimensions & Shape Flow

user_ids: (E,), timestamps: (E,) -> session_ids: (E,)

🛡️ Numerical Stability & Pitfalls Checklist

  • Remember np.lexsort prioritizes the last key in the tuple
  • Scatter back via out[order] so downstream feature columns stay aligned with caller rows
  • Handle zero-event empty inputs cleanly without indexing errors

💻 Reference Code Implementation

import numpy as np

def assign_session_ids(user_ids: np.ndarray, timestamps: np.ndarray, gap_threshold: float) -> np.ndarray:
    n = len(user_ids)
    if n == 0:
        return np.array([], dtype=np.int64)
    order = np.lexsort((timestamps, user_ids))
    u_sorted = user_ids[order]
    t_sorted = timestamps[order]
    is_new = np.zeros(n, dtype=np.int64)
    is_new[0] = 1
    if n > 1:
        user_switch = u_sorted[1:] != u_sorted[:-1]
        time_gap = (t_sorted[1:] - t_sorted[:-1]) > gap_threshold
        is_new[1:] = (user_switch | time_gap).astype(np.int64)
    sess_sorted = np.cumsum(is_new) - 1
    out = np.empty(n, dtype=np.int64)
    out[order] = sess_sorted
    return out

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

u = np.array([2, 1, 1, 1, 2])
t = np.array([50.0, 10.0, 20.0, 60.0, 10.0])
sid = assign_session_ids(u, t, 30.0)
assert sid[1] == sid[2] and sid[2] != sid[3] and sid[4] != sid[0]
assert len(np.unique(sid)) == 4
print('O4 PASS')

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why must mobile clickstream logs be re-sorted before sessionization?
Offline retries and batched telemetry uploads deliver events out of chronological order; computing gaps without sorting corrupts sequential features.
Q2:How would you distribute this over 5 billion events?
Hash-partition or shuffle by user_id across workers first so every user's timeline fits locally within one executor.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: CUPED A/B Variance Reduction with Pooled Covariate ControlBack to Part O & All 105Next: K-Means++ D-Squared Initialization for RFM & Codebooks →