TalentMe Science Coding: Deep Learning & LLM Core Coding Engine
105 essential kernels covering activations, Transformer mechanics, alignment RL, distributed parallelism, inference serving, and eval calibration—with math derivations, assertions, and offline export.
Science Coding Mastery Tracker
0% DoneLocally synced in your browser. Track, review, and master all industrial kernels.
Numerically Stable Softmax
Numerically stable implementation subtracting the axis maximum to prevent exponent overflow while preserving dimensional broadcasting.
Numerically Stable Sigmoid & Log-Sigmoid
Piecewise formulation avoiding overflow on large negatives; Log-Sigmoid powered by logaddexp for robust BCE training.
Gaussian Error Linear Unit (GELU)
Stochastic regularizer gating activation based on standard Gaussian CDF, standard in BERT and GPT-2.
SwiGLU Gated Feed-Forward Network
Flagship FFN operator powering LLaMA and DeepSeek, combining SiLU gated branch with up-projection and down-projection.
Softplus & LeakyReLU with Overflow Guards
Softplus with threshold clipping against overflow, alongside LeakyReLU negative slope leak.
Layer Normalization with Affine Transformation
Per-token normalization across the feature dimension with consistent train/eval behavior and learned gamma/beta.
Root Mean Square Normalization (RMSNorm)
Omit mean centering and normalize solely by root-mean-square to save memory passes in modern LLMs.
BatchNorm1d with Train vs Eval Distinction
Track batch statistics with momentum into running stats during training; freeze and apply running stats at inference.
Group Normalization & Instance Normalization
Partition channels into G groups and normalize independently per sample, removing batch-size dependence.
Inverted Dropout with Train/Eval Scaling
Scale kept activations by 1/(1-p) during training so expectations match without extra work at inference.
Sinusoidal Positional Encoding
Classic Transformer fixed sine/cosine encodings across geometric wavelengths enabling linear relative shifts.
Rotary Position Embedding (RoPE)
Apply 2D pairwise rotations to Q and K so dot products depend strictly on relative token distance.