LLM Interview Questions

A developer specifies a model, objective, and training procedure; an optimizer learns parameters from examples. For instance, a spam classifier learns correlations between message features and labels rather than relying entirely on handwritten rules. Learning does not guarantee improvement on future data: success requires an appropriate objective, representative data, and evaluation on held-out examples.

Training and inference are separate activities. During training, a spam model computes a score from message features, compares it with the known label, and changes its weights to reduce a loss. At inference, the trained weights stay fixed while the model scores a newly arrived message. Updating the model requires another learning step; merely using it does not automatically teach it.

A useful formulation is : is the available input, the learned parameters, and the prediction. The designer chooses what information enters and what mistakes the loss penalizes. A classifier trained on old spam may learn yesterday’s vocabulary and fail on a new campaign. Evaluate chronologically held-out messages and choose a decision threshold reflecting the cost of blocking legitimate mail. The central achievement is generalization to new examples, rather than remembering the training set or discovering a causal explanation of every correlation.

These tasks contain patterns that are difficult to enumerate as rules. A recommender ranks items from interaction history; a speech model maps audio to text; a forecasting model uses past demand and known covariates. ML works best when the data contains predictive signal and mistakes can be measured. Compare against simple rules or statistical baselines before accepting the added operational complexity.

Consider demand forecasting for a grocery store. Inputs might include past sales, day of week, holidays, price, and planned promotions; the output is next week’s demand. A useful baseline predicts the same weekday’s recent average. ML becomes worthwhile when it captures interactions the baseline misses, such as a promotion having different effects on weekends. Historical stockouts complicate labels: recorded sales can be lower than actual demand.

For visual inspection, labeled images teach a classifier to distinguish acceptable parts from defects. The evaluation should hold out production batches or cameras, because random image splits can leak nearly identical backgrounds. For recommendations, the model estimates relevance from interactions, but clicks also reflect which items were previously shown. Offline accuracy therefore needs an online test of user outcomes.

Across these examples, the prerequisite is a repeated decision with observable feedback. Use rules when the condition is explicit and stable; use ML when examples contain useful patterns and the benefit exceeds the cost of collecting data, monitoring errors, and maintaining the pipeline.

Clustering can reveal customer segments, while principal component analysis compresses correlated measurements. Anomaly detection can flag unusual observations, but unusual does not automatically mean fraudulent or defective. Association-rule mining discovers co-occurring items. Because there are no target labels, validation often combines structural metrics, stability checks, and usefulness on a downstream task.

The absence of labels changes both the objective and the interpretation. K-means minimizes within-cluster squared distances, so it prefers roughly compact groups in the chosen feature space. If customer income is measured in dollars and visit count in single digits, unscaled income may dominate the result. Standardization and feature selection therefore express assumptions about what similarity should mean.

PCA instead finds orthogonal directions that retain the most variance. Compressing 100 correlated sensor measurements into 10 components can reduce storage and noise, but a low-variance direction might still be essential for detecting a rare fault. An anomaly detector asks whether a measurement is unusual relative to a reference distribution; it cannot conclude the cause without further evidence.

Validate by rerunning on different samples, checking whether groups remain stable, and testing a downstream use such as routing support requests. A pleasing cluster visualization is insufficient: the structure may reflect language, document length, or recording equipment rather than the phenomenon the application cares about.

The model learns a decision function from those examples and predicts labels or probabilities for new messages. In practice, semi-supervised learning can use unlabeled email, and anomaly detection can help find new campaigns. Evaluate precision and recall under realistic class imbalance; a high overall accuracy can hide costly false positives.

A labeled training row consists of the email representation and a target, for example spam = 1. The model can learn ; an application then compares that score with a threshold. Moving the threshold upward generally catches fewer spam messages but reduces accidental blocking of legitimate messages. The threshold is a product decision distinct from learning the probability model.

Suppose a test set contains 990 legitimate messages and 10 spam messages. A system that always predicts legitimate achieves 99% accuracy while detecting no spam. If another system flags 12 messages, of which 8 are spam, precision is and recall is . Those numbers make the tradeoff visible.

Build the holdout from a later time period and group duplicate campaigns to avoid an inflated score. User reports provide useful labels but are incomplete and selection-biased. Unsupervised clustering can discover a new campaign for investigation, yet assigning the spam decision still requires a definition of undesirable mail and appropriate validation.

A typical pipeline streams records, computes features with bounded memory, and updates an incremental estimator. The feature representation must remain consistent across chunks; fitting a new vocabulary independently on every chunk would break this contract. Mini-batch training enables out-of-core processing only if loading, preprocessing, and caching also avoid materializing the full dataset.

The memory budget must cover an entire pipeline, not just the estimator. Imagine a 200 GB log collection and 16 GB of RAM. Read a bounded block, parse it, transform features, update the model, and discard the block before reading the next. A hashing representation can keep the feature dimension fixed without storing an ever-growing vocabulary. An incremental linear classifier can update its coefficients through partial_fit; a conventional estimator whose fit requires the full matrix cannot simply be substituted.

Some preprocessing needs an additional pass. For standardization, first accumulate counts, means, and variances, then transform chunks using those fixed statistics during training. Alternatively, deliberately use an online procedure and account for its changing state. Repeated passes may improve optimization but increase disk traffic.

Order matters when adjacent chunks come from different dates or categories. Shuffle within a bounded buffer or interleave sources where appropriate. Keep a separate holdout stream, record the last consumed position for recovery, and distinguish learning from finite data on disk from adapting continuously to a changing live distribution.

Weights, biases, and learned embeddings are parameters. Learning rate, weight decay, layer count, and batch size are hyperparameters. Hyperparameters can be selected by a search algorithm and can change during training, as with a learning-rate schedule; they are not necessarily fixed manually before every run. Validation data guides their selection, while test data remains reserved for final evaluation.

In a linear predictor , the fitted slope and intercept are parameters. The learning rate determines how far an optimizer moves them at each step; the regularization strength determines how expensive large weights are. Those are hyperparameters because they govern the procedure rather than being the resulting fitted coefficients.

For example, train three candidate models with different regularization strengths on the same training split. Select the strength using validation error, then report the chosen system’s performance on an untouched test set. Repeatedly changing the strength after observing test results would turn that test set into another development set.

The distinction is about role, not whether a human typed the value. Automated search can choose learning rates, while a learned temperature can be a model parameter when optimized within the model’s objective. Architecture choices such as layer count also become part of the saved model configuration. Reproducibility requires preserving both the learned weights and the configuration, tokenizer, preprocessing, and selection procedure that produced them.

A model can fail even with low training loss: labels may be inconsistent, the target may poorly represent the product goal, or production inputs may differ from training inputs. Latency, memory, privacy, and feedback loops also matter. Diagnose errors by relevant slices, inspect the data pipeline, and compare with a simple baseline before changing model capacity.

A practical diagnosis separates data, modeling, and operation. Suppose a support classifier appears excellent offline but routes new requests badly. First inspect whether training labels were assigned consistently and whether evaluation accidentally included replies written after the outcome. Then check representation: truncating each ticket before the error message may remove the predictive information entirely. Finally compare deployment preprocessing with the training pipeline.

Even a correct model can fail operationally. A feature service may return stale values, a rare category may arrive in a new language, or the chosen threshold may overload the human review queue. Collect error examples with input version, prediction, and eventual outcome so these explanations can be distinguished.

A useful response is to rank failures by frequency and consequence, establish a baseline, fix the largest verified cause, and reevaluate the same slices. More parameters are appropriate when capacity is the limitation; they do not repair missing labels, an unsuitable objective, or unavailable production features. Deployment requires a fallback and ongoing measurement because the environment can change after a successful launch.

First evaluate untouched data drawn from the training distribution. If performance is already poor there, consider regularization, earlier stopping, deduplication, or more representative examples. If that holdout works but production-like data fails, investigate domain shift. Also check preprocessing consistency, label definitions, and group or time leakage. Simplifying the model is one possible remedy, not a universal fix.

Use three comparisons rather than applying regularization immediately. Measure training performance, a clean holdout from the same data source, and a holdout representing deployment. If training accuracy is 99% and same-source holdout accuracy is 75%, inspect model complexity, effective dataset size, and near-duplicate handling. If both are 95% but deployment is 70%, a shifted vocabulary, sensor, population, or labeling policy is more plausible.

Learning curves help choose a remedy. A large generalization gap that narrows as more independent examples are added suggests additional data could help. Training and validation errors that are both high suggest underfitting or inadequate input information. Early stopping chooses a checkpoint before validation begins deteriorating; regularization constrains the fitted solution; augmentation teaches selected invariances.

Do not tune directly against the final holdout while investigating. Create a development set for repeated decisions, preserve the test set, and check whether the proposed fix helps important subgroups. A reduced training score can be acceptable if deployment performance and calibration improve.

Keep it out of training, hyperparameter tuning, prompt selection, and repeated model selection. Use group-based or chronological splits when examples share users, documents, or time dependencies. Report uncertainty and important slices alongside the aggregate score. A test set does not automatically give an unbiased estimate of real-world performance if its sampling or labels are flawed.

The test set answers a specific question: how well does this fully selected system perform on examples drawn in this way? It does not automatically answer how the system will perform everywhere. A fraud model deployed next month needs a time-based test with features available at decision time; a medical-image model intended for new hospitals needs separation by institution or patient rather than individual image.

Define the evaluation unit before splitting. Ten photographs of one object are not ten independent objects; placing copies across splits rewards recognition of that object rather than generalization. Fit scalers, vocabularies, and other learned preprocessing without using test data.

Report the denominator as well as the score. Ninety correct predictions out of 100 provide less precise evidence than 9,000 out of 10,000, and correlated examples reduce effective sample size. Include relevant error categories and uncertainty estimates. When deployment changes materially, construct a new representative test rather than assuming an old benchmark remains sufficient. An independent test supports a bounded empirical claim, not a guarantee about unseen conditions.

Repeatedly comparing candidates on validation data can itself overfit that set. Keep a separate final test set, and use cross-validation or nested cross-validation when data is limited and the selection procedure warrants it. Fit preprocessing only on the training portion of each split. Good validation design improves decision quality; it does not guarantee generalization.

The training set determines the fitted weights; the validation set helps choose among possible training procedures. Suppose three checkpoints have validation losses of 0.8, 0.6, and 0.7 while training loss decreases throughout. Early stopping would choose the second checkpoint because the later improvement on training examples did not generalize to development examples.

For limited data, cross-validation repeats this comparison over several train/validation splits. Each fold must refit preprocessing using only its training portion. If the same folds select hyperparameters and are reported as the final unbiased performance estimate, selection can make that estimate optimistic; nested cross-validation separates those roles.

Match validation to deployment. Randomly splitting a forecasting series allows future information to influence earlier predictions, so use chronological windows. Group related users or documents where independence matters. Track multiple important slices, but avoid endlessly choosing models to improve whichever validation item failed last. Once model choices are settled, evaluate once on a separate test set and describe the selection process alongside the result.

Suppose training error is 2%, train-dev error is 3%, and production-like validation error is 15%. The small first gap and large second gap suggest distribution mismatch is the main problem. A 2% versus 14% train-dev gap instead suggests poor generalization within the training distribution. Use consistent metrics and examine sampling uncertainty; the comparison is diagnostic evidence rather than a perfect decomposition of error.

The terminology can be confusing: here “train-dev” means a held-out slice drawn like the training data, while “validation” or “dev” represents the desired deployment data. Neither slice is used for gradient updates. This arrangement is useful when abundant training data comes from a convenient source but deployment comes from another source.

For example, a speech model may train on clean recordings and deploy on phone audio. A same-source train-dev set tests whether the model learned to generalize to unseen clean recordings. A phone-audio validation set then tests the extra difficulty of background noise, microphones, accents, and compression. If clean generalization is good, gathering more of the same clean audio might be less useful than acquiring representative phone recordings.

Keep label quality and evaluation criteria comparable across both holdouts. Otherwise, a gap can reflect annotation differences rather than input distribution alone. The error differences are diagnostic rather than additive causal components: shift can interact with overfitting, and small sets have sampling noise. Inspect concrete errors before deciding which dataset or model change to make.

Even without taking gradients on test examples, choosing the best of many candidates using test scores adapts the system to that set. Tune on validation data and use the test set only after choices are settled. If extensive test-driven iteration has already happened, obtain a new independent holdout or explicitly report that the old test set became development data.

Model selection can exploit random noise in a score. Imagine 100 equally capable configurations evaluated on one small test set. Some will score higher by chance. Choosing the highest-scoring configuration and presenting that same number as independent evidence overstates expected performance, even though none of its weights was trained directly on test examples.

Leakage can also happen indirectly: editing prompts after reading failed test questions, selecting a retrieval configuration based on test answers, or retaining whichever checkpoint has the best test score. These choices all make the system depend on information from the test set.

Use a development set for iterative work and decide the metric, candidate selection procedure, and acceptance criteria before opening the final test. If a test uncovers a real bug, fixing it is appropriate, but the resulting system needs fresh independent evaluation for an unbiased final estimate. When new data is unavailable, disclose that the set has become development data and use cross-validation or uncertainty analysis without pretending it restores complete independence.

For squared-error regression, expected prediction error decomposes into squared bias, variance, and irreducible noise. Cross-entropy language modeling does not use that identical decomposition. In LLM fine-tuning, limited or repetitive data can encourage memorization, while insufficient capacity or optimization can cause underfitting. More parameters can improve generalization when accompanied by adequate data and training. Hallucination is not a synonym for high variance or overfitting, and a single pretraining epoch can still contain trillions of tokens.

For a fixed input , imagine repeatedly drawing a training dataset and refitting the same learning procedure. Bias compares its average prediction with the true regression function. Variance measures how much the fitted predictions fluctuate around that average. The noise term represents variation in the outcome that the available inputs cannot predict. A model can have low training error and still high variance because its solution depends strongly on accidental training examples.

A simple polynomial illustrates the idea: a straight line may miss a curved relationship systematically, whereas a very flexible curve may follow each noisy observation. Regularization can reduce sensitivity at the cost of imposing additional structure. This intuition is useful, but the familiar U-shaped curve is not a universal law for overparameterized networks.

For an LLM adaptation, compare multiple data samples or seeds, train/holdout loss, and task-level results. An instruction model that memorizes a small set of long answers may fail paraphrases despite low training loss. That is evidence about generalization; a factual error alone does not identify which statistical component caused it.

With trainable parameters or updates :

Elastic net can be useful for a sparse task head or adapter, but it is not a standard requirement for LLM fine-tuning. State whether the penalty targets adapter weights, full weights, or the update from the pretrained checkpoint. L1 can yield exact zeros with an appropriate optimization method; ordinary gradient steps do not guarantee them.

Neither penalty inherently makes the data-fitting loss robust to target outliers. L2’s unique-solution property for strictly convex ridge regression does not extend to nonconvex neural networks. Sparse weights do not automatically accelerate dense GPU kernels. Also, AdamW’s decoupled weight decay differs from adding an L2 penalty to Adam’s loss.

For a single coefficient, the L2 penalty contributes gradient , so its pull toward zero is stronger for larger weights and vanishes near zero. L1 contributes a magnitude-independent pull away from zero on either side, and has a kink at zero. With proximal optimization, a small coefficient can be set exactly to zero through soft thresholding. This explains the common sparsity difference without claiming that every neural optimizer produces sparse weights.

Suppose a frozen language model feeds a linear head with many correlated features. L1 may select a small set of features, but the selected set can change when similar features substitute for one another. Adding L2 can stabilize that selection. For a LoRA update, penalizing its factors is not identical to penalizing the product ; rescaling one factor and inversely rescaling the other preserves the update while changing some penalties.

Choose penalties using held-out task quality and the intended goal, such as smaller updates or interpretable selection. If the goal is faster inference, verify that the resulting sparsity has a structure supported by the serving kernel.

Let and let a softmax linear model have logits . Define its normalized weighted geometric ensemble by:

The expected log-normalizer cancels across classes. This is not the arithmetic average . In a nonlinear deep network, replacing every random mask by its mean generally does not reproduce the expected logits, so the exact argument fails. Modern inverted dropout scales retained activations by during training and makes dropout an identity at evaluation.

The ensemble interpretation comes from training on many masks while sharing the same weights. With two independently retained units, four subnetworks are possible; with many units, explicitly training each subnetwork separately would be infeasible. Dropout instead samples masks during optimization, encouraging a useful prediction even when some intermediate features are unavailable.

For inverted dropout with keep probability , a retained activation is multiplied by , and a dropped activation becomes zero. Its expectation equals the original activation: . That equality concerns the local activation, not the final output after subsequent nonlinear layers.

The geometric-ensemble identity above is exact because a linear logit is linear in the mask and the class-independent normalizer cancels when renormalized. An arithmetic average of probabilities generally gives a different result. At ordinary evaluation, disable dropout; leaving it enabled makes predictions stochastic. Deliberately repeating stochastic evaluations can be an uncertainty heuristic, but variation across masks is not automatically a calibrated probability of being wrong.

Covering a -dimensional grid with values per coordinate requires points. Distance concentration can also weaken nearest-neighbor intuition, depending on the distribution and metric. Real data may occupy a much lower-dimensional manifold.

Unmasked self-attention without positional information is permutation-equivariant: permuting inputs permutes outputs. Positional embeddings, relative biases, or rotary embeddings make position available. A causal mask already introduces directional structure, but does not make explicit positional methods a dimensionality-reduction technique. Sinusoidal or rotary formulas can be evaluated at new positions; reliable length extrapolation still requires validation.

Take 10 bins along each feature axis. A two-dimensional space has 100 cells; a ten-dimensional space has . A dataset that seems large in two dimensions becomes sparse in ten. This example assumes a full Cartesian grid; correlated real features may occupy a much smaller effective region. Feature learning and prior structure help by restricting what relationships the model must learn.

Position addresses a different ambiguity. The same token multiset can describe “the dog chased the cat” or “the cat chased the dog.” A model that treats every permutation identically cannot reliably distinguish the roles. Adding a position-dependent representation, or modifying query-key comparisons according to relative positions, gives attention a way to condition on ordering.

A positional representation does not reduce the number of semantic features or supply examples for every possible combination. Likewise, being able to calculate a sinusoid at position one million does not establish useful behavior there: the trained attention patterns may not transfer. Evaluate order-sensitive tasks and long-range evidence use separately from dimensionality reduction or memory efficiency.

For an autoregressive model, factor the probability using the chain rule and maximize its log:

A beginning-of-sequence convention defines the initial context. Teacher forcing supplies observed prefixes during training. Taking logs turns products into sums; minimizing their negative gives the usual next-token cross-entropy objective. This trains a distribution over text, not a direct truth-verification objective.

For a toy two-token sequence, suppose the model assigns probability 0.2 to the first token and 0.5 to the second given the first. The sequence probability is , and its negative log-likelihood is nats. Summing individual losses gives the same value: . Dividing by two gives the mean per-token loss.

During teacher-forced training, the second prediction receives the observed first token even if the model would have generated something else. Many positions can therefore be trained in parallel under a causal mask. At generation time, the model conditions on its own earlier outputs, so a mistake can alter later predictions.

Maximum likelihood rewards matching the distribution represented by the corpus. If that corpus contains repeated errors or biased sampling, learning those patterns can also improve likelihood. Data curation and separate evaluations are needed for factuality, usefulness, and safe behavior. Loss comparisons additionally require consistent tokenization, masking, and reduction; a smaller reported number is not meaningful when the denominator changes.

If are vocabulary logits predicting token , then:

Here is 1 for supervised targets and 0 for padding or excluded prompt tokens. The model’s output at a sequence position usually predicts the following token, so labels must be shifted correctly. A probability of 0.5 gives about 0.693 nats of loss; 0.01 gives about 4.605 nats. A sequence sum and a per-token mean have different scales.

The log-sum-exp term normalizes logits into a probability distribution. Differentiating the loss with respect to a logit gives : gradient descent raises the target logit’s relative score and lowers competing scores in proportion to their predicted probability. This gives a direct connection between the scalar loss and what learning changes.

Consider targets with predicted probabilities 0.5 and 0.25. Their losses are about 0.693 and 1.386 nats; the mean is 1.040 nats. If the second position is padding and masked out, the correct mean is 0.693, not half that value. The denominator must count supervised targets, and a batch with no supervised targets needs explicit handling instead of division by zero.

For causal training, input tokens at positions 0 through predict targets at positions 1 through . Some model implementations perform that shift internally, so shifting again in the caller is a bug. Verify one tiny token sequence manually and inspect loss masks before interpreting full-training curves.

For mean NLL measured with natural logs, . BLEU combines modified n-gram precision with a brevity penalty and does not need model probabilities. It can reward high overlap despite a crucial factual error and penalize valid paraphrases.

Compare perplexities only with compatible tokenization, evaluation text, context, and masking. Fluency alone does not determine a sentence’s perplexity, and changing “cat” to “dog” does not establish either a perplexity or BLEU score. Autoregressive encoder-decoder models also have well-defined conditional perplexity. Evaluate factuality and task success separately.

Perplexity is the geometric inverse of the probabilities assigned to observed tokens. If a model gives every observed token probability , its mean loss is and perplexity is 4. It behaves, in that limited sense, as if choosing among four equally likely alternatives. This does not mean it literally considers only four vocabulary entries at each position.

Tokenization changes the prediction units. A rare word split into four subwords contributes four targets, whereas a tokenizer that encodes it as one token contributes one. Raw per-token perplexities from those systems are not directly comparable. Repeated boilerplate can also lower corpus perplexity while important factual questions remain unanswered.

BLEU evaluates generated text rather than next-token probabilities. Clipping prevents a repeated word from earning unlimited credit, and the brevity penalty discourages excessively short outputs. Nevertheless, a translation that changes “not approved” to “approved” can retain high overlap. For an assistant, combine likelihood or overlap measures with task correctness, evidence support, and appropriately designed human evaluation.

For vocabulary size , one-hot target , and a uniform prior:

The correct class receives . An alternative convention assigns to the correct class and to each other class; do not mix the two definitions. Benefits depend on the task and smoothing strength, and the smoothed training loss is not ordinary hard-target perplexity.

For three classes, target class 1, and , the uniform-mixture convention gives target probabilities approximately . The logit gradient becomes instead of . A prediction of probability exactly one for class 1 would heavily penalize the nonzero target mass assigned to the other classes, so the optimum no longer requires infinite separation of the winning logit.

This can discourage overconfident fitting to hard labels, especially when annotations are imperfect. It also deliberately introduces bias: an example whose correct class is certain is trained as if there were uncertainty. Too much smoothing weakens the learning signal, and a uniform prior over a large vocabulary is not a model of which alternative words are semantically plausible.

When reporting perplexity, recompute ordinary NLL against the observed tokens rather than exponentiating the smoothed objective. If comparing distillation or calibration behavior, examine it directly; improved classification accuracy does not prove improved confidence calibration. Document the convention because the two common definitions produce different target distributions at the same numeric epsilon.

Standard attention computes weights with softmax and combines all allowed values. Hard attention might sample one location or select a subset; training the selection may require a score-function estimator, relaxation, or another surrogate. A fixed causal or sparse mask does not by itself make attention “hard”: the permitted positions can still receive continuous softmax weights.

Suppose three source values are and softmax produces weights . Soft attention returns . Small changes to the scores change the weights smoothly, allowing ordinary backpropagation through both the scoring function and the values. The result is a mixture, rather than a selected original value.

Hard attention might choose only by an argmax or sample a location according to the weights. Argmax is piecewise constant, so gradients through the discrete choice do not tell the scorer how a different choice would change the loss. Sampling estimators or continuous relaxations address this, each with variance or bias tradeoffs.

Discrete selection can save computation if the implementation avoids processing unselected content, but it can also discard necessary evidence. A top-k retrieval stage followed by soft attention combines hard candidate selection with differentiable mixing inside the selected set. Similarly, setting future-position logits to negative infinity still leaves ordinary soft attention over the allowed past positions.

import math

def kl(p, q):
    if len(p) != len(q) or not p: raise ValueError('matching nonempty support required')
    if any(not math.isfinite(x) or x < 0 for x in [*p,*q]): raise ValueError('invalid probability')
    if not math.isclose(sum(p),1) or not math.isclose(sum(q),1): raise ValueError('not normalized')
    total = 0.
    for a,b in zip(p,q):
        if a == 0: continue
        if b == 0: return math.inf
        total += a*math.log(a/b)
    return total

assert abs(kl([.5,.5],[.5,.5])) < 1e-12
assert math.isinf(kl([1,0],[0,1]))
assert math.isclose(kl([1,0],[.5,.5]), math.log(2))

Natural logarithms give nats; base-two logs give bits. For continuous variables, use densities with respect to the same measure and integrate, or derive a closed form for a known family. A histogram estimate depends on binning and finite-sample bias. Different variable names do not define comparable distributions until support and representation are aligned; do not compute KL directly between raw observations and call it a distributional divergence.

For and , the divergence is nats. Reversing the arguments gives a different value, about 0.1438 nats. This asymmetry reflects that the expectation is taken under the first distribution: errors in regions where it places more mass matter more.

KL is nonnegative for valid distributions and is zero when they agree almost everywhere under the relevant measure. Individual summands can be negative, so do not clip each term separately. A zero target mass contributes zero by a limit, while positive target mass against zero predicted mass gives infinite divergence.

For sampled data, first define comparable supports or density models. Two lists of observations with different scales or meanings are not probability vectors. Smoothing a histogram avoids infinite estimates but changes the distribution being measured, so disclose that choice. In numerical code, small negative totals from floating-point error may occur near equality, but substantial negatives indicate invalid inputs or a faulty computation.

Cross-entropy for true class y is . If argmax predicts y, no other probability exceeds , hence . Therefore and . A probability of one gives zero loss; finite softmax logits usually approach rather than attain it.

For binary classification with a 0.5 threshold, the analogous range is zero to . With a nonstandard threshold, label smoothing, sample weights or regularization, these bounds change. A model can predict correctly with low confidence and still incur substantial loss.

For zero-one loss, correct classification gives exactly zero. For a custom loss on unbounded scores, accuracy may impose no useful finite upper bound. Also distinguish perfect empirical accuracy from accuracy over the entire data distribution; the former says nothing about unseen examples.

Take a three-class prediction [0.34,0.33,0.33] with the first class correct. Accuracy for that example is one, but cross-entropy is , close to . A much more confident correct prediction [0.98,0.01,0.01] has loss about 0.0202. Accuracy collapses both to the same binary result, while cross-entropy distinguishes confidence.

The upper bound follows only when the decision is an argmax over a normalized K-class distribution and the loss is the unsmoothed negative log-probability of the true class. For binary thresholding at a different cutoff, a positive prediction can be correct with probability below 0.5, changing the bound. Adding weight decay contributes a parameter-dependent term unrelated to that example’s accuracy.

Clarify whether “accuracy is 1” means this one prediction or all examples in a finite training set. Even perfect training accuracy does not bound held-out loss or establish generalization. State the assumptions before giving a numerical bound rather than treating accuracy and loss as interchangeable metrics.

Start with the task’s decision metric: false negatives and false positives may have different consequences, and class imbalance can make accuracy misleading. Examine the confusion matrices and examples where the models disagree. Use a paired bootstrap or an appropriate paired test rather than treating the two accuracy estimates as independent.

Check whether the 3-percentage-point gap is stable across time, users and important slices. Compare calibration if probabilities drive decisions; adjust thresholds on validation data using the actual cost tradeoff. Measure inference cost and latency under the production workload.

If the higher-accuracy model remains better on the relevant utility metric and meets constraints, select it and canary the release. Otherwise choose the simpler or safer alternative, or route cases selectively. Keep the test set untouched while tuning; repeatedly choosing variants on it turns it into training feedback.

An illustrative cost example shows why the ranking can reverse. Suppose one model makes fewer total errors but most are missed high-cost fraud cases, while the other makes more low-cost false alarms. Expected utility weights those errors by consequence, whereas raw accuracy gives each equal importance. The appropriate threshold may also differ between models if their probability calibration differs.

Compare predictions on the same cases and inspect the disagreement table: how often does A succeed when B fails, and vice versa? The uncertainty in a three-point gap depends on that paired structure and sample size. A gap on 100 examples is less convincing than the same gap on a large representative holdout, and neither establishes future stability under distribution shift.

After offline selection, canary the model with clear rollback criteria and measure actual outcomes, latency and cost. If the models have complementary strengths, an ensemble or routed system is an option only after accounting for extra complexity and leakage-free router evaluation. The answer should be a decision process with evidence, not a reflexive choice of the larger percentage.

import pandas as pd

train = pd.DataFrame({'age':[20., None, 40.], 'city':['A', None, 'B']})
test = pd.DataFrame({'age':[None, 30.], 'city':[None, 'C']})
median_age = train['age'].median()

def clean(frame):
    result = frame.copy()
    result['age_missing'] = result['age'].isna()
    result['age'] = result['age'].fillna(median_age)
    result['city'] = result['city'].fillna('Unknown')
    return result

assert clean(test)['age'].tolist() == [30., 30.]
assert clean(test)['age_missing'].tolist() == [True, False]

Dropping rows can bias the sample if missingness is related to the outcome. Forward-fill is only valid when ordering and domain semantics justify carrying the previous value; never fill across unrelated entities. An all-missing training column has no usable median and needs a separate policy. Pandas distinguishes several missing sentinels and nullable dtypes, so test both the values and resulting column types.

The example computes the age median once from training data, then reuses it for test data. Recomputing a median on the full dataset leaks information from evaluation into preprocessing. The separate age_missing feature preserves the fact that a value was absent, which can itself carry signal but may also encode a problematic collection process.

Distinguish structural absence, data corruption and unobserved values. A missing apartment number may mean “not applicable,” while a missing income may correlate with the outcome and require a different treatment. Dropping rows changes the sample; filling every numeric column with zero invents a value that may have a real meaning.

Check nullable integer, boolean and categorical dtypes after cleaning, and preserve an explicit policy for all-missing columns or unseen categories. For time-series data, sort within entity before forward filling and avoid using future observations to fill past predictions. Evaluate the entire preprocessing pipeline on held-out data, with missingness rates monitored after deployment so a broken upstream feed does not silently become a model-quality regression.

For a fixed input x and noisy target with zero-mean noise independent of training, the expected squared prediction error is

The expectation is over hypothetical training datasets and test noise. A rigid linear model on a nonlinear target may have high bias; a highly flexible model trained on few noisy examples may vary dramatically between samples. Regularization, more data and ensembling affect these terms differently.

Use learning curves and held-out errors to diagnose the practical problem. High training and validation error suggests underfitting or poor features; a large generalization gap suggests overfitting, though distribution shift can produce a similar pattern. Classification losses have different decompositions, and overparameterized models can show double-descent behavior. Do not infer bias and variance solely from parameter count.

Bias and variance concern a learning procedure across possible training samples, not just one fitted model’s residuals. Imagine repeatedly collecting datasets and fitting the same polynomial degree. A low-degree fit may miss a curved trend consistently, while a high-degree fit may follow noise differently in each sample. Their average predictions and spread define the two terms.

More data often reduces variance without requiring a simpler model. Bagging reduces variance when member errors are not perfectly correlated; regularization usually trades some added bias for lower sensitivity. Better features can reduce bias without merely adding parameters. These interventions explain why “increase complexity until the curves meet” is too simplistic.

Use training-size and regularization learning curves with deployment-like validation splits. A generalization gap may arise from distribution shift or leakage mistakes as well as variance. For modern overparameterized models, interpolation and implicit optimization biases can produce behavior outside the classic U-shaped cartoon. The squared-error decomposition remains useful under its assumptions, but model selection still requires measured performance and error analysis.

Write . Full column rank makes the coefficient solution identifiable. The exogeneity condition is central to unbiased coefficient estimation; omitted variables correlated with included features can violate it. Homoscedastic, uncorrelated errors give the familiar Gauss–Markov efficiency result among linear unbiased estimators.

Check residual plots for nonlinear patterns and changing variance, inspect time or group structure, and examine feature collinearity. Use robust or clustered standard errors when appropriate, but remember that they do not repair a misspecified conditional mean or causal confounding.

Normal predictors are not required, and independent predictors are not required; only perfect linear dependence prevents a unique ordinary least-squares solution. For forecasting, evaluate on future periods. For causal claims, additional identification assumptions are necessary beyond the regression formula.

Linearity means the conditional mean is linear in the chosen coefficients, not necessarily in raw inputs. A model containing , and an interaction term is still linear regression in its parameters. Choosing those features is part of specifying the mean function. Residual curvature suggests that the chosen representation may miss structure.

Heteroscedasticity mainly affects conventional standard-error formulas and efficiency; robust standard errors can address inference under suitable conditions but do not remove omitted-variable bias. Serially correlated errors in time data require time-aware evaluation and appropriate uncertainty treatment. Strong but imperfect collinearity can make individual coefficients unstable even when predictions remain useful.

Check the assumption relevant to the intended claim. Computing least-squares predictions requires less than making an unbiased coefficient interpretation, and causal interpretation requires still more. Normality of input features is not a requirement. Use residual diagnostics, holdout performance and domain reasoning together, and avoid treating a successful normality test as proof that a regression model answers the underlying scientific or business question.

L2 regularization shrinks coefficients, L1 can encourage sparsity, dropout perturbs training representations and early stopping limits optimization. Each introduces a bias or constraint that may reduce generalization error. The correct strength depends on data size, noise, architecture and objective.

Validation must reflect deployment: split by time for forecasting, by entity when repeated users or patients could leak, and by source when near-duplicate documents exist. Fit preprocessing inside the training fold, including imputation, scaling and feature selection. Otherwise the validation score includes information from the examples it is supposed to hold out.

Compare learning curves and multiple plausible settings; do not add regularization simply because a model is large. If both training and validation performance are poor, stronger regularization may worsen underfitting. If the validation distribution differs from deployment, no tuning method can make its estimate reliably representative.

A simple experiment illustrates the distinction. Fit several ridge models with different penalties on the training split, evaluate each on validation data and select a penalty using that evidence. The penalty changes the fitted coefficients; the validation split measures consequences without participating in those coefficient updates. The final untouched test set estimates performance after the choice is made.

Repeatedly trying hundreds of variations against the same validation set can overfit the selection process even if every individual training run is regularized. Nested cross-validation or a fresh holdout can be useful when model selection is extensive. Group and time boundaries should match deployment so related observations do not leak across folds.

Early stopping uses validation to choose how long training continues, making it both a selection procedure and an implicit regularization mechanism. Keep that dual role explicit. If the model underfits, stronger regularization may worsen both training and validation error; if labels are wrong, neither a penalty nor a clever split repairs the data. Diagnose the failure before choosing the tool.

Let and , with . Rewrite the loss as . Differentiating gives .

For stability, let and compute . The loss remains finite even when a naive softmax would underflow a low-probability target to zero. Framework cross-entropy functions typically expect raw logits; applying softmax first changes the computation and harms stability.

Check a two-class example with equal logits and target class zero: p is (0.5,0.5) and the gradient is (-0.5,0.5). The components sum to zero because adding a constant to every logit leaves probabilities unchanged. Batch averaging adds a factor of 1 / batch_size; class weights and unnormalized targets alter the formula.

The softmax Jacobian is . Combining it with gives , which reduces to for normalized targets. This also shows exactly why unnormalized target weights change the gradient.

With logits [1000,0] and the second class correct, the true loss is close to 1000. A naive exponentiation overflows on the first logit, while a separately rounded softmax may assign the target probability zero and produce infinite loss. Log-sum-exp keeps the computation finite by subtracting the maximum before exponentiation.

For a numerical check, perturb one logit by a small positive and negative amount and compare the central finite difference with the analytic gradient. Test moderate logits for the derivative check and extreme logits for stability. Account for reduction conventions: summing examples and averaging examples differ by batch size, and masking changes the effective denominator.

For , . Keeping the top r singular values gives the best rank-r approximation in Frobenius and spectral norm. The squared Frobenius reconstruction error is the sum of squared discarded singular values.

One application is compressing a dense weight matrix into two smaller factors, reducing parameters from mn to approximately . This is beneficial only when r is sufficiently small and the hardware executes the factorization efficiently; fine-tuning may be needed to recover task quality.

Another application is PCA of centered activation or embedding matrices: singular directions reveal dominant variation and can support dimensionality reduction or diagnostics. Dominant variance is not necessarily the most task-relevant information. State shapes and whether the matrix is centered before interpreting the decomposition, and avoid equating numerical rank with an exact count of semantic concepts.

For singular values [5,2,1], keeping rank one discards squared energy , so the Frobenius reconstruction error is . This is the best rank-one approximation under that matrix norm, but not necessarily the best compression for the downstream task: a small singular direction can still encode a critical feature.

For a square width-D weight, two rank-r factors use about parameters rather than . The saving requires , and two smaller matrix multiplications may introduce overhead even when arithmetic decreases. Measure actual serving behavior after factorization and fine-tuning.

For PCA, center the observations before SVD if the goal is directions of variation around the mean. Without centering, a dominant mean offset can determine the first singular vector. Inspect explained variance and held-out task effects separately. LoRA resembles a low-rank update but learns factors directly; it does not need to decompose the full pretrained weight matrix with SVD at every training step.

Let be the expected remaining flips with no current head streak, and the expectation after one head. From state zero, a tail returns to zero and a head moves to one:

From state one, a head completes the task while a tail resets:

Substitute the second equation into the first: , hence and .

For a coin with head probability p greater than zero, the same equations give . At p=1, the answer is two; as p approaches zero it diverges. These checks support the derivation. Simulation can verify the estimate statistically but is not a replacement for the exact state-based argument.

The state must retain exactly the history relevant to future success. After a tail, previous heads no longer matter, so the process returns to state zero. After one head, another head finishes, while a tail loses that partial progress. This two-state description handles overlapping attempts automatically.

The tempting answer four comes from treating disjoint pairs as independent trials with success probability one quarter. But the actual process checks overlapping pairs and the waiting-time accounting differs; that shortcut does not model what happens after a single head. The expectation equations charge one flip for every transition, including transitions back to the same state.

For a biased coin, and . Solving yields , recovering six at . At , two flips are necessary and sufficient; at , completion never occurs. These endpoint checks are useful whenever deriving a waiting-time formula from a state machine.

Q-learning estimates action values using a bootstrapped target such as . Replay can improve sample efficiency, but function approximation, off-policy data and bootstrapping can create instability. Large or continuous action spaces make the maximization difficult.

Policy gradients optimize expected return through gradients of action log-probabilities weighted by return or advantage. They naturally handle parameterized continuous distributions, but gradient variance and data requirements can be substantial. Baselines, advantage estimation and constrained updates improve stability.

Compare exploration, reward sparsity, safety constraints, environment cost and action structure. If collecting experience is expensive, sample efficiency matters more than a small implementation difference. If actions are generated sequences, policy-gradient-style objectives may fit naturally, but reward quality and credit assignment remain central challenges. Validate against a simpler control or bandit baseline where appropriate.

For a small discrete control problem, Q-learning can store or approximate the value of each action and choose the maximum efficiently. Reusing past transitions through replay can be valuable when environment interaction is expensive. However, an inaccurate maximum can amplify overestimation, and combining bootstrapping with off-policy function approximation requires stabilization and careful evaluation.

For continuous control, a policy can directly output parameters of a distribution over actions, avoiding a difficult maximization over every possible real-valued action. Policy gradients use sampled returns or advantages to adjust that distribution, but estimates can be noisy. Actor–critic methods learn a value estimate to reduce variance while retaining a parameterized policy.

Choose using action structure, data availability, exploration needs and the cost of failure. Offline logged data creates additional coverage problems for either family: optimizing actions absent from the data can exploit estimation errors. Compare learning curves in environment steps and wall time, and evaluate under the same reward and safety constraints. Algorithm names alone do not settle the design; reward quality and trustworthy evaluation often dominate the outcome.

Suggested technical approach: recover the exact input features, preprocessing, model version, score and threshold for the decision. Verify data accuracy and distinguish a policy rule from the learned model’s score. If a transparent model or approved reason-code system exists, use its documented mapping rather than generating prose from intuition.

With query access, test plausible feature perturbations and compare local surrogate predictions with the original model. Counterfactuals should respect feasible changes and dependencies; changing an immutable attribute or independently altering correlated variables can produce misleading advice. Feature attribution describes a model’s behavior under assumptions, not the causal effect of changing a person’s circumstances.

Report uncertainty and route disputed or consequential cases to the designated review process. The interview answer should emphasize faithful evidence and correction paths. It should not promise that SHAP, LIME or a language-model explanation automatically satisfies legal or fairness obligations.

LIME provides one concrete local-surrogate method; its neighborhood and fidelity assumptions still require validation for the decision being explained.

An illustrative local test changes one feasible financial input while holding the approved preprocessing and other inputs fixed, then observes the score change. This can reveal sensitivity, but it does not prove that changing that factor in the real world would cause the same outcome. Correlated features and policy rules can make naive counterfactuals impossible or misleading.

A local surrogate should be tested on nearby cases not used to fit it, with an explicit fidelity measure. If its predictions disagree substantially with the actual model, do not present its feature ranking as the reason for the decision. Likewise, a fluent language-model explanation generated from a score alone is unsupported.

The user-facing process should offer a way to correct inaccurate source data and request review through the authorized channel. Preserve the exact decision version and evidence so the explanation is reproducible. This technical framework does not prescribe lending compliance requirements; those depend on the applicable context and must be validated by the responsible specialists.

Check randomization, sample-ratio balance, logging and confidence intervals. Determine whether “2%” means relative or absolute AUC change. Compare calibration curves, log loss and expected-versus-observed CTR by segment and score range. A global AUC gain may hide degradation in the high-score candidates that actually enter auctions.

Trace how predicted CTR affects bid ranking, prices, budgets and ad eligibility. Poor probability scale can alter economic decisions even when ordering improves. Delayed conversions, advertiser mix and user response may further disconnect CTR from revenue.

Recalibrate on a representative holdout, rerun auction simulations and test with revenue, user experience and advertiser-value guardrails. Avoid tuning repeatedly on the same experiment. If the gain does not improve the product objective after diagnosis, retain the incumbent and use the failure to improve offline evaluation.

AUC is unchanged by any strictly increasing transformation of scores, but calibration can change dramatically. Squaring probabilities may preserve ranking while making the numerical values wrong. If the auction uses predicted CTR multiplied by bid, those values influence economic decisions, so a ranking gain alone does not establish revenue improvement.

Inspect the region of the score distribution that actually wins impressions. Offline AUC can improve mostly on easy negatives or low-value traffic that never reaches the auction. Training-serving feature differences, delayed labels and changed advertiser budgets can also break the offline-online relationship. Verify sample-ratio balance and confidence intervals before diagnosing a model effect.

Recalibrate using a representative validation set, then evaluate log loss, calibration by segment and simulated auction outcomes without repeatedly tuning on the live experiment. Run another controlled test only after identifying a plausible mechanism. Revenue-neutral results may still hide changes in user experience or advertiser value, so retain those guardrails. If the improvement does not survive the full decision pipeline, keeping the incumbent is a valid outcome.

The objective has gradient . Setting it to zero gives the normal equations. Rank deficiency means coefficients need not be unique, although fitted predictions are unique; the pseudoinverse selects the minimum-norm solution.

For dense data with rows and features, direct methods can require roughly work when , whereas each full gradient pass costs . Sparse, distributed, or streaming data often favors iterative solvers. Standardize features and select a stable learning rate; stop based on convergence and validation, not a fixed arbitrary number of steps. Ridge regularization improves conditioning, but changes the objective and usually leaves the intercept unpenalized.

Solve without explicitly inverting a matrix

When X has full column rank, the algebraic expression is . In implementation, solve the least-squares system using QR or SVD instead of computing the inverse: forming squares the condition number and can amplify numerical error. If two columns encode the same feature, rank deficiency makes the inverse undefined, while SVD still supplies a minimum-norm solution.

For points (0,1), (1,3), and (2,5), include an intercept column so X has rows [1,0], [1,1], [1,2]. Then and , yielding intercept 1 and slope 2. Gradient descent approaches the same optimum for this convex objective with a suitable step size, but avoids a large dense factorization when features or observations are enormous. Compare conditioning, sparsity, precision, and convergence requirements before choosing the solver.

For logistic negative log likelihood , common objectives are and . L1 is nondifferentiable at zero, so proximal or coordinate methods handle its thresholding behavior. L2 has a smooth gradient and improves conditioning.

With highly correlated predictors, L1 may arbitrarily retain one, making selected features unstable across samples; L2 tends to distribute weight among them. Elastic net combines both when sparse grouped behavior is useful. Standardize numerical features using training data only, choose strength with appropriate cross-validation, and evaluate calibration as well as discrimination. Do not interpret a zeroed coefficient as proof that a variable is causally irrelevant. Avoid penalizing the intercept unless there is an explicit reason.

The update reveals the difference

For an L2 penalty, the gradient adds , continuously shrinking coefficients toward zero. For L1, a proximal step applies soft thresholding to a gradient-updated coefficient u: . That interval mapping to exactly zero explains why L1 performs feature selection while ordinary L2 usually leaves small nonzero weights.

If income is measured in dollars and age in years, identical penalty strength has different effects unless scaling is handled. Standardize using training statistics and apply the same transformation during serving. With two nearly identical predictors, sparse selection can switch between them across samples even though prediction is stable; do not overinterpret which one survives. Tune the penalty against heldout log loss or the decision metric, not training fit. Under complete separation, regularization can also prevent coefficients from growing without bound, but existence and uniqueness depend on which parameters are penalized.

For labels and ,

Compute this with a stable softplus implementation rather than directly exponentiating large logits. The gradient is and Hessian is , where . Thus every Hessian quadratic form is nonnegative and the loss is convex; any attained stationary point is globally optimal.

Counterexample: one sample has loss , which decreases toward zero as without reaching it. Adding with supplies a positive-definite Hessian and forces loss to grow as norm grows. If the intercept is unpenalized, check existence separately, especially when all labels have one class.

Convex does not guarantee an attained minimum

For any vector v, , which proves convexity. If all labels are separable by a hyperplane, scaling its coefficients can push predicted probabilities toward 0 or 1 and drive the loss toward its infimum without attaining it at finite coefficients. Therefore the question’s unqualified request to prove a global minimum exists is false.

For a simple nonseparable intercept-only example with one positive and one negative observation, the optimum is probability 1/2, coefficient 0, and total loss . The gradient vanishes there, and convexity makes it global. Adding a positive L2 penalty on every coefficient makes the objective coercive and strongly convex: it must attain one unique minimum. If the intercept is exempt, check class coverage and separation rather than carrying over that proof unchanged. Stable softplus and finite-value checks remain necessary even for a mathematically well-behaved objective.

The identities are and . For a one-hot target, . If changes with learned parameters, the supposedly constant term is no longer necessarily constant. KL is asymmetric; reversing its arguments changes the optimization behavior.

An InfoNCE-style contrastive objective treats a positive pair as the correct class among negatives using similarity scores divided by temperature. Its behavior depends on negative sampling, false negatives, batch composition, and representation normalization. Use cross-entropy for supervised categorical prediction, KL for distribution matching with an explicit direction, and contrastive learning for representation relationships. Numerical implementations require stable log probabilities and careful handling of zero target mass.

A small distribution example

Let target p=(0.75,0.25) and prediction q=(0.5,0.5), using natural logarithms. Cross-entropy is about 0.6931 nats, target entropy about 0.5623, and forward KL about 0.1308. When optimizing q with fixed p, cross-entropy and forward KL differ only by that constant, so their gradients with respect to q’s model parameters are identical.

Contrastive learning instead defines which examples should be close. An image and its matching caption form a positive pair; other captions in the batch act as negatives. Applying cross-entropy over their similarity scores produces an InfoNCE loss, so “contrastive” describes the supervision and candidate construction rather than a completely unrelated mathematical family. False negatives, such as two captions describing the same image content, can push compatible representations apart. Temperature controls the sharpness of similarity comparisons. Choose the objective according to whether you need calibrated class probabilities, teacher-distribution matching, or reusable embeddings.

Forest trees can generally be trained independently, enabling broad parallelism. Boosting rounds depend on previous predictions, limiting parallelism across rounds, although split finding and operations within a round can be parallelized. Their actual runtime depends on tree depth, number of trees, histogram construction, sparsity, hardware, and stopping rules; neither is always faster.

Random forests often provide a strong low-tuning baseline and are comparatively robust to individual noisy trees. Boosted trees can fit complex tabular patterns efficiently but need control of learning rate, depth, sampling, and number of rounds. Use early stopping on a valid holdout and compare quality at matched latency and memory budgets.

How the ensemble is constructed

Bagging fits multiple predictors on resampled data and averages or votes their outputs. Random Forest also randomizes feature choices at splits, reducing correlation between trees so averaging can reduce variance. Boosting builds a sequence: each new tree corrects errors expressed through the current objective’s gradients, and its contribution is scaled by a learning rate.

In XGBoost, a candidate split is evaluated using accumulated first- and second-order loss information plus regularization. Histograms reduce the number of candidate thresholds that need examination, and column/row subsampling further change the cost. A forest can train trees concurrently, whereas the next boosting round depends on prior predictions. At inference both generally traverse many trees, so depth and tree count determine latency more directly than the label “boosting.” For noisy labels, aggressive boosting can keep fitting hard mislabeled examples; a forest is not immune but averages independently constructed trees. Compare tuned configurations under the same validation split and resource budget.

For squared error under standard assumptions, expected prediction error decomposes into squared bias, variance, and irreducible noise. More flexible models may reduce bias while increasing variance, but the relationship is not a universal monotonic law for every modern training regime.

Choose splits that match deployment: grouped splits for repeated users or patients, chronological splits for forecasting, and nested validation when estimating performance after extensive tuning. Perform imputation, scaling, and feature selection inside each training fold to prevent leakage. In high dimensions, use domain structure, regularization, or learned representations; arbitrary feature deletion can remove useful interactions. Compare learning curves: persistent training error suggests bias, while a large train/validation gap suggests overfitting or distribution mismatch.

Interpret the three ideas together

A shallow model that misses a curved relationship may have high bias: both training and validation errors remain high. A deep tree that memorizes a small sample may have high variance: training error is low but predictions change substantially with the sample. Cross-validation estimates how choices generalize, provided the folds reproduce the relevant independence assumptions.

Dimensionality makes coverage expensive. A toy grid with ten bins per feature has cells for d features; adding a feature multiplies the possible cells by ten. Real data often lies on lower-dimensional structure, but this example explains why sparse coverage and unreliable distances can appear as features accumulate. Regularization, suitable representations, and more relevant data address different parts of the problem. Use learning curves to distinguish inadequate capacity from inadequate sample size. If validation error stays high despite abundant data and low training error, also investigate a distribution shift or leaked split rather than forcing every failure into the bias–variance story.

One GRU convention is , , , and . Some implementations reverse the update-gate convention or place reset multiplication differently; match equations to code.

When the update gate preserves state, the recurrence offers a near-identity route for information and gradients. Long sequences still benefit from gradient clipping, normalization, and careful initialization. BiLSTM concatenates or otherwise combines forward and reverse outputs for tagging or offline sequence analysis. It cannot use the full reverse pass for a live prefix without waiting for future input, so causal tasks need a different latency/accuracy compromise.

Gates provide a controllable memory path

In the convention above, an update gate near zero copies the old hidden state, while a gate near one replaces it with the candidate. The reset gate controls how much old state contributes when forming that candidate. This allows a GRU to preserve a useful signal across many steps without repeatedly multiplying it by an unconstrained recurrent matrix. It reduces a common source of vanishing gradients but does not guarantee perfect long-term memory.

A BiLSTM runs one recurrent network left to right and another right to left, then combines their outputs at each position. In “the bank raised rates,” future words help disambiguate “bank” for tagging or extraction. In streaming speech or next-token generation, those future words are not yet available, so a full bidirectional pass would violate causality or add waiting time. Compare GRU’s simpler state with LSTM’s separate cell and hidden state using actual task accuracy, training stability, and inference latency.

Scaled dot-product attention computes , where masks invalid positions. Self-attention draws queries, keys, and values from the same sequence; cross-attention uses a different source for keys and values. Multi-head attention learns multiple projections, allowing different relationships to be represented.

Attention weights are not automatically faithful explanations of a model’s decision; downstream transformations and correlated features matter. Likewise, a network with no hidden layer can still classify linearly separable data well, but cannot represent arbitrary nonlinear boundaries in the original input space. Feature engineering can make a linear classifier expressive in transformed features. Attention Is All You Need specifies the standard scaled dot-product formulation.

Attention computes a contextual mixture

For a query whose scores against two keys are equal, softmax assigns each weight 1/2 and the output is the average of their value vectors. If one score exceeds the other by , its weight becomes 3/4. Learned projections decide which similarities matter, while masking prevents access to padding or future tokens. The values carry the information being combined; keys and queries determine the combination.

Removing all hidden layers leaves an affine map followed by the chosen output function. Binary logistic regression can separate points with a linear boundary, but cannot classify XOR in the original two-dimensional coordinates: opposite corners belong to the same class. Adding nonlinear hidden units or an engineered interaction feature makes that pattern representable. Multiple affine layers without intervening nonlinearities also collapse to one affine map. Attention itself is input-dependent because its weights come from softmax over similarities, so it should not be confused with a fixed weighted average or a purely linear layer.

With true positives TP, false positives FP, and false negatives FN, precision is , recall , and F1 . Define behavior when denominators are zero. For rare positives, inspect precision-recall curves and performance at realistic prevalence; a seemingly strong ROC curve can conceal poor precision.

A safety alert may favor recall but still needs a tolerable false-alarm load. A manual fraud-review queue may optimize recall subject to a precision or review-capacity constraint. Report class-specific metrics and macro/micro aggregation rules for multiclass problems. Calibrate probabilities when decisions require expected costs, and validate thresholds on temporally separate data to account for distribution shift.

Choose a threshold using consequences

Suppose a detector finds 80 of 100 true cases and raises 20 false alarms. Precision and recall are both 0.8, so F1 is 0.8. Lowering the threshold might find 95 cases but raise 95 false alarms: recall becomes 0.95, precision 0.5, and F1 about 0.655. Whether this is better depends on the cost of misses and the capacity to investigate alerts, not the F1 number alone.

F1 ignores true negatives and gives equal harmonic emphasis to precision and recall. It is useful for some retrieval tasks but does not encode every operational objective. A support review team may require at least 90% precision and maximize recall subject to that limit. A medical screening workflow has different downstream costs and requires domain-specific validation. Select thresholds on a heldout set, then monitor changing prevalence: even unchanged sensitivity and specificity can yield very different precision when positives become rarer.

For imbalance, use class-weighted loss or resampling inside training folds and evaluate precision-recall at realistic prevalence. Adjust thresholds to costs; oversampling is not a substitute for gathering rare positive examples. For collinearity, inspect redundant features and use ridge or elastic net when stable coefficients matter; prediction can remain good despite unstable individual coefficients.

Fit feature selectors only on training data, retaining domain-critical variables and checking stability across folds. Tune regularization strength alongside model capacity using grouped or temporal validation when needed. Compare calibration, subgroup performance, and operational cost. Investigate missingness and leakage first: a highly predictive feature recorded after the outcome is an invalid shortcut, regardless of regularization.

Build a leakage-safe experiment

Consider rare churn prediction with many correlated billing features. Split customers by time first, then fit imputation, scaling, feature selection, and any resampling within each training fold. A cancellation timestamp recorded after churn is leakage and must be removed, even if regularization gives it a small coefficient. Duplicating rare positives before splitting can place copies in both train and validation and inflate performance.

Use a simple regularized baseline to identify whether complexity adds value. Class weighting changes the optimization’s emphasis; threshold selection changes the decision rule; probability calibration addresses whether predicted probabilities match observed rates. These are distinct steps. Correlated predictors may not hurt a tree’s predictions but can distort importance rankings or destabilize linear coefficients. Evaluate selected features across folds and prefer stable, available-at-serving-time inputs. Report the review workload or cost at the chosen threshold alongside precision, recall, and calibration, so the solution addresses the actual imbalanced decision problem.

Choose user-, account-, or cluster-level randomization according to spillovers. Validate exposure logging and run sample-ratio mismatch checks before interpreting results. Set duration to cover important cycles, define exclusions in advance, and estimate effects with confidence intervals. Repeated unplanned peeking inflates false positives; use a valid sequential design if early stopping is required.

Inspect guardrails such as latency, errors, retention, and subgroup harm. Distinguish statistical significance from practical value and account for multiple comparisons. A null result can mean no effect, insufficient power, implementation failure, or dilution from non-exposure. For network or marketplace features, user-level independence may fail; cluster randomization or switchback designs can be more appropriate.

Interpret the statistical evidence

A p-value is the probability, under the specified null hypothesis and assumptions, of obtaining a test statistic at least as incompatible with the null as the observed one. It is not the probability that the null is true, nor the probability the result will replicate. Report an effect estimate and confidence interval to show magnitude and uncertainty.

For an illustrative binary metric rising from 10.0% to 10.5%, the absolute effect is 0.5 percentage points and the relative change is 5%. Under independent user outcomes, 80% power, and a two-sided 5% significance threshold, a rough normal approximation needs about 58,000 users per arm. Clustering, repeated observations, or multiple testing can increase that requirement. Decide the minimum useful effect before launch. Confirm assignment and exposure logs, then examine guardrails and practical benefit before shipping. A significant tiny change may not justify operational cost, while a wide interval around zero is inconclusive rather than proof of no effect.

For independent observations, . Bayes’ rule gives . A maximum a posteriori estimate is a posterior mode, not the entire posterior; regularized optimization often corresponds to a particular prior.

For a Bernoulli probability, MLE is successes divided by trials. A Beta prior yields a Beta posterior, allowing a posterior mean and credible interval even with few observations. A frequentist confidence interval and a Bayesian credible interval have different interpretations. Choose based on the decision, uncertainty requirements, prior information, and computation available; Bayesian does not mean assumption-free, and MLE does not mean uncertainty cannot be estimated.

Work through a coin example

After eight heads in ten independent flips, the Bernoulli MLE is 0.8. With a uniform Beta(1,1) prior, the posterior is Beta(9,3), whose mean is 9/12=0.75 and mode is 8/10=0.8. The posterior mean differs because it averages over uncertainty rather than selecting the most likely parameter value. A stronger prior would influence this small dataset more.

If all ten flips are heads, MLE is 1, while the same posterior mean is 11/12. This avoids claiming absolute certainty from a small sample, though the conclusion depends on the chosen prior and model. Bayesian prediction integrates over the posterior; plug-in MLE prediction uses the fitted parameter, with uncertainty estimated separately when needed. As data grows under regular conditions, likelihood information often dominates a weak prior. Neither approach repairs a wrong independence assumption or biased sampling: modeling how observations were collected matters before choosing an estimator.

Begin by checking leakage, duplicated entities across splits, and labels derived from future information. A perfectly fitted deep tree may simply memorize IDs or rare combinations. Use temporal or grouped validation when the data requires it, and tune complexity on held-out performance rather than training accuracy.

For random forests, compare maximum depth and minimum leaf size, use feature subsampling, and inspect out-of-bag estimates only when their assumptions fit the sampling process. For boosting, tune learning rate jointly with number of rounds and stop on a valid holdout. Class weighting affects imbalance, not necessarily overfitting. Evaluate calibration and important subgroups, and prefer the simplest model within the acceptable performance range when latency or interpretability matters.

Control how finely the model partitions data

A tree with a leaf for every training example can reproduce labels, including noise. Increasing minimum leaf size requires each decision region to be supported by more observations; limiting depth caps interaction complexity; pruning removes splits whose validation benefit does not justify their cost. These controls directly reduce the tree’s ability to memorize rare patterns.

For a forest, averaging many decorrelated trees reduces variance, but adding trees does not undo leakage or biased labels. For boosting, a small learning rate spreads fitting across more rounds; it needs a suitable stopping rule because enough rounds can still overfit. Use validation curves for depth, leaf size, and rounds rather than changing all controls simultaneously. An illustrative warning sign is training accuracy rising to 100% while temporal validation stalls or declines. Inspect whether a customer ID or post-outcome feature creates shortcuts before merely shrinking the model. Report performance on future periods and rare classes, not only a random split.

The moments are and . Bias correction compensates for initialization at zero; the update divides the corrected first moment by the square root of the corrected second moment plus a stabilizer.

For image classification, compare optimizers at matched training budgets with suitable learning-rate schedules, warmup, augmentation, and regularization. SGD with momentum may perform competitively and uses less state; Adam can help heterogeneous or noisy gradients. Do not change optimizer while leaving all hyperparameters fixed and call that a fair comparison. Check validation accuracy, calibration, convergence speed, and memory.

Adaptive steps are useful but not free

Adam divides each parameter’s smoothed gradient by a smoothed estimate of its recent magnitude. Parameters with consistently large gradients receive smaller normalized steps, while sparse or differently scaled coordinates can still progress. Bias correction uses and so the initial zero state does not systematically shrink early estimates.

SGD with momentum stores one velocity buffer and follows a shared learning-rate scale; Adam typically stores two moment buffers, increasing memory. AdamW applies weight decay separately from the adaptive gradient update, avoiding the coupling between an L2 penalty and per-coordinate scaling. For image classification, either family can work well with appropriate schedules and augmentation. Compare time to a target validation accuracy, final generalization, calibration, and peak memory. A faster initial loss drop does not establish a better final model, and using SGD’s learning rate unchanged for Adam is not a meaningful comparison.

Inspect a stratified sample of false positives and false negatives, including image quality, age, pose, mixed breeds, and annotator disagreement. Define a labeling policy for genuinely ambiguous cases instead of forcing certainty. Use expert review or multiple annotations where needed, and prevent near-duplicate photos from leaking across splits.

Improve cropping or localization if background dominates, add diverse hard examples, and consider a pretrained visual backbone with calibrated outputs. Label smoothing or noise-aware methods can reduce memorization, but cannot replace correct labels. Allow abstention or a broader category when visual evidence is insufficient. Evaluate per-class recall and calibration on independent sources; do not treat a model’s disagreement with a label as automatic proof the label is wrong.

Diagnose whether the evidence is visual or spurious

If most pug images were taken indoors and pit bull images outdoors, a model may learn background rather than breed-related appearance. Evaluate cropped subjects and changed backgrounds to test that hypothesis. If errors concentrate in puppies or mixed breeds, the class definition may be ambiguous rather than the network too small. Introduce an explicit uncertain or mixed category only if it matches the product’s actual needs.

Use a trusted adjudicated test set before relabeling training data. Review disagreement clusters with annotators and maintain correction history. Hard-example mining should include genuinely confusing correct labels and mislabeled examples separately; blindly training harder on wrong labels reinforces noise. A pretrained backbone with adequate image resolution and balanced sampling may outperform a new architecture trained on poor labels. Evaluate precision, recall, and calibration by pose, source, age, and image quality. For consequential use, allow abstention and explain the limits of visual breed classification instead of treating every photo as uniquely identifiable.

Check how assignments change under resampling, small feature perturbations, and reasonable hyperparameter choices. Inspect cluster sizes, outliers, and representative points, and ask domain experts whether the partitions support a useful action. Compare against simple baselines and a no-clustering workflow.

Select a distance metric and scaling appropriate to the data; Euclidean distance over mixed unscaled features may produce meaningless groups. For density-based clustering, account for noise points rather than rewarding arbitrary exclusion of difficult observations. If clusters drive campaigns or anomaly review, measure downstream outcome improvements through a valid experiment. A stable cluster may still encode an irrelevant artifact, so inspect provenance and potential leakage.

Internal quality is not the same as usefulness

Silhouette compares within-cluster distance with distance to the nearest other cluster; it favors compact, separated groups under the chosen metric. Density-based or elongated meaningful clusters may score poorly under that geometry. Conversely, clustering images by camera watermark can yield clean separation with no value for the intended task.

Inspect representative examples, boundary points, and clusters dominated by one data source. Refit on bootstrap samples and measure whether similar groups recur; instability suggests the solution depends heavily on sampling or hyperparameters. For customer segmentation, test whether clusters support different effective actions in a controlled experiment rather than merely producing interesting names. Reserve a small human-labeled audit when possible, even if full ground truth is unavailable. Compare several reasonable feature scalings and distance measures. Report noise/unassigned points and tiny clusters honestly; excluding difficult observations can make internal metrics look better while reducing practical coverage.

Verify assignment, exposure, model version, feature values, and sample-ratio balance. Compare offline replay predictions with logged online predictions for the same inputs. Inspect latency and fallback rates: a better ranker cannot help if it times out and the old system serves most requests.

Then examine the metric. Better click prediction may not improve satisfaction, and a static dataset reflects the previous ranking policy. Look for cohort differences, novelty effects, saturation, and interference between users. Estimate confidence intervals and whether the test could detect the expected gain. Use a small controlled intervention or targeted experiment to isolate the likely mechanism. Keep both negative and positive findings in the model’s evaluation record.

Follow the causal chain

First establish that the new model actually served the treatment traffic with the intended features. A better offline ranker that times out frequently may produce an online experience dominated by fallback results. Compare a sample of online inputs and outputs with offline replay to isolate implementation or feature-parity errors.

Then ask whether the offline target influences the desired outcome. Better click prediction can favor provocative content without improving retention or satisfaction. A historical dataset also reflects the old policy’s exposures, so unseen candidates and position effects limit replay validity. Finally check experimental power, novelty effects, and interference. A small true gain may be undetectable, while a short-lived gain may fade. Use a targeted experiment to test the leading hypothesis—for example, equalize latency or randomize a small candidate subset. Do not keep tuning on the same failed online comparison until a noisy positive result appears; update the evaluation model and run a properly specified test.

Log shown items, positions, context, and actual exposure, including non-clicked results. Small randomized swaps or exploration can identify propensity effects under stated assumptions. Inverse propensity weighting upweights feedback from low-exposure positions, but tiny propensities create high variance; clipping trades variance for bias.

Validate propensity estimates and require overlap: items never exposed cannot be evaluated reliably from click logs alone. Account for trust bias, presentation differences, and user intent changes. Use independent relevance judgments and online experiments to check whether debiasing improves outcomes. Unbiased Learning-to-Rank with Biased Feedback develops the counterfactual approach and its assumptions.

A click is conditional on being seen

Under a simplified examination model, click probability is examination probability at the shown position times relevance conditional on examination. A relevant item at position ten may receive fewer clicks than a mediocre item at position one simply because fewer users see it. Training directly on clicks can then reinforce the original ranking.

If a clicked example had exposure propensity 0.1, inverse propensity weighting gives it weight ten, while an example at propensity 0.5 gets weight two. This compensates for unequal observation under the model’s assumptions. Very small propensities produce unstable weights, and missing support cannot be fixed: an item never shown has no observed outcome to reweight. Randomized swaps can estimate position effects while limiting user impact. Check whether examination depends on context or presentation, not position alone. Evaluate debiased learning with independent relevance labels and online outcomes; a mathematically corrected estimator can still fail when its propensity model is wrong.

This is a representative family list, not a ranking or exhaustive catalog of the newest releases. Check each checkpoint’s license, model card, architecture, training-data disclosure, and permitted uses. A family can contain dense, mixture-of-experts, and multimodal variants; do not assign every release the properties of one model. Start with the official Llama, Qwen, DeepSeek, and OLMo resources.

Compare releases along separate dimensions. Weights let a user run inference locally. Training and inference code help inspect and reproduce implementation behavior. Tokenizers, data descriptions, data-processing code, and intermediate checkpoints improve reproducibility. A license determines permitted use and redistribution; downloading a weight file alone does not answer those questions.

For an application, first establish the required capability and deployment budget, then compare exact checkpoints on the same task. A dense model activates most parameters for each token; a mixture-of-experts model can activate a subset but still require storage and communication for its experts. A multimodal model also needs the corresponding encoders and preprocessing, so the text parameter count alone does not describe the whole system.

For example, choosing a model for offline document extraction requires checking language coverage, structured-output reliability, memory at the desired context length, and whether the intended distribution is permitted. Keep the checkpoint revision and tokenizer together. The family names in the brief answer are representative examples, not a current performance ranking or a claim that every version shares one architecture or license.

For translation, a causal decoder generates French tokens while attending to all preceding prompt and output tokens. It does not lose access to the English prompt. A prefix LM lets the English prefix tokens contextualize one another bidirectionally, while output tokens attend to the prefix and preceding outputs. An encoder-decoder encodes the source bidirectionally, then a causal decoder cross-attends to those representations. The distinction is the attention connectivity, not merely the wording of the prompt.

Use a four-position example with source tokens and target tokens . In a causal decoder, the representation of cannot see , while the representation used to predict a target can see the preceding source and target tokens. In a prefix decoder, both source representations can see both source tokens; target representations retain causal restrictions. In an encoder-decoder, the source has its own bidirectional stack and the target stack reads it through separate cross-attention projections.

This affects where work is performed. An encoder-decoder computes the source representation once and can cache its cross-attention keys and values. A causal decoder caches the prompt’s keys and values in the same stream used for generated tokens. Both can generate incrementally; a prefix decoder requires the whole bidirectional prefix before computing its representations correctly.

Choose architecture and objective together. Translation naturally separates source and target, while general continuation naturally fits a single stream. Merely labeling a prompt “source” does not change an already-trained causal model into a prefix-attention architecture.

BERT predicts selected masked tokens, while T5 reconstructs corrupted text. Decoder-only causal models maximize sequence likelihood. Supervised fine-tuning trains on desired responses, often masking prompt tokens from the loss. Preference learning and reinforcement learning optimize preference or reward objectives. Always specify the model and training stage before saying “the LLM objective.”

The objective determines which conditional distribution receives supervision. In next-token training, “the cat sat” yields targets for later tokens given earlier ones. In masked-token training, the model might reconstruct “cat” from “the [MASK] sat,” using information on both sides. Denoising sequence-to-sequence training can remove spans and train a decoder to reconstruct them. These objectives expose different information and therefore produce different learning signals.

During SFT, an instruction and context often enter the model as input, while the loss is applied only to the desired response. The mathematical token loss can remain cross-entropy even though the data distribution and masking have changed. Preference training instead compares responses or optimizes a reward-related objective; its supervision is not simply another unlabeled document.

A low loss is meaningful only relative to that objective. Predicting common tokens accurately does not establish reliable tool execution, factual answers, or obedience to complex constraints. Evaluate those capabilities explicitly and audit the examples supplying supervision. Architecture names and training objectives are related historically but should not be treated as interchangeable definitions.

Many tasks can be serialized as a prefix followed by a completion. A single stack simplifies training infrastructure, and cached keys and values support incremental decoding. Their prevalence also reflects successful scaling recipes and ecosystem investment. They are not inherently the fastest or best for every task: encoder-decoder models can be competitive for conditional generation, and encoder-only models can be efficient for classification or retrieval. Long-context quality depends on more than the architecture label.

A causal model can learn from ordinary text without constructing a separate input-output pair for every task. Every prefix supplies a context and each next token a target. At inference, tasks such as translation, classification, code completion, and conversation can all be represented as a prompt followed by a completion. This common interface makes data mixing and serving infrastructure convenient.

The causal structure also permits KV caching: after processing a prompt, previous token representations do not need to be recomputed when another token is appended. Training can still evaluate all token positions in parallel because the causal mask prevents future-token leakage.

There are costs. Generating a one-word label through a large autoregressive model may be slower than a small encoder with a classifier head. Bidirectional source representations can be valuable for conditional tasks, and a decoder’s full prompt competes with generated text for cache capacity. The prevalence of decoder-only systems reflects a successful combination of objective, scale, tooling, and flexible task serialization; it does not prove a universal architectural superiority or that a separate encoder is obsolete.

BERT-style encoders commonly use bidirectional attention for classification, tagging, and reranking. A suitable pooling method and retrieval training are still needed for high-quality sentence embeddings. GPT/Llama-style decoders predict the next token using a causal mask. T5/BART-style encoder-decoders use bidirectional source attention plus causal output attention and cross-attention. These are architectural tendencies rather than absolute restrictions on possible tasks.

Consider three tasks on the same review text. Sentiment classification needs a fixed label: an encoder can contextualize the whole review and feed a pooled representation into a small classifier. Translating the review needs a new sequence: an encoder-decoder can encode the review once and generate the translation with cross-attention. Writing a continuation needs an open-ended next-token distribution: a decoder-only model handles that directly.

The attention masks explain the distinction. An encoder token can usually read tokens before and after it. A causal decoder token can read preceding tokens but not future output. An encoder-decoder contains both patterns plus a bridge from decoder queries to source keys and values. Padding still needs masking in all three.

These choices affect training and latency, not merely supported tasks. A decoder can classify by generating a label, and an encoder can produce useful embeddings after appropriate training. Compare the complete pipeline, including pooling, task heads, output validation, and generation length. Raw pretrained representations are not automatically optimized for sentence similarity or any other downstream objective.

For prefix length , prefix queries can attend only to prefix positions, while a suffix query at position can attend to every position up to . Both usually include the position itself; the representation predicts the next token. Prefix language modeling is not prefix tuning: prefix tuning learns small task-specific continuous prefixes and need not change a causal model’s mask.

For an input prefix of two tokens followed by two output positions, the allowed attention pattern is conceptually:

          p1 p2 o1 o2
p1         1  1  0  0
p2         1  1  0  0
o1         1  1  1  0
o2         1  1  1  1

A one means that the row may read the column; zeros are masked. The ordinary causal mask would replace the first row’s second entry with zero. The prefix therefore gets bidirectional contextualization without revealing future output to suffix predictions.

The loss must respect this information flow. Applying ordinary next-token loss indiscriminately inside the bidirectional prefix can leak the next token through attention. Prefix-LM training generally treats the designated prefix as conditioning information and supervises the continuation according to the chosen objective. At generation time, cache the fully processed prefix, then append suffix tokens causally. Changing a pretrained model’s mask at inference without compatible training is not a reliable substitute for training it to use that mask.

For token vector :

BatchNorm’s reduction axes depend on the layer and data layout; it commonly uses running statistics at inference. Batch dependence, padding, and train/inference differences make it awkward for autoregressive language models. RMSNorm saves centering work, but speed and training stability depend on kernels and the full architecture.

For , ignoring epsilon and using unit scales, LayerNorm subtracts mean 2 and divides by standard deviation 1, producing . RMSNorm divides by , producing approximately . RMSNorm controls magnitude while retaining the common offset; LayerNorm removes that offset. Learned scale and optional bias then allow the model to adjust the normalized features.

For a Transformer tensor shaped [batch, tokens, hidden], ordinary LayerNorm and RMSNorm operate independently on each token’s hidden axis. Their output for one token therefore does not depend directly on the other examples sharing its batch. BatchNorm generally estimates statistics across examples and possibly spatial positions for each channel, making batching and padding relevant to the estimate.

Epsilon prevents division by zero and affects nearly constant or very small vectors. Mixed-precision kernels often accumulate reductions more accurately than the input storage dtype. Normalization placement remains a separate architectural decision: replacing LayerNorm with RMSNorm does not specify whether it occurs before or after a residual addition.

Its sublayer has the form . For a decoder-only stack of blocks, the paper gives and initialization gain for selected projections. Encoder-decoder stacks use different constants. This is a fourth-root rule in that case, not a generic square-root scaling rule, and it is not simply an alternative normalization statistic.

The purpose of the two coefficients is to control the relative scale of the residual stream and newly added transformations as depth grows. A larger residual multiplier keeps each sublayer’s initial perturbation modest relative to the carried state; shrinking selected initial weights also limits the transformed branch. LayerNorm then rescales the combined result. Neither operation alone fully describes the proposed recipe.

For a decoder-only example with , and . These are dimensionless coefficients. They are not a learning-rate schedule and should not be applied indiscriminately to every parameter tensor. The selected projection matrices and initialization conventions matter.

To assess such a change, inspect loss, activation scale, and gradient behavior as depth increases, comparing otherwise controlled configurations. Successful optimization is distinct from useful scaling: a stable very deep model can still be inefficient for a latency-sensitive application. Also distinguish training a model with DeepNorm from altering the residual coefficients of an existing checkpoint, which changes its learned computation and can destroy behavior.

The DeepNet study demonstrated Transformers up to 1,000 layers and reported gains on its translation benchmarks. Residual scaling introduces little computation, but it must accompany the prescribed initialization. The results do not establish that any 1,000-layer LLM will train well, outperform a shallower model, or be efficient to serve. Depth still adds sequential computation.

The immediate advantage is optimization stability at depths where an ordinary post-normalized stack may be difficult to train. If each layer causes an uncontrolled change, increasing the number of layers can amplify that problem; coordinating residual scale and initialization aims to keep accumulated changes manageable. This lets depth become a practical experimental axis instead of causing a run to diverge before quality can be assessed.

A fair comparison holds the training budget, data, and evaluation task as constant as possible. Compare loss trajectories and final task quality, then measure wall-clock cost. A narrower deep model and a wider shallow model with similar parameter counts need not have similar throughput: the deep model has more sequential layer dependencies and additional activation handling.

For a deployment decision, separate three claims: the architecture can be optimized, its benchmark quality improves, and its serving cost is acceptable. Evidence for the first does not imply the other two. DeepNorm is most relevant when residual-stack optimization is the bottleneck; it does not address bad training data, insufficient memory, or the cost of reading a long KV cache during generation.

Two common forms are:

Pre-norm gives gradients an identity path through the residual stream and often simplifies optimization at depth. It does not by itself bound residual magnitudes. Post-norm can work well with suitable initialization and learning-rate schedules, including deep variants such as DeepNorm. “Sandwich norm” is not a single universally identical implementation; specify the exact equations.

In pre-norm, the derivative of the residual addition contains an identity contribution: a gradient can travel through without passing entirely through the sublayer. This helps explain its optimization behavior. In post-norm, the gradient through also passes through normalization at each block, so initialization and the relative scale of the branches affect propagation differently.

The activation behavior differs too. Pre-norm adds each branch output to an unnormalized residual stream, so its magnitude can grow across layers even though sublayer inputs are normalized. A final normalization is commonly used before prediction. Post-norm explicitly normalizes after each residual addition, but that does not guarantee stable gradients for arbitrary depth and initialization.

For a checkpoint conversion, copying weights while moving the normalization layer is not semantics-preserving. The same matrix now receives a different input. When comparing variants, specify the formula, the final norm, initialization, residual coefficients, and any learned bias. Those details are more useful than assuming “pre” or “post” alone determines accuracy.

The original Transformer uses post-LayerNorm. T5 uses a mean-free normalization closely related to RMSNorm, and Gemma 2 adds normalization around sublayers beyond a simple pre-norm layout. Therefore, “all modern LLMs use pre-RMSNorm” is too broad. Confirm the normalization class, placement, epsilon, and learned scale/bias in the model configuration and implementation.

A checkpoint’s normalization choice has two independent parts: the statistic being computed and its location in the block. Reading only a configuration field called layer_norm_eps does not establish that the implementation subtracts the mean, and seeing an RMSNorm class does not establish that there is only one normalization per sublayer.

Trace one block in order: input state, normalization, attention, residual addition, normalization, FFN, residual addition, and any output normalization. Then inspect whether each normalization has learned scale, learned bias, or an offset convention for the scale. This tells you what happens to a token vector and which checkpoint tensors are expected.

For example, implementing a BERT-style post-norm block using a GPT-2-style pre-norm order while loading the same tensor shapes can pass shape checks yet compute a different function. Likewise, a different epsilon can matter near a low-variance input. The named model examples refer to specific published designs. New releases and converted checkpoints should be checked individually rather than inheriting assumptions from the family name.

Using column vectors:

where , , and . The hidden width usually exceeds . Gated variants use two input projections and multiply their outputs before projecting back; applying SiLU to a single projection is not the same as SwiGLU. Some architectures omit biases. Attention mixes tokens; the FFN mixes features within each token.

The same matrices are applied to every token position. For a batch of length , the FFN can process all vectors together, but no token reads another token inside this operation. Information from other positions has already entered the vector through attention. The nonlinearity is essential: without it, two linear projections collapse into a single linear transformation, limiting the role of the expanded hidden space.

Ignoring biases, the two matrices contain parameters and require roughly multiply-accumulate operations per token. With , that is parameters. The output returns to width so that it can be added to the residual stream. The intermediate activation of width contributes memory during training.

For a concrete shape example, a 768-dimensional token projected to width 3,072 becomes a 3,072-element nonlinear feature vector, then returns to 768 dimensions. In a gated FFN, a second expanded projection multiplies the first; this increases the number of matrices and changes the hidden-width choice for a matched parameter budget.

Its exact and common approximate forms are:

Unlike ReLU, GELU smoothly attenuates negative values rather than setting all of them to zero. Exact and approximate implementations have slightly different numerics, so preserve the checkpoint’s activation convention.

The factor acts as a smooth, input-dependent attenuation. At zero, the output is zero and the derivative is one half. At , , so GELU returns about 0.8413; at , it returns about . A large positive input is almost unchanged, while a sufficiently negative input is pushed toward zero from below.

Its derivative is , where is the standard normal density. Consequently, the function is not simply a probabilistic on/off switch executed at runtime, and it is not monotonic everywhere. The smooth transition supplies a different gradient shape from ReLU’s hard zeroing of negative values.

In a Transformer FFN, apply GELU to the expanded feature vector before the output projection. Choosing the tanh approximation can change numerical results slightly; using a different activation when loading existing weights changes the model rather than merely optimizing its implementation. Judge speed and accuracy for the actual kernel and model, since a mathematical formula alone does not predict end-to-end throughput.

Here is the logistic sigmoid. Swish is smooth and non-monotonic: small negative inputs can produce negative outputs, while large positive inputs approach the identity. Some definitions allow a learned or fixed ; many Transformer implementations simply use SiLU with . A SwiGLU FFN combines that activation with a separate multiplicative projection.

With , evaluate SiLU at three simple points: , , and . The sigmoid factor suppresses strongly negative values and leaves strongly positive values almost unchanged. Unlike sigmoid itself, the resulting activation is not bounded above.

The derivative is . At zero it equals one half. For some negative inputs it becomes negative, explaining the non-monotonic region. As a positive beta grows very large, the function approaches ReLU away from zero; at beta zero it is simply .

A model using SiLU in an ordinary two-projection FFN is different from a SwiGLU model. SwiGLU multiplies the activated branch by another learned projection, creating feature-dependent interactions before the output projection. Preserve the checkpoint’s definition and surrounding block structure. Smoothness can help optimization in a particular setting, but it does not guarantee a superior model or remove the need for normalization and suitable initialization.

With column vectors and hidden width :

and . The gate and value vectors must have the same shape. The elementwise product controls each hidden feature separately; it is not an attention distribution over tokens.

Separate the branches conceptually. The value projection proposes hidden features; the gate projection determines how strongly each is transmitted. If a gate logit is zero, its sigmoid is 0.5, so that value coordinate is halved. A gate logit of 2 yields about 0.881, while -2 yields about 0.119. Different tokens can therefore pass different combinations of features through the same learned matrices.

The gate is not a probability distribution over features: its coordinates do not sum to one. Nor does it decide which source token to attend to. It modulates features within one token, then the output projection combines those gated features back into the model width.

Ignoring biases, the block has parameters, compared with for a conventional FFN at the same hidden width. Reducing by roughly two thirds matches that projection parameter count, subject to implementation rounding. Sigmoid gates can saturate and yield small gate gradients, so initialization and activation scale matter. Replacing the sigmoid with GELU or SiLU creates a different gated variant, not an equivalent algebraic rearrangement.

A bias-free column-vector form is:

Both input projections have shape , and the output projection has shape . Unlike a sigmoid gate, the GELU branch is not bounded between 0 and 1. This block has three projection matrices, so comparisons with an ordinary two-projection FFN should account for hidden width and parameter count.

GEGLU has two learned views of the same input. One becomes a smooth nonlinear gate; the other remains a linear value branch. For a gate preactivation of 1 and a value of 2, the product is approximately . With gate preactivation -1, the same value produces approximately -0.3174. Thus the gate can attenuate, change sign, or amplify features rather than merely admit a fraction between zero and one.

This multiplicative interaction increases expressiveness within the FFN. The output projection then mixes gated features and restores the residual width. The operation still acts independently at each token position; it does not replace attention’s communication across positions.

For a conventional hidden width of , a two-matrix FFN has about parameters. A three-matrix GEGLU block achieves the same count at . Actual models round widths for hardware alignment and may use biases. A comparison that keeps the same hidden width silently gives GEGLU more parameters, so report the architecture budget when interpreting a quality difference.

The common bias-free form is:

The elementwise product is projected back to the model width. For comparable parameter counts, a gated FFN’s hidden width is often reduced relative to a standard FFN, though exact ratios depend on the architecture and rounding. The activation alone does not guarantee an accuracy or speed improvement.

Let the gate branch output and the value branch output . The expanded vector is , which then passes through . The value branch remains independently learned; multiplying SiLU by its own input again would compute a different function. Some implementations name the projections gate_proj, up_proj, and down_proj, but the tensor operation is the defining property.

For one coordinate with and , the gated value is about 1.4622. With , it is about -0.5379. SiLU gates are not bounded between zero and one and need not suppress every negative feature entirely.

The approximate parameter count is , so choosing gives the projection count of a standard FFN. This is a budget calculation, not a required hidden-size rule. During inference, fused kernels can reduce intermediate memory movement; during training, the expanded branches still contribute activation memory. Benchmark the actual implementation rather than inferring speed from the activation’s name.

The original Transformer used ReLU. Names such as SiLU in a configuration describe a pointwise activation; the surrounding multiplication and projections determine whether the full block is SwiGLU. Inspect a specific version rather than assuming all models in a family share the same FFN.

The useful distinction is between a pointwise activation and a complete FFN design. BERT’s ordinary FFN projects up, applies GELU, and projects down. A SwiGLU FFN performs two input projections, applies SiLU to one, multiplies them, then projects down. Those are different computation graphs even if a configuration exposes just one activation string.

When loading a checkpoint, verify the number of projection matrices and their shapes. Two matrices generally indicate an ordinary FFN, while a gate/value/output layout indicates a gated one. Then inspect the exact activation convention, including GELU approximation or any scaling. Checkpoint tensor names can help, but implementation is the final authority.

This matters for both correctness and comparisons. Replacing GELU with SiLU in a pretrained ordinary FFN does not turn it into a pretrained SwiGLU network, and matching hidden width across gated and ungated blocks does not match parameter count. The model examples above refer to their published versions. Evaluate alternative activations through controlled training or adaptation rather than assuming a fashionable activation is a drop-in quality improvement.

Vanilla SGD uses . Adam maintains exponential moving averages of and , bias-corrects them, and updates with . These two moment buffers increase optimizer memory. AdamW additionally decouples weight decay and is a common Transformer training choice. Neither optimizer universally generalizes better; compare tuned schedules and equal training budgets.

Adam’s first moment smooths the direction of recent gradients, while its second moment estimates their squared scale. Bias correction compensates for starting both averages at zero. Dividing by the square root of the second moment makes the step sensitive to each coordinate’s history, so two parameters with different gradient scales can receive more comparable updates.

For vanilla SGD with learning rate 0.01 and gradient 2, the update is -0.02. For a first Adam step with zero-initialized moments, bias correction gives and , so the update is approximately -0.01 when epsilon is negligible. This is a calculated illustration, not evidence that Adam’s step is always smaller. Later steps depend on the entire moving-average history.

Momentum SGD usually stores one extra tensor; Adam stores two, often in higher precision. That matters for large-model memory planning. Tune learning rate, warmup, decay schedule, batch size, and weight decay for each optimizer rather than copying one configuration. Compare held-out quality and time to reach it, since a cheaper update can still need more updates overall.

For length and hidden width , the score/value products cost ; projections add . A naive multi-head implementation stores attention entries. During cached one-token decoding, attention over an existing context is linear in that context length, but reading the growing KV cache can become a bandwidth bottleneck. Large context windows also create a modeling challenge: access to distant evidence does not guarantee its effective use.

For an 8,192-token sequence, one head’s score matrix contains entries. At two bytes per entry that is 128 MiB for one matrix, before additional intermediates. Thirty-two heads would require 4 GiB just for that materialized matrix. Doubling sequence length quadruples this term. This calculation explains why a model can run out of memory even when its parameter weights fit comfortably.

Training and prefill compute many query positions at once, allowing large matrix multiplications. Cached decoding computes only one new query per sequence but reads earlier keys and values. Its arithmetic may be modest while transferring cache data consumes substantial time. Weight reads, FFN work, and communication can also dominate depending on batch size and hardware.

Diagnose by measuring the intended phase and shape. An optimization that accelerates long-prompt prefill may have little effect on short-context decoding. Finally, resource efficiency and reasoning quality are separate: avoiding an out-of-memory error does not establish that the model can combine facts spread across a long document.

FlashAttention preserves dense softmax attention while improving its execution. Sliding-window or block-sparse attention reduces attended pairs; MQA/GQA reduce KV cache size; low-rank and kernel-based methods change the approximation or representation. Retrieval can reduce how much context must enter the model. Choose based on the actual bottleneck—prefill compute, decode bandwidth, memory, or quality—and measure the relevant workloads. These techniques are often complementary.

Start by naming the resource being reduced. A fused exact-attention kernel avoids storing a large intermediate matrix while keeping the same allowed token interactions. A local mask changes those interactions, reducing work but preventing direct access to distant tokens. Sharing KV heads reduces stored and transferred keys and values, while retaining separate query heads. These mechanisms solve different bottlenecks and can be combined in one model.

For a document chat system, long prompts may make prefill expensive, while many simultaneous conversations make KV storage the capacity constraint. Retrieval can reduce prompt length; GQA can reduce per-conversation cache; an efficient attention kernel can reduce prefill memory traffic. A local-attention model might process longer streams but needs other paths to connect distant evidence.

Measure time to first token, time per output token, throughput at realistic concurrency, memory, and answer quality. Compare against the same prompt and output lengths. If reducing context removes the needed exception from a policy, a faster answer is not an improvement. Architectural approximations generally require compatible training, while an exact backend replacement should be checked for numerical and mask equivalence.

Self-attention versus cross-attention identifies where queries and keys/values originate. Causal, local, and block-sparse masks define connectivity. MHA, GQA, and MQA define KV sharing. Relative biases and RoPE encode position. Linear attention changes the attention formulation; FlashAttention is an efficient algorithm for supported attention operations. These categories overlap: a causal GQA layer with rotary embeddings can run using a FlashAttention kernel.

A concrete layer can be described along several axes simultaneously: causal self-attention, 32 query heads, 8 KV heads, rotary position encoding, and an optimized fused execution kernel. “Causal” says which positions are accessible; “self” says where tensors originate; GQA says how keys and values are shared; rotary encoding says how positions affect matching. None of these labels substitutes for the others.

For a translation decoder, its self-attention may be causal GQA while a separate cross-attention layer reads all valid encoder positions. A sliding-window mask can be added to restrict self-attention to nearby tokens. Linear attention instead changes the aggregation formula and maintained state, so it is not simply another mask or implementation backend.

When asked to compare variants, choose the axis relevant to the question and state its consequence. Head sharing primarily changes cache size and K/V representation capacity; connectivity changes accessible evidence; execution kernels change how the operation uses hardware. This classification avoids claims such as “FlashAttention replaces cross-attention” or “all sparse attention is hard attention,” which conflate independent design choices.

Separate KV projections for every query head increase stored state during autoregressive decoding. The cost depends on head dimensions, sequence length, batch size, and layer count. GQA/MQA reduce KV heads, while optimized kernels address memory traffic. Redundant heads are an empirical finding for some models, not proof that arbitrary heads can be removed without retraining or quality loss.

Each MHA head has its own projected key and value features. During generation, those features for earlier tokens must remain available, so storage grows with layers, sequences, tokens, heads, and head width. With a fixed model width, increasing head count while proportionally decreasing head width does not necessarily increase total KV elements; compare actual dimensions rather than head count alone.

Separate heads can learn different relationships, such as local syntax or distant references, but useful diversity is not guaranteed. Pruning a head based only on a low average attention weight is unreliable: a rare but important query may depend on it. Measure quality after pruning, preferably with adaptation, rather than treating apparent redundancy as proof of dispensability.

For serving, alternatives include GQA or MQA to reduce KV projections, quantizing cached tensors where supported, and optimized kernels to reuse memory efficiently. These have different accuracy and compatibility implications. Converting a trained MHA checkpoint by simply dropping most KV heads changes the function; it requires an explicit conversion and validation strategy. Profile first because FFN compute or inter-device communication may dominate the complete model.

Each query head still computes its own attention weights, so shared K/V does not mean identical attention outputs. With query heads, KV cache storage can fall by roughly relative to MHA at the same head dimension. The Q projection and output projection remain, and score computation still serves every query head. This trades some representational flexibility for lower memory traffic.

With 32 query heads and head width 128, MHA produces 4,096 key features and 4,096 value features per token per layer. MQA produces only 128 of each. Every query head compares its own 128-dimensional query with that shared key sequence and mixes the shared values using its own weights. The resulting head outputs can differ because the queries differ.

An implementation should preserve sharing in storage. Physically repeating the one KV head 32 times into a cache would discard much of the memory benefit, even if it produces the correct mathematical output. A suitable attention kernel reuses the compact K/V representation across queries.

The tradeoff is that the keys and values have fewer independently learned feature subspaces. This can affect quality, particularly if applied to an existing MHA model without appropriate training. GQA offers an intermediate number of KV heads. Compare accuracy together with latency and maximum useful batch size; a smaller cache can allow more concurrent requests even when a single request’s latency changes only modestly.

For query heads and KV heads:

The ratios assume equal layers, batch size, context length, head dimension, and dtype. GQA offers a tunable compromise; quality and speed depend on the checkpoint and implementation.

For 32 query heads and 8 KV heads, partition the query heads into eight groups of four. Each group reads its assigned K/V representation, but each query still computes independent scores. At one token position, MHA stores 32 key vectors and 32 value vectors, GQA stores eight of each, and MQA stores one of each. Queries are transient for the new token; keys and values accumulate in the cache.

The ratio in the table applies to KV storage, not to total model memory or latency. Weights, activations, output projections, and temporary buffers remain. If model weights dominate the memory budget, a fourfold cache reduction does not mean a fourfold reduction in overall GPU use.

Selecting a variant is an architectural and serving decision. More KV heads provide more separate feature representations; fewer can improve cache capacity and bandwidth use. Existing checkpoint conversion must account for training and quality effects. At deployment, confirm the backend supports the intended grouping without expanding the cache into duplicated heads, and compare equal sequence lengths, batch sizes, precision, and quality targets.

Its largest benefit is commonly in memory-bandwidth-limited decoding. It does not reduce all model FLOPs by the query-head count, eliminate attention’s sequence-length dependence, or guarantee equal quality. If the FFN or another subsystem dominates latency, end-to-end gains can be smaller. Compare tokens per second, time per output token, peak memory, and task quality at matched workloads.

Imagine cache capacity is the limiting factor for simultaneous conversations. Reducing the number of KV heads decreases the bytes needed for every stored token, so the same available memory can hold more requests or longer contexts. Larger batches can then amortize weight reads and improve aggregate throughput. This is a capacity benefit even when individual-request latency is constrained by something else.

During one-token decoding, the new query still needs to interact with earlier positions for every query head. MQA therefore does not divide all attention arithmetic by the query-head count. Instead, compact shared keys and values reduce how much unique cached data must be transferred, provided the kernel reuses it effectively.

Use a workload matrix rather than a single throughput number: vary prompt length, generated length, concurrency, and cache precision. Record whether the comparison is latency per request or total tokens across requests. Check held-out quality because reducing independent K/V representations can change model behavior. If the application is dominated by long FFNs, network latency, or tool calls, those components limit the end-to-end speedup.

These version-specific examples show why a family name alone is insufficient. Check the ratio between query-head and KV-head counts in the checkpoint configuration; equal counts indicate MHA, one KV head indicates MQA, and an intermediate count indicates GQA. Some models use other compressed attention designs that should not be classified solely by this rule.

Read num_attention_heads, num_key_value_heads, and the head dimension in the specific checkpoint configuration. For an ordinary attention implementation, 32 query heads with 32 KV heads indicate MHA; 32 with eight indicate GQA; 32 with one indicate MQA. Some implementations omit a separate KV field when it defaults to the query-head count, so inspect the implementation rather than interpreting a missing field as one.

The choice affects deployment planning. A model with more total parameters can still use less KV memory per token than another model if it has fewer layers, KV heads, or smaller head dimensions. Weight-memory and cache-memory comparisons must therefore be performed separately.

The named examples are historical checkpoint designs, not statements about every later family member. A quantized checkpoint usually retains the attention grouping while changing numerical representation, but converted models may alter other details. Compressed-latent or hybrid recurrent architectures need their own state-size calculation; forcing them into a simple MHA/GQA/MQA label can hide the actual memory layout. Verify tensor shapes and supported serving kernels before using a family name to estimate capacity.

It tiles Q, K, and V into on-chip memory and uses an online softmax calculation to accumulate outputs without storing the full score/probability matrix in GPU high-bandwidth memory. Backward computation recomputes selected intermediates. Dense attention still has quadratic arithmetic in sequence length; the improvement is IO efficiency and intermediate memory use, with normal floating-point differences.

A naive implementation writes attention scores to device memory, reads them to apply softmax, writes probabilities, then reads those probabilities to multiply by values. The large intermediate matrices can cause expensive traffic even when the final output is only one vector per token. Tiling keeps smaller blocks near the compute units and accumulates their contribution directly.

The online softmax must remain correctly normalized as new key blocks arrive. Maintain a running maximum, a running exponential sum, and a weighted value sum for each query row. If a later block raises the maximum, rescale the earlier sums before combining them. After all allowed blocks, divide the weighted sum by the normalizer. This is why the algorithm can avoid retaining every score while producing the same mathematical softmax result.

“Exact” describes the attention operation, not bit-for-bit identity across different floating-point reduction orders. Check output and gradient tolerances, causal masks, padding, and supported shapes. Dense pairwise arithmetic still grows quadratically during full-sequence processing; reduced memory traffic does not turn dense attention into a linear-attention model.

Avoiding the full attention matrix can make longer sequences feasible. Gains depend on sequence length, dtype, head dimension, GPU, masks, and kernel version; short workloads may gain little. It does not remove the autoregressive KV cache or make dense attention linear-time. Distinguish the original algorithm from later implementations when quoting benchmarks.

The principal memory saving is in intermediates: the algorithm need not keep every query-key score and probability for backward or output computation. That can make a larger sequence or microbatch fit, which may improve useful throughput beyond the speedup of one kernel. Recomputation during backward spends arithmetic to avoid expensive stored intermediates.

Separate this from model-state storage. Parameters, optimizer moments, and the autoregressive KV cache still occupy memory. If a training run is dominated by optimizer state rather than attention matrices, the total saving may be modest. Likewise, short sequences can offer less opportunity to amortize kernel setup or exploit tiling.

Benchmark using the target mask, dtype, sequence length, head dimensions, and hardware. Report whether the result is attention-kernel time, a complete training step, or end-to-end request latency. Include compilation or warmup consistently. An application may become faster because a larger batch now fits, but that is a different experiment from comparing equal batch sizes. The algorithm enables more efficient execution of supported attention; it does not establish the model’s ability to reason over the extra context.

Actual dispatch depends on the framework version, hardware, dtype, mask, and requested features. A PyTorch scaled-dot-product-attention call may choose a fused kernel or fall back to another implementation. Requesting attention weights can also affect the path. Verify the configured backend and profiler output rather than inferring it from the checkpoint name.

The checkpoint defines weights and architecture, while the runtime chooses how to execute compatible operations. The same Llama checkpoint can run with an eager attention implementation, a framework-dispatched scaled-dot-product kernel, or another supported fused backend. Which path actually runs can differ from the requested configuration if an unsupported dtype, mask, or feature triggers fallback.

A practical verification uses one representative forward pass with the exact serving configuration, then inspects profiler traces and memory use. Check both prefill and decoding because they have different shapes and may use different kernels. Requesting full attention weights for visualization can force materialization that the efficient path was intended to avoid.

Correctness comes first: compare a small example against a reference implementation, including padded batches and causal masking, within suitable floating-point tolerances. Then measure warm-run latency and peak memory. Backend availability changes with library and hardware versions, so record those versions with results. Merely observing that a model loads successfully, or that a configuration contains a backend name, is insufficient evidence that the intended optimized kernel executed.

One common schematic is:

A sequential block instead feeds the attention-updated state into the FFN. Some parallel designs share a normalization; exact details vary. The parallel structure can improve execution scheduling but changes within-block information flow. It is not the same as tensor or pipeline parallelism across GPUs.

In a sequential block, attention first produces , then the FFN operates on to produce the next state. The FFN can react immediately to information collected by that attention layer. In a parallel block, both branches read the original ; the FFN sees the new attention information only in a later block after the branch outputs have been combined.

This removes a dependency between the two branches and can permit scheduling or projection optimizations. It does not guarantee simultaneous execution on the hardware: both branches still compete for compute, memory bandwidth, and kernel resources. The actual benefit depends on the implementation and matrix shapes.

The residual sum requires both branch outputs to match the model width. Normalization may be shared or separate, and residual scaling can affect initialization and training stability. State those choices when describing a concrete model. Do not convert a sequential pretrained checkpoint by rearranging operations and assume equivalence, since the FFN input changes. Also distinguish this within-layer topology from distributing layers or tensor shards across multiple accelerators.