Research workbench

Algorithms.

Interactive notes on graph algorithms, retrieval, matching, and the attention mechanisms I keep studying.

Algorithm concept

Scaled Dot-Product Attention

A step-by-step explanation of the mechanism introduced by Vaswani et al.

Key insight: A contextual vector is a weighted reading of the other Value vectors.
Complexity: O(n²d) per head

ReferenceAttention Is All You Need ↗

Vaswani et al. · 2017 · NeurIPS

Architecture note

Input embeddings

Token vectors give the encoder a starting representation.

Original SVG diagram: a clean-room conceptual reconstruction, not a copy of the paper figure.

\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{\mathsf T}}{\sqrt{d_k}}\right)V

Query token

Attention head

Enable causal masking

Head 1: Local context. These are illustrative educational patterns, not weights from a trained production model.

Attention matrix · softmax weights

Q · K · V calculation

Query learns · Key dimension dₖ = 4

Q[0.42, 0.18, −0.31, 0.57]

Kᵀ[token keys × Q → raw similarities]

Weighted V combination

The 0.09

model 0.24

learns 0.32

which 0.16

words 0.09

matter 0.06

. 0.04

Select a calculation step.

Original implementations · Python / TypeScript

The pseudocode line below follows the active explanation step; complete highlighted code is rendered with Shiki.

1  project embeddings into Q, K, V
2  scores = QKᵀ
3  scores /= √dₖ
4  optionally mask future positions
5  weights = softmax(scores)
6  contextual = weightsV

Python · PyTorch

import math
import torch

def scaled_dot_product_attention(query, key, value, causal_mask=False):
    # query, key, value: [tokens, d_k]
    scores = (query @ key.transpose(-2, -1)) / math.sqrt(key.size(-1))
    if causal_mask:
        future = torch.triu(torch.ones_like(scores, dtype=torch.bool), diagonal=1)
        scores = scores.masked_fill(future, float("-inf"))
    weights = torch.softmax(scores, dim=-1)
    contextual = weights @ value
    return contextual, weights

# Example projections are supplied by an embedding/projection layer.
contextual, weights = scaled_dot_product_attention(Q, K, V)

TypeScript · typed arrays

type Matrix = Float32Array[];

export function scaledDotProductAttention(
  query: Matrix, key: Matrix, value: Matrix, causalMask = false,
): { contextual: Matrix; weights: Matrix } {
  const tokenCount = query.length;
  const dimension = key[0].length;
  const weights = Array.from({ length: tokenCount }, () => new Float32Array(tokenCount));
  const contextual = Array.from({ length: tokenCount }, () => new Float32Array(value[0].length));
  for (let i = 0; i < tokenCount; i++) {
    let max = -Infinity;
    for (let j = 0; j < tokenCount; j++) {
      let dot = 0; for (let d = 0; d < dimension; d++) dot += query[i][d] * key[j][d];
      weights[i][j] = causalMask && j > i ? -Infinity : dot / Math.sqrt(dimension);
      max = Math.max(max, weights[i][j]);
    }
    let sum = 0; for (let j = 0; j < tokenCount; j++) sum += weights[i][j] = Math.exp(weights[i][j] - max);
    for (let j = 0; j < tokenCount; j++) {
      weights[i][j] /= sum;
      for (let d = 0; d < value[0].length; d++) contextual[i][d] += weights[i][j] * value[j][d];
    }
  }
  return { contextual, weights };
}

Why I keep returning to attention

Context is an engineering material.

Attention keeps connecting my interests in machine learning, computer vision, sequence models, and SunCast: each asks what evidence should matter now, and what context should remain visible. This notebook explores the idea; it does not claim that I invented or trained the Transformer architecture.