Wes Kennedy

I have ADHD.

I’ve spent a lot of time in the last decade+ maintaining a website/blog.

graph LR A[Input Text]:::hl --> B[Tokenizer]:::hl --> C[Token IDs]:::hl --> D[Embedding]:::hl --> E["Layers ×80"]:::hl --> F[Predict]:::hl --> G[Output Token]:::hl G -.

graph LR A["Layers ×80"] --> B["Predict next token"]:::hl --> C{"Is it EOS?"}

graph TD A["Token vector (8,192-d)"] --> S["Split into 64 heads"]:::hl S --> H1["Head 1 (128-d) Q₁·K₁ → scores → ΣV₁"]:::hl S --> H2["Head 2 (128-d) Q₂·K₂ → scores → ΣV₂"]:::hl S --> H3["..."]

graph TD subgraph MHA["Multi-Head Attention (MHA)"] direction LR Q1a["Q₁"] --- K1a["K₁"] Q2a["Q₂"] --- K2a["K₂"] Q3a["Q₃"] --- K3a["K₃"] Q4a["Q₄"] --- K4a["K₄"] end subgraph GQA["Grouped-Query Attention (GQA)"] direction LR Q1b["Q₁"] --- K1b["K₁"] Q2b["Q₂…

graph LR A["Active tokens (recent)"]:::s1 --> HBM["GPU HBM (192 GB, ~8 TB/s)"]:::s1 B["Warm tokens (medium-range)"]:::s2 --> CPU["CPU RAM (1-2 TB, ~200 GB/s)"]:::s2 C["Cold tokens (distant past)"]:::s3 --> NVM["NVMe / Storage (many TB, ~10-50 GB/s)"]:::s3…

graph LR A["Request 1 KV Cache"]:::hl --> PM["Page Manager (vLLM)"]:::hl B["Request 2 KV Cache"]:::hl --> PM C["Request 3 KV Cache"]:::hl --> PM PM --> P1["Page 1"]:::hl PM --> P2["Page 2"]:::hl PM --> P3["Page 3"]:::hl PM --> P4["..."]

graph LR A[Input Text] --> B[Tokenizer] --> C[Token IDs] --> D["Embedding (quantizable)"]:::hl --> E["Q,K,V Proj (quantizable)"]:::hl --> F["Attn Scores"] --> G["FFN (quantizable)"]:::hl --> H["→ ×80"] --> I["Vocab Proj (quantizable)"]:::hl --> J[Output] E

graph LR A["Token vectors"] --> B["Q,K,V Proj"] --> C["Approximate attention"]:::hl --> D["Output"] classDef hl fill:#2d6a4f,stroke:#1b4332,color:#d8f3dc classDef default fill:#1a1a2e,stroke:#16213e,color:#e0e0e0 click A "/series/inference/what-happens/to…

graph LR A["Token vectors"] --> B["Q,K,V Proj"] --> C["Attention Scores (T² → sparse)"]:::hl --> D["Softmax"]:::hl --> E["Weighted Sum (fewer tokens)"]:::hl --> F["Output"] classDef hl fill:#2d6a4f,stroke:#1b4332,color:#d8f3dc classDef default fill:#1a1a2…

graph LR A["Final hidden state (8,192-d)"]:::hl --> B["× Vocab matrix (128K dot products)"]:::hl --> C["Logits (128K scores)"]:::hl --> D["Softmax"]:::hl --> E["Probabilities"]:::hl --> F["Sample (temp, top-p)"]:::hl --> G["Output Token"]:::hl classDef hl…

graph LR A["Post-attention vector (8,192-d)"] --> B["× W_up (8,192 → 32,768)"]:::hl --> C["SiLU activation (element-wise)"]:::hl --> D["× W_down (32,768 → 8,192)"]:::hl --> E["+ input (residual)"]:::hl --> F["Output vector (8,192-d)"] classDef hl fill:#2d…

graph LR A["Token vectors"] --> B["× W_Q → Query"]:::hl A --> C["× W_K → Key"]:::hl A --> D["× W_V → Value"]:::hl B --> E["Q · K^T (T² dot products)"]:::hl C --> E E --> F["Softmax (normalize)"]:::hl F --> G["× Values (weighted sum)"]:::hl D --> G G --> H…

graph LR A[Token IDs] --> D["Embedding 128K × D"]:::hl --> E["Q,K,V Proj D × D"]:::hl --> F["Attn Scores T² × D"] --> G["FFN W_up D × 4D"]:::hl --> H["FFN W_down 4D × D"]:::hl --> I["→ ×80"] --> J["Vocab Proj D × 128K"]:::hl --> K[Output] classDef hl fill…

graph LR A["Input Text"]:::hl --> B["Normalize (CPU)"]:::hl --> C["BPE Split (CPU, memory-bound)"]:::hl --> D["Token IDs"]:::hl --> E[Embedding] --> F["Layers ×80"] --> G[Predict] --> H[Output] classDef hl fill:#2d6a4f,stroke:#1b4332,color:#d8f3dc classDef

graph LR A["Complex task"] --> B["Model generates plan (tokens)"]:::hl B --> C["Step 1: tool call"]:::hl --> D["Result 1"] D --> E["Step 2: tool call"]:::hl --> F["Result 2"] F --> G["Step 3: generate response"]:::hl --> H["Output"] classDef hl fill:#2d6a…

graph LR A["User: /skill-name"] --> B["Harness expands skill → prompt"]:::hl --> C["Expanded prompt + tool definitions"]:::hl --> D["Model generates (may call tools)"] --> E["Output"] classDef hl fill:#2d6a4f,stroke:#1b4332,color:#d8f3dc classDef default…

graph LR A[Input Text] --> B[Tokenizer] --> C[Token IDs] --> D[Embedding] --> E[Q,K,V Proj] --> F[Attn Scores] E -->|"K,V cached"| KV["KV Cache (HBM)"]:::hl KV -->|"read during decode"| F:::hl F --> G[Weighted Sum] --> H[Residual] --> I[FFN] --> J[Residua…

graph LR A[Input Text] --> B[Tokenizer] --> C[Token IDs] --> D[Embedding] --> E[Q,K,V Proj] --> F["Attn Scores (Q·K dot prod)"]:::hl --> G[Weighted Sum] --> H[Residual] --> I[FFN] --> J[Residual] --> K["→ ×80"] --> L["Vocab Proj (dot prod)"]:::hl --> M[Ou…

graph LR A[Input Text] --> B[Tokenizer] --> C[Token IDs] --> D["Embedding → embedding"]:::s1 --> E["Layer 1"]:::s2 --> F["Layer 2...79"]:::s2 --> G["Layer 80"]:::s2 --> H["Final hidden state"]:::s3 --> I[Vocab Proj] --> J[Output] classDef s1 fill:#264653,…

graph LR A[Input Text] --> B[Tokenizer] --> C[Token IDs] --> D[Embedding] --> E["Q,K,V Proj (matmul)"]:::hl --> F["Attn Scores (matmul)"]:::hl --> G[Weighted Sum] --> H[Residual]:::hl --> I["FFN W_up (matmul)"]:::hl --> J["Activation (nonlinear)"]:::hl -->

graph LR A[Input Text] --> B[Tokenizer] --> C[Token IDs] --> D[Embedding] --> E[Q,K,V Proj]:::hl --> F[Attn Scores]:::hl --> G[Weighted Sum]:::hl --> H[Residual]:::hl --> I["FFN"]:::hl --> J[Residual]:::hl --> K["→ ×80"]:::hl --> L[Vocab Proj] --> M[Outpu…

graph LR A[Input Text] --> B[Tokenizer] --> C[Token IDs] --> D[Embedding]:::hl --> E[Attention] --> F[FFN] --> G["→ ×80"] --> H[Vocab Proj] --> I[Output] classDef hl fill:#2d6a4f,stroke:#1b4332,color:#d8f3dc classDef default fill:#1a1a2e,stroke:#16213e,co…

graph LR A[Training Text]:::hl --> B[Forward Pass]:::hl --> C[Prediction]:::hl --> D[Loss Calc]:::hl --> E[Backward Pass]:::hl --> F[Gradients]:::hl --> G[Weight Update]:::hl --> H["Repeat ×trillions"]:::hl classDef hl fill:#2d6a4f,stroke:#1b4332,color:#d…

graph LR A[Input Text] --> B[Tokenizer] --> C[Token IDs] --> D[Embedding]:::hl --> E[Q,K,V Proj]:::hl --> F[Attn Scores] --> G[Weighted Sum] --> H["FFN: W_up"]:::hl --> I[Activation] --> J["FFN: W_down"]:::hl --> K["→ ×80"] --> L[Vocab Proj]:::hl --> M[Ou…

graph LR A[Input Text]:::hl --> B[Normalize]:::hl --> C[BPE Split]:::hl --> D[Token IDs]:::hl --> E[Embedding] --> F[Attention] --> G[FFN] --> H["→ ×80"] --> I[Vocab Proj] --> J[Output] classDef hl fill:#2d6a4f,stroke:#1b4332,color:#d8f3dc classDef default

graph LR A["User message"] --> B["Harness checks memory store"]:::hl B --> C["Retrieve relevant memories"]:::hl C --> D["Inject into context"]:::hl D --> E["Model sees memories as tokens"] --> F["Layers ×80"] --> G["Response"] G --> H{"Save new memories?"}

graph LR A[User message] --> B["Model generates tokens"] B --> C{"Tool call token pattern?"}

graph LR A[User message] --> B[Tokenize + Embed] --> C["Layers ×80"] C --> D{"Generate thinking tokens"}:::hl D -->|"loop (hidden)"| C D -->|"done thinking"| E{"Generate response tokens"}:::hl E -->|"loop (visible)"| C E --> F[Output to user] classDef hl…

graph LR A[Input Text] --> B[Tokenizer] --> C[Token IDs] --> D[Embedding] --> E["Layers ×80"] E --> F[Predict] --> G[Output Token] G -.

graph LR A[Input Text] --> B[Tokenizer] --> C[Token IDs] --> D[Embedding]:::hl --> E["Layers ×80"] --> F[Predict] --> G[Output Token] G -.

graph LR A[Input Text]:::hl --> B[Tokenizer]:::hl --> C[Token IDs]:::hl --> D[Embedding] --> E["Layers ×80"] --> F[Predict] --> G[Output Token] G -.

graph LR A[Input Text] --> B[Tokenizer] --> C[Token IDs] --> D[Embedding]:::hl --> E["Layers ×80"]:::hl --> F[Predict]:::hl --> G[Output Token] G -.