# Gwern.net Infrastructure Analysis

Comprehensive breakdown of gwern.net's build system, blog pipeline, AI collaboration model, and annotation-transclusion architecture. Based on source analysis of the infra repo at `~/src/parsing/gwern.net/`.

---

## Blog System (`/blog/`)

### Overview

The `/blog/` is not a traditional blog. It is an **annotation-transclusion system**. Blog posts are annotations stored in the metadata database that get automatically compiled into standalone pages at build time. This eliminates the overhead of creating top-level essays (naming, formatting, stable URLs, summaries) while still making short-form and off-site writings discoverable, linkable, and searchable.

### Pipeline

1. Gwern writes something off-site (LessWrong, HN, Reddit, Twitter) or writes a standalone annotation
2. The writing is saved as an **annotation** in the metadata DB with:
   - A unique ID (e.g. `gwern-2025-bell-crow-moon`)
   - Author field (e.g. `"Gwern, Claude-4-opus, Gemini-2.5-pro, GPT-4o3"`)
   - Abstract containing the full text (600-30,000 chars)
   - Key-value metadata (status, importance, confidence, description, thumbnail, etc.)
3. At compile time, `Blog.hs` runs `filterForAuthoredAnnotations` which finds all annotations where:
   - Author starts with `"Gwern"` (`authorU \`isPrefixOf\` aut`)
   - Abstract length > 600 chars (`lengthMin = 600`)
   - URL is either external or under `/blog/`
   - URL does not end with `/index`
4. For each match, a standalone Markdown page is generated at `blog/YYYY/slug.md` that **transcludes** the annotation
5. Two index pages are also generated:
   - `blog/index.md` -- full reverse-chronological listing with year-grouped sections + collapsible transclusions
   - `blog/newest.md` -- latest 29 entries, intended for homepage transclusion

### Generated Page Structure

Each blog entry page is a thin shell that transcludes the annotation:

```markdown
---
title: 'Some Title'
author: Gwern, Claude-4-opus
description: "Short description here"
created: 2025-05-25
modified: 2025-05-26
status: finished
importance: 0
confidence: log
css-extension: dropcaps-de-zs toc-not
backlink: False
placeholder: True
index: True
---

[Description text](https://original-url.com){.include-annotation .include-strict rel='canonical' data-include-template='annotation-blockquote-not' .include-spinner-not .id-not}

[Return to blog index](/blog/index)
```

The `.include-annotation` class triggers Pandoc/Hakyll to inline the full annotation content at render time. For external URLs, `rel='canonical'` points back to the original. For recursive `/blog/` entries (self-hosted annotations), `rel='canonical'` is omitted.

### Index Page Structure

`blog/index.md` has two sections:

1. **Quick index** -- bullet list of title links grouped by year headers, with MM-DD dates in parentheses
2. **View Full Posts** -- collapsible year containers with annotation transclusions (first entry per year is always expanded via `include-even-when-collapsed`)

### Key Constraints

| Parameter | Value | Purpose |
|-----------|-------|---------|
| `lengthMin` | 600 chars | Minimum abstract length to qualify as a blog post |
| `lengthMax` | 30,000 chars | Warning threshold; suggests splitting into a full essay |
| `titleMax` | 51 chars | Prevents title line-wrapping on mobile |
| ID max length | 47 chars | Set by longest real ID (`gwern-2025-you-could-have-invented-transformers`) |

### ID Format

Blog post IDs follow the pattern `gwern-YYYY/slug` or `gwern-YYYY-slug`. The ID is extracted from the annotation's key-value metadata field `("id", "gwern-2025-drl-scaling")`. Both separator formats (`/` and `-` after year) are accepted and normalized to `blog/YYYY/slug.md` output paths.

### Recursive Blog Posts

A "recursive" entry is an annotation whose URL is itself `/blog/20xx/foo`. The annotation creates its own page. This allows true standalone blog posts that don't reference any external URL.

### Why This Design

Problem: top-level essays on gwern.net require heavy metadata, formatting, naming, and imply long-term maintenance commitment. Meanwhile, some of Gwern's most popular writings are one-off comments on other platforms.

Solution: store everything as annotations (which already accumulate backlinks, embeddings, similar-links, archive mirroring). The `/blog/` system just gives them stable URLs and a browsable index. If an annotation grows large enough, it can be promoted to a full essay.

---

## AI Collaboration Model

### Multi-Author Entries

The blog system supports AI co-authorship natively. The author field is a comma-separated string. As long as the string starts with "Gwern", `filterForAuthoredAnnotations` picks it up. Known examples:

| Entry | Authors | Type |
|-------|---------|------|
| Bell, Crow, Moon: 11 Variations | `Gwern, Claude-4-opus, Gemini-2.5-pro, GPT-4o3` | Poetry |
| Parliament of Rag & Bone | `Gwern, GPT-4.5, GPT-4-o3, GPT-4-o4-mini-high, Claude-3.7` | Poetry (Yeats homage) |
| 30 Questions for Hans Moravec | `Gwern, GPT-5.4 Pro, Claude-4.6-opus` | Interview prep |
| 25 Questions for Alexander Young | Unknown, likely similar | Interview prep |

### Workflow Pattern

The AI collaboration follows a consistent pattern:

1. **Gwern provides creative direction** -- a concept, a prompt, a source text
2. **LLMs generate variations** -- multiple models are used in parallel (Claude, GPT, Gemini)
3. **Gwern curates and edits** -- selects best outputs, cleans up, reorders
4. **LLMs are tested for critical judgment** -- asked to prune weaker entries
5. **Gwern arranges final structure** -- ordering, scansion annotations, presentation
6. **AI generates illustrations** -- Midjourney, DALL-E for accompanying images

This is described in "Towards Better LLM Creative Writing" (`/blog/2025/better-llm-writing`) as "intensive search + personalization" -- treating LLMs as creative collaborators in a generate-rank-select loop rather than as autonomous authors.

### Author Field Convention

- Solo human: `Gwern`
- Human + AI: `Gwern, Claude-4-opus, GPT-4o3` (human always first)
- The author names use the model's marketing name (e.g. `Claude-4-opus` not `claude-opus-4`)

---

## AI Infrastructure: Embedding + Clustering Pipeline

### Overview

Gwern uses AI for three behind-the-scenes organizational tasks: embedding generation, similarity clustering, and cluster labeling. This powers the "Sort By Magic" sections on tag directory pages.

### Phase 1: Embedding Generation

- **Script**: `build/embed.sh`
- **API**: OpenAI `text-embedding-3-large`
- **Input**: article title + author + date + keywords + abstract + backlinks
- **Output**: 512-dimensional vectors stored in `metadata/embeddings.bin`
- **Format**: `(URL, timestamp, plaintext, model-id, [float...])`

### Phase 2: Similarity Clustering

- **Script**: `build/GenerateSimilar.hs`
- **Algorithm**: RP-Tree (random projection forest) for nearest-neighbor queries
- **Process**:
  1. Embed all annotations via OpenAI API
  2. Build RP-tree index from embeddings
  3. Walk through nearest neighbors starting from newest item
  4. Group into clusters when cosine distance exceeds threshold
  5. Cache results in `metadata/listsortedmagic.hs`

**Configuration** (`Config/GenerateSimilar.hs`):

| Setting | Value | Purpose |
|---------|-------|---------|
| `bestNEmbeddings` | 20 | Neighbors to search for |
| `maximumLength` | 32,700 chars | Token limit for embedding API |
| `minimumSuggestions` | 3 | Don't show "Similar Links" below this |
| `iterationLimit` | 6 | Max refinement rounds for kNN |
| `maxDistance` | 0.95 | Cosine distance threshold for clustering |
| `minTagAuto` | 3 | Only show clusters with >= 3 inferred tags |
| `maxTitlesForTagGuessing` | 30 | Max titles fed to tag guesser |

### Phase 3: Cluster Labeling

- **Script**: `build/tagguesser.py`
- **API**: OpenAI `gpt-4o-mini`
- **Input**: parent tag name + blacklist of previous labels + 30 shuffled article titles
- **Process**:
  1. Feed titles to GPT-4o-mini
  2. Request 5 tag suggestions (specific, lowercase, hyphen-separated, URL-safe)
  3. Extract final single suggestion
  4. Cache in `metadata/listname.hs`

### Phase 4: Directory Page Generation

- **Script**: `build/generateDirectory.hs`
- **Output**: HTML sections with "Sort By Magic" header + auto-labeled clusters
- **Display**: labeled sections with collapsed transcluded annotations

### Full Pipeline

```
annotations --> embed.sh (OpenAI API) --> embeddings.bin
embeddings.bin --> GenerateSimilar.hs (RP-tree kNN) --> listsortedmagic.hs
clusters --> tagguesser.py (GPT-4o-mini) --> listname.hs
caches --> generateDirectory.hs --> HTML "Sort By Magic" sections
```

---

## Metadata Schema

### Core Type

```haskell
type MetadataItem = (String, String, String, String, [(String,String)], [String], String)
                  -- (Title, Author, Date, DateCreated, K-V pairs, Tags, Abstract)
```

### Key-Value Metadata Fields

Used in the 5th tuple position for per-annotation configuration:

- `id` -- unique identifier (required for blog posts)
- `description` -- short summary
- `status` -- `finished`, `draft`, etc.
- `importance` -- numeric priority
- `confidence` -- epistemic status (`log`, `likely`, etc.)
- `css-extension` -- page-specific CSS classes
- `thumbnail` -- image path
- `thumbnail-text` -- alt text for thumbnail

### Caching Databases

| File | Content |
|------|---------|
| `metadata/embeddings.bin` | Serialized embedding vectors |
| `metadata/listsortedmagic.hs` | Cached sort-by-magic orderings |
| `metadata/listname.hs` | Cached tag name suggestions |

---

## Author Metadata Processing

`Config/Metadata/Author.hs` (104K+ tokens) handles author string normalization:

- Regex-based cleanup of academic suffixes (PhD, MD, BSc, etc.)
- Initial formatting (`J.Smith` --> `J. Smith`)
- Surname-initial reordering (`Smith, J.` --> `J. Smith`)
- Semicolon-to-comma normalization
- Author linking to Wikipedia for known figures
- Collapsible display for 5+ author lists (show 3, collapse rest)
- Author link blacklisting

---

## Build System Overview

The site compiles via Hakyll (Haskell static site generator) with extensive custom modules:

### Key Build Files

| File | Purpose |
|------|---------|
| `Blog.hs` | Blog annotation-transclusion system |
| `Annotation.hs` | Core annotation handling |
| `GenerateSimilar.hs` | Embedding + clustering engine |
| `generateDirectory.hs` | Directory/tag page builder with Sort-by-Magic |
| `embed.sh` | OpenAI embedding API wrapper |
| `tagguesser.py` | GPT-4o-mini cluster label generator |
| `LinkID.hs` | Annotation ID extraction |
| `LinkMetadata.hs` | Metadata sorting/filtering |
| `LinkMetadataTypes.hs` | Core type definitions |
| `Config/Misc.hs` | Global config (author name, root dir, current year) |
| `Config/Metadata/Author.hs` | Author string normalization (massive) |
| `Config/GenerateSimilar.hs` | Clustering tuning parameters |

### Site Root

All paths are relative to `~/wiki/` on Gwern's system (`Config/Misc.hs` line 21: `root = unsafePerformIO getHomeDirectory ++ "/wiki/"`).

---

## Blog Entry Inventory (as of 2026-05-23)

### 2026
- `/blog/2026/questions-moravec` -- 30 Questions for Hans Moravec (AI co-authored)
- `/blog/2026/questions-young` -- 25 Questions for Alexander Young
- `/blog/2026/rabbit` -- Jacky Rabbit and the Empty Jam Pot (human, fiction)
- `/blog/2026/make-me-care` -- First, Make Me Care

### 2025 (20 entries)
- `/blog/2025/conference-size` -- Conference Fermi Problems
- `/blog/2025/perfume` -- Perfume Reviews
- `/blog/2025/bot-location` -- Tracking Phishing Bots Via Locations
- `/blog/2025/better-llm-writing` -- Towards Better LLM Creative Writing
- `/blog/2025/conference` -- Conferences As D&D Tabletops
- `/blog/2025/pinball-hacking` -- Hacking Pinball High Scores
- `/blog/2025/bell-crow-moon` -- Bell, Crow, Moon: 11 Variations (AI co-authored)
- `/blog/2025/you-could-have-invented-transformers` -- You Could've Invented Transformers
- `/blog/2025/human-cannibalism` -- Human Cannibalism Alignment Chart
- `/blog/2025/meta-doomsday` -- The Meta-LW Doomsday Argument
- `/blog/2025/ai-cannibalism` -- AI Cannibalism Can Be Good
- `/blog/2025/area-man` -- Area Man Angry AI Not Silver Bullet
- `/blog/2025/parliament` -- Parliament of Rag & Bone (AI co-authored)
- `/blog/2025/good-ai-samples` -- Adding Bits Beats AI Slop
- `/blog/2025/sake` -- Review: The Birth of Sake
- `/blog/2026/questions-henrich` -- 10 Questions for Joseph Henrich
- `/blog/2025/large-files` -- Gwern.net Large File Support
- `/blog/2025/llms-can-be-faster` -- LLMs Can Be Faster Than You Think
- `/blog/2025/plumbing-vs-internet` -- Plumbing vs Internet, Revisited

### 2024 (26 entries)
- `/blog/2024/trajectoid-words`, `/blog/2024/tlon`, `/blog/2024/virtual-comments`, `/blog/2024/winning-arms-races`, `/blog/2024/china-dl`, `/blog/2024/diminishing-returns`, `/blog/2024/program-reliability`, `/blog/2024/e-positive`, `/blog/2024/pemmican`, `/blog/2024/llm-acceptable-use-policy`, `/blog/2024/tail-collapse`, `/blog/2024/hardware-hedging`, `/blog/2024/writing-online`, `/blog/2024/marvel`, `/blog/2024/sydney`, `/blog/2024/rss`, `/blog/2024/gamblers-verity`, `/blog/2024/personalization`, `/blog/2024/regexp-ai`, `/blog/2024/openai-mojo`, `/blog/2024/semantic-derealization`, `/blog/2024/tools-for-thought-failure`, `/blog/2024/sick-kids`, `/blog/2024/tolkien`, `/blog/2024/multiuser-wiki`, `/blog/2024/marx-brothers`

### 2023
- `/blog/2023/good-writing`, `/blog/2023/mode-collapse`, `/blog/2023/littles-law`, `/blog/2023/subtle-poisons`

### 2022
- `/blog/2022/active-learning`, `/blog/2022/cleanup-when`, `/blog/2022/pdf-forgery`, `/blog/2022/fastest-human`

### 2021
- `/blog/2021/dream-morality`, `/blog/2021/cat-tail`

### 2020
- `/blog/2020/mlscaling`, `/blog/2020/smallcaps-filter`, `/blog/2020/gpt2-poetry-collaboration`

### 2019
- `/blog/2019/twdne-website`

### 2018
- `/blog/2018/brain-imitation-learning`

### 2015
- `/blog/2015/dmt-factoring`

### 2013
- `/blog/2013/weirdness-points`

### 2011
- `/blog/2011/i-zombies`

---

## Extractable Ideas for krisyotam.com / srice

### Annotation-Transclusion Blog
- Store short-form and off-site writings as annotations in content.db
- Generate lightweight blog pages that transclude annotation content
- Automatic index generation (full + recent subset for homepage)
- Zero-friction publishing: add annotation with ID, page appears at next build

### AI Co-Authorship Metadata
- Author field already supports comma-separated names
- Credit AI models by marketing name when they contribute meaningfully
- Filter by primary author (human) to auto-collect blog posts

### Embedding-Based Related Content
- Embed all content.db entries via OpenAI API
- Build similarity index for "related posts" sections
- RP-tree or similar ANN structure for fast lookups

### Auto-Clustering for Tag/Index Pages
- Group content by embedding similarity instead of manual tagging alone
- Use cheap model (gpt-4o-mini) for cluster label generation
- "Sort By Magic" alternative browsing mode

### Recursive Self-Hosting
- Allow blog annotations to reference their own URL
- Enables standalone posts without external source URLs