~kris/dots

srice

srice/doc/gwern.md -rw-r--r-- 14.8 KiB
e98f3b03 — Kris Yotam chore: sync local state after restore (push updates, no pull) a month ago

#Gwern.net Infrastructure Analysis

Comprehensive breakdown of gwern.net's build system, blog pipeline, AI collaboration model, and annotation-transclusion architecture. Based on source analysis of the infra repo at ~/src/parsing/gwern.net/.

#Blog System (/blog/)

#Overview

The /blog/ is not a traditional blog. It is an annotation-transclusion system. Blog posts are annotations stored in the metadata database that get automatically compiled into standalone pages at build time. This eliminates the overhead of creating top-level essays (naming, formatting, stable URLs, summaries) while still making short-form and off-site writings discoverable, linkable, and searchable.

#Pipeline

  1. Gwern writes something off-site (LessWrong, HN, Reddit, Twitter) or writes a standalone annotation
  2. The writing is saved as an annotation in the metadata DB with:
    • A unique ID (e.g. gwern-2025-bell-crow-moon)
    • Author field (e.g. "Gwern, Claude-4-opus, Gemini-2.5-pro, GPT-4o3")
    • Abstract containing the full text (600-30,000 chars)
    • Key-value metadata (status, importance, confidence, description, thumbnail, etc.)
  3. At compile time, Blog.hs runs filterForAuthoredAnnotations which finds all annotations where:
    • Author starts with "Gwern" (authorU \isPrefixOf` aut`)
    • Abstract length > 600 chars (lengthMin = 600)
    • URL is either external or under /blog/
    • URL does not end with /index
  4. For each match, a standalone Markdown page is generated at blog/YYYY/slug.md that transcludes the annotation
  5. Two index pages are also generated:
    • blog/index.md -- full reverse-chronological listing with year-grouped sections + collapsible transclusions
    • blog/newest.md -- latest 29 entries, intended for homepage transclusion

#Generated Page Structure

Each blog entry page is a thin shell that transcludes the annotation:

---
title: 'Some Title'
author: Gwern, Claude-4-opus
description: "Short description here"
created: 2025-05-25
modified: 2025-05-26
status: finished
importance: 0
confidence: log
css-extension: dropcaps-de-zs toc-not
backlink: False
placeholder: True
index: True
---

[Description text](https://original-url.com){.include-annotation .include-strict rel='canonical' data-include-template='annotation-blockquote-not' .include-spinner-not .id-not}

[Return to blog index](/blog/index)

The .include-annotation class triggers Pandoc/Hakyll to inline the full annotation content at render time. For external URLs, rel='canonical' points back to the original. For recursive /blog/ entries (self-hosted annotations), rel='canonical' is omitted.

#Index Page Structure

blog/index.md has two sections:

  1. Quick index -- bullet list of title links grouped by year headers, with MM-DD dates in parentheses
  2. View Full Posts -- collapsible year containers with annotation transclusions (first entry per year is always expanded via include-even-when-collapsed)

#Key Constraints

Parameter Value Purpose
lengthMin 600 chars Minimum abstract length to qualify as a blog post
lengthMax 30,000 chars Warning threshold; suggests splitting into a full essay
titleMax 51 chars Prevents title line-wrapping on mobile
ID max length 47 chars Set by longest real ID (gwern-2025-you-could-have-invented-transformers)

#ID Format

Blog post IDs follow the pattern gwern-YYYY/slug or gwern-YYYY-slug. The ID is extracted from the annotation's key-value metadata field ("id", "gwern-2025-drl-scaling"). Both separator formats (/ and - after year) are accepted and normalized to blog/YYYY/slug.md output paths.

#Recursive Blog Posts

A "recursive" entry is an annotation whose URL is itself /blog/20xx/foo. The annotation creates its own page. This allows true standalone blog posts that don't reference any external URL.

#Why This Design

Problem: top-level essays on gwern.net require heavy metadata, formatting, naming, and imply long-term maintenance commitment. Meanwhile, some of Gwern's most popular writings are one-off comments on other platforms.

Solution: store everything as annotations (which already accumulate backlinks, embeddings, similar-links, archive mirroring). The /blog/ system just gives them stable URLs and a browsable index. If an annotation grows large enough, it can be promoted to a full essay.

#AI Collaboration Model

#Multi-Author Entries

The blog system supports AI co-authorship natively. The author field is a comma-separated string. As long as the string starts with "Gwern", filterForAuthoredAnnotations picks it up. Known examples:

Entry Authors Type
Bell, Crow, Moon: 11 Variations Gwern, Claude-4-opus, Gemini-2.5-pro, GPT-4o3 Poetry
Parliament of Rag & Bone Gwern, GPT-4.5, GPT-4-o3, GPT-4-o4-mini-high, Claude-3.7 Poetry (Yeats homage)
30 Questions for Hans Moravec Gwern, GPT-5.4 Pro, Claude-4.6-opus Interview prep
25 Questions for Alexander Young Unknown, likely similar Interview prep

#Workflow Pattern

The AI collaboration follows a consistent pattern:

  1. Gwern provides creative direction -- a concept, a prompt, a source text
  2. LLMs generate variations -- multiple models are used in parallel (Claude, GPT, Gemini)
  3. Gwern curates and edits -- selects best outputs, cleans up, reorders
  4. LLMs are tested for critical judgment -- asked to prune weaker entries
  5. Gwern arranges final structure -- ordering, scansion annotations, presentation
  6. AI generates illustrations -- Midjourney, DALL-E for accompanying images

This is described in "Towards Better LLM Creative Writing" (/blog/2025/better-llm-writing) as "intensive search + personalization" -- treating LLMs as creative collaborators in a generate-rank-select loop rather than as autonomous authors.

#Author Field Convention

  • Solo human: Gwern
  • Human + AI: Gwern, Claude-4-opus, GPT-4o3 (human always first)
  • The author names use the model's marketing name (e.g. Claude-4-opus not claude-opus-4)

#AI Infrastructure: Embedding + Clustering Pipeline

#Overview

Gwern uses AI for three behind-the-scenes organizational tasks: embedding generation, similarity clustering, and cluster labeling. This powers the "Sort By Magic" sections on tag directory pages.

#Phase 1: Embedding Generation

  • Script: build/embed.sh
  • API: OpenAI text-embedding-3-large
  • Input: article title + author + date + keywords + abstract + backlinks
  • Output: 512-dimensional vectors stored in metadata/embeddings.bin
  • Format: (URL, timestamp, plaintext, model-id, [float...])

#Phase 2: Similarity Clustering

  • Script: build/GenerateSimilar.hs
  • Algorithm: RP-Tree (random projection forest) for nearest-neighbor queries
  • Process:
    1. Embed all annotations via OpenAI API
    2. Build RP-tree index from embeddings
    3. Walk through nearest neighbors starting from newest item
    4. Group into clusters when cosine distance exceeds threshold
    5. Cache results in metadata/listsortedmagic.hs

Configuration (Config/GenerateSimilar.hs):

Setting Value Purpose
bestNEmbeddings 20 Neighbors to search for
maximumLength 32,700 chars Token limit for embedding API
minimumSuggestions 3 Don't show "Similar Links" below this
iterationLimit 6 Max refinement rounds for kNN
maxDistance 0.95 Cosine distance threshold for clustering
minTagAuto 3 Only show clusters with >= 3 inferred tags
maxTitlesForTagGuessing 30 Max titles fed to tag guesser

#Phase 3: Cluster Labeling

  • Script: build/tagguesser.py
  • API: OpenAI gpt-4o-mini
  • Input: parent tag name + blacklist of previous labels + 30 shuffled article titles
  • Process:
    1. Feed titles to GPT-4o-mini
    2. Request 5 tag suggestions (specific, lowercase, hyphen-separated, URL-safe)
    3. Extract final single suggestion
    4. Cache in metadata/listname.hs

#Phase 4: Directory Page Generation

  • Script: build/generateDirectory.hs
  • Output: HTML sections with "Sort By Magic" header + auto-labeled clusters
  • Display: labeled sections with collapsed transcluded annotations

#Full Pipeline

annotations --> embed.sh (OpenAI API) --> embeddings.bin
embeddings.bin --> GenerateSimilar.hs (RP-tree kNN) --> listsortedmagic.hs
clusters --> tagguesser.py (GPT-4o-mini) --> listname.hs
caches --> generateDirectory.hs --> HTML "Sort By Magic" sections

#Metadata Schema

#Core Type

type MetadataItem = (String, String, String, String, [(String,String)], [String], String)
                  -- (Title, Author, Date, DateCreated, K-V pairs, Tags, Abstract)

#Key-Value Metadata Fields

Used in the 5th tuple position for per-annotation configuration:

  • id -- unique identifier (required for blog posts)
  • description -- short summary
  • status -- finished, draft, etc.
  • importance -- numeric priority
  • confidence -- epistemic status (log, likely, etc.)
  • css-extension -- page-specific CSS classes
  • thumbnail -- image path
  • thumbnail-text -- alt text for thumbnail

#Caching Databases

File Content
metadata/embeddings.bin Serialized embedding vectors
metadata/listsortedmagic.hs Cached sort-by-magic orderings
metadata/listname.hs Cached tag name suggestions

#Author Metadata Processing

Config/Metadata/Author.hs (104K+ tokens) handles author string normalization:

  • Regex-based cleanup of academic suffixes (PhD, MD, BSc, etc.)
  • Initial formatting (J.Smith --> J. Smith)
  • Surname-initial reordering (Smith, J. --> J. Smith)
  • Semicolon-to-comma normalization
  • Author linking to Wikipedia for known figures
  • Collapsible display for 5+ author lists (show 3, collapse rest)
  • Author link blacklisting

#Build System Overview

The site compiles via Hakyll (Haskell static site generator) with extensive custom modules:

#Key Build Files

File Purpose
Blog.hs Blog annotation-transclusion system
Annotation.hs Core annotation handling
GenerateSimilar.hs Embedding + clustering engine
generateDirectory.hs Directory/tag page builder with Sort-by-Magic
embed.sh OpenAI embedding API wrapper
tagguesser.py GPT-4o-mini cluster label generator
LinkID.hs Annotation ID extraction
LinkMetadata.hs Metadata sorting/filtering
LinkMetadataTypes.hs Core type definitions
Config/Misc.hs Global config (author name, root dir, current year)
Config/Metadata/Author.hs Author string normalization (massive)
Config/GenerateSimilar.hs Clustering tuning parameters

#Site Root

All paths are relative to ~/wiki/ on Gwern's system (Config/Misc.hs line 21: root = unsafePerformIO getHomeDirectory ++ "/wiki/").

#Blog Entry Inventory (as of 2026-05-23)

#2026

  • /blog/2026/questions-moravec -- 30 Questions for Hans Moravec (AI co-authored)
  • /blog/2026/questions-young -- 25 Questions for Alexander Young
  • /blog/2026/rabbit -- Jacky Rabbit and the Empty Jam Pot (human, fiction)
  • /blog/2026/make-me-care -- First, Make Me Care

#2025 (20 entries)

  • /blog/2025/conference-size -- Conference Fermi Problems
  • /blog/2025/perfume -- Perfume Reviews
  • /blog/2025/bot-location -- Tracking Phishing Bots Via Locations
  • /blog/2025/better-llm-writing -- Towards Better LLM Creative Writing
  • /blog/2025/conference -- Conferences As D&D Tabletops
  • /blog/2025/pinball-hacking -- Hacking Pinball High Scores
  • /blog/2025/bell-crow-moon -- Bell, Crow, Moon: 11 Variations (AI co-authored)
  • /blog/2025/you-could-have-invented-transformers -- You Could've Invented Transformers
  • /blog/2025/human-cannibalism -- Human Cannibalism Alignment Chart
  • /blog/2025/meta-doomsday -- The Meta-LW Doomsday Argument
  • /blog/2025/ai-cannibalism -- AI Cannibalism Can Be Good
  • /blog/2025/area-man -- Area Man Angry AI Not Silver Bullet
  • /blog/2025/parliament -- Parliament of Rag & Bone (AI co-authored)
  • /blog/2025/good-ai-samples -- Adding Bits Beats AI Slop
  • /blog/2025/sake -- Review: The Birth of Sake
  • /blog/2026/questions-henrich -- 10 Questions for Joseph Henrich
  • /blog/2025/large-files -- Gwern.net Large File Support
  • /blog/2025/llms-can-be-faster -- LLMs Can Be Faster Than You Think
  • /blog/2025/plumbing-vs-internet -- Plumbing vs Internet, Revisited

#2024 (26 entries)

  • /blog/2024/trajectoid-words, /blog/2024/tlon, /blog/2024/virtual-comments, /blog/2024/winning-arms-races, /blog/2024/china-dl, /blog/2024/diminishing-returns, /blog/2024/program-reliability, /blog/2024/e-positive, /blog/2024/pemmican, /blog/2024/llm-acceptable-use-policy, /blog/2024/tail-collapse, /blog/2024/hardware-hedging, /blog/2024/writing-online, /blog/2024/marvel, /blog/2024/sydney, /blog/2024/rss, /blog/2024/gamblers-verity, /blog/2024/personalization, /blog/2024/regexp-ai, /blog/2024/openai-mojo, /blog/2024/semantic-derealization, /blog/2024/tools-for-thought-failure, /blog/2024/sick-kids, /blog/2024/tolkien, /blog/2024/multiuser-wiki, /blog/2024/marx-brothers

#2023

  • /blog/2023/good-writing, /blog/2023/mode-collapse, /blog/2023/littles-law, /blog/2023/subtle-poisons

#2022

  • /blog/2022/active-learning, /blog/2022/cleanup-when, /blog/2022/pdf-forgery, /blog/2022/fastest-human

#2021

  • /blog/2021/dream-morality, /blog/2021/cat-tail

#2020

  • /blog/2020/mlscaling, /blog/2020/smallcaps-filter, /blog/2020/gpt2-poetry-collaboration

#2019

  • /blog/2019/twdne-website

#2018

  • /blog/2018/brain-imitation-learning

#2015

  • /blog/2015/dmt-factoring

#2013

  • /blog/2013/weirdness-points

#2011

  • /blog/2011/i-zombies

#Extractable Ideas for krisyotam.com / srice

#Annotation-Transclusion Blog

  • Store short-form and off-site writings as annotations in content.db
  • Generate lightweight blog pages that transclude annotation content
  • Automatic index generation (full + recent subset for homepage)
  • Zero-friction publishing: add annotation with ID, page appears at next build

#AI Co-Authorship Metadata

  • Author field already supports comma-separated names
  • Credit AI models by marketing name when they contribute meaningfully
  • Filter by primary author (human) to auto-collect blog posts

#Embedding-Based Related Content

  • Embed all content.db entries via OpenAI API
  • Build similarity index for "related posts" sections
  • RP-tree or similar ANN structure for fast lookups

#Auto-Clustering for Tag/Index Pages

  • Group content by embedding similarity instead of manual tagging alone
  • Use cheap model (gpt-4o-mini) for cluster label generation
  • "Sort By Magic" alternative browsing mode

#Recursive Self-Hosting

  • Allow blog annotations to reference their own URL
  • Enables standalone posts without external source URLs