Comprehensive breakdown of gwern.net's build system, blog pipeline, AI collaboration model, and annotation-transclusion architecture. Based on source analysis of the infra repo at ~/src/parsing/gwern.net/.
/blog/)The /blog/ is not a traditional blog. It is an annotation-transclusion system. Blog posts are annotations stored in the metadata database that get automatically compiled into standalone pages at build time. This eliminates the overhead of creating top-level essays (naming, formatting, stable URLs, summaries) while still making short-form and off-site writings discoverable, linkable, and searchable.
gwern-2025-bell-crow-moon)"Gwern, Claude-4-opus, Gemini-2.5-pro, GPT-4o3")Blog.hs runs filterForAuthoredAnnotations which finds all annotations where:
"Gwern" (authorU \isPrefixOf` aut`)lengthMin = 600)/blog//indexblog/YYYY/slug.md that transcludes the annotationblog/index.md -- full reverse-chronological listing with year-grouped sections + collapsible transclusionsblog/newest.md -- latest 29 entries, intended for homepage transclusionEach blog entry page is a thin shell that transcludes the annotation:
---
title: 'Some Title'
author: Gwern, Claude-4-opus
description: "Short description here"
created: 2025-05-25
modified: 2025-05-26
status: finished
importance: 0
confidence: log
css-extension: dropcaps-de-zs toc-not
backlink: False
placeholder: True
index: True
---
[Description text](https://original-url.com){.include-annotation .include-strict rel='canonical' data-include-template='annotation-blockquote-not' .include-spinner-not .id-not}
[Return to blog index](/blog/index)
The .include-annotation class triggers Pandoc/Hakyll to inline the full annotation content at render time. For external URLs, rel='canonical' points back to the original. For recursive /blog/ entries (self-hosted annotations), rel='canonical' is omitted.
blog/index.md has two sections:
include-even-when-collapsed)| Parameter | Value | Purpose |
|---|---|---|
lengthMin |
600 chars | Minimum abstract length to qualify as a blog post |
lengthMax |
30,000 chars | Warning threshold; suggests splitting into a full essay |
titleMax |
51 chars | Prevents title line-wrapping on mobile |
| ID max length | 47 chars | Set by longest real ID (gwern-2025-you-could-have-invented-transformers) |
Blog post IDs follow the pattern gwern-YYYY/slug or gwern-YYYY-slug. The ID is extracted from the annotation's key-value metadata field ("id", "gwern-2025-drl-scaling"). Both separator formats (/ and - after year) are accepted and normalized to blog/YYYY/slug.md output paths.
A "recursive" entry is an annotation whose URL is itself /blog/20xx/foo. The annotation creates its own page. This allows true standalone blog posts that don't reference any external URL.
Problem: top-level essays on gwern.net require heavy metadata, formatting, naming, and imply long-term maintenance commitment. Meanwhile, some of Gwern's most popular writings are one-off comments on other platforms.
Solution: store everything as annotations (which already accumulate backlinks, embeddings, similar-links, archive mirroring). The /blog/ system just gives them stable URLs and a browsable index. If an annotation grows large enough, it can be promoted to a full essay.
The blog system supports AI co-authorship natively. The author field is a comma-separated string. As long as the string starts with "Gwern", filterForAuthoredAnnotations picks it up. Known examples:
| Entry | Authors | Type |
|---|---|---|
| Bell, Crow, Moon: 11 Variations | Gwern, Claude-4-opus, Gemini-2.5-pro, GPT-4o3 |
Poetry |
| Parliament of Rag & Bone | Gwern, GPT-4.5, GPT-4-o3, GPT-4-o4-mini-high, Claude-3.7 |
Poetry (Yeats homage) |
| 30 Questions for Hans Moravec | Gwern, GPT-5.4 Pro, Claude-4.6-opus |
Interview prep |
| 25 Questions for Alexander Young | Unknown, likely similar | Interview prep |
The AI collaboration follows a consistent pattern:
This is described in "Towards Better LLM Creative Writing" (/blog/2025/better-llm-writing) as "intensive search + personalization" -- treating LLMs as creative collaborators in a generate-rank-select loop rather than as autonomous authors.
GwernGwern, Claude-4-opus, GPT-4o3 (human always first)Claude-4-opus not claude-opus-4)Gwern uses AI for three behind-the-scenes organizational tasks: embedding generation, similarity clustering, and cluster labeling. This powers the "Sort By Magic" sections on tag directory pages.
build/embed.shtext-embedding-3-largemetadata/embeddings.bin(URL, timestamp, plaintext, model-id, [float...])build/GenerateSimilar.hsmetadata/listsortedmagic.hsConfiguration (Config/GenerateSimilar.hs):
| Setting | Value | Purpose |
|---|---|---|
bestNEmbeddings |
20 | Neighbors to search for |
maximumLength |
32,700 chars | Token limit for embedding API |
minimumSuggestions |
3 | Don't show "Similar Links" below this |
iterationLimit |
6 | Max refinement rounds for kNN |
maxDistance |
0.95 | Cosine distance threshold for clustering |
minTagAuto |
3 | Only show clusters with >= 3 inferred tags |
maxTitlesForTagGuessing |
30 | Max titles fed to tag guesser |
build/tagguesser.pygpt-4o-minimetadata/listname.hsbuild/generateDirectory.hsannotations --> embed.sh (OpenAI API) --> embeddings.bin
embeddings.bin --> GenerateSimilar.hs (RP-tree kNN) --> listsortedmagic.hs
clusters --> tagguesser.py (GPT-4o-mini) --> listname.hs
caches --> generateDirectory.hs --> HTML "Sort By Magic" sections
type MetadataItem = (String, String, String, String, [(String,String)], [String], String)
-- (Title, Author, Date, DateCreated, K-V pairs, Tags, Abstract)
Used in the 5th tuple position for per-annotation configuration:
id -- unique identifier (required for blog posts)description -- short summarystatus -- finished, draft, etc.importance -- numeric priorityconfidence -- epistemic status (log, likely, etc.)css-extension -- page-specific CSS classesthumbnail -- image paththumbnail-text -- alt text for thumbnail| File | Content |
|---|---|
metadata/embeddings.bin |
Serialized embedding vectors |
metadata/listsortedmagic.hs |
Cached sort-by-magic orderings |
metadata/listname.hs |
Cached tag name suggestions |
Config/Metadata/Author.hs (104K+ tokens) handles author string normalization:
J.Smith --> J. Smith)Smith, J. --> J. Smith)The site compiles via Hakyll (Haskell static site generator) with extensive custom modules:
| File | Purpose |
|---|---|
Blog.hs |
Blog annotation-transclusion system |
Annotation.hs |
Core annotation handling |
GenerateSimilar.hs |
Embedding + clustering engine |
generateDirectory.hs |
Directory/tag page builder with Sort-by-Magic |
embed.sh |
OpenAI embedding API wrapper |
tagguesser.py |
GPT-4o-mini cluster label generator |
LinkID.hs |
Annotation ID extraction |
LinkMetadata.hs |
Metadata sorting/filtering |
LinkMetadataTypes.hs |
Core type definitions |
Config/Misc.hs |
Global config (author name, root dir, current year) |
Config/Metadata/Author.hs |
Author string normalization (massive) |
Config/GenerateSimilar.hs |
Clustering tuning parameters |
All paths are relative to ~/wiki/ on Gwern's system (Config/Misc.hs line 21: root = unsafePerformIO getHomeDirectory ++ "/wiki/").
/blog/2026/questions-moravec -- 30 Questions for Hans Moravec (AI co-authored)/blog/2026/questions-young -- 25 Questions for Alexander Young/blog/2026/rabbit -- Jacky Rabbit and the Empty Jam Pot (human, fiction)/blog/2026/make-me-care -- First, Make Me Care/blog/2025/conference-size -- Conference Fermi Problems/blog/2025/perfume -- Perfume Reviews/blog/2025/bot-location -- Tracking Phishing Bots Via Locations/blog/2025/better-llm-writing -- Towards Better LLM Creative Writing/blog/2025/conference -- Conferences As D&D Tabletops/blog/2025/pinball-hacking -- Hacking Pinball High Scores/blog/2025/bell-crow-moon -- Bell, Crow, Moon: 11 Variations (AI co-authored)/blog/2025/you-could-have-invented-transformers -- You Could've Invented Transformers/blog/2025/human-cannibalism -- Human Cannibalism Alignment Chart/blog/2025/meta-doomsday -- The Meta-LW Doomsday Argument/blog/2025/ai-cannibalism -- AI Cannibalism Can Be Good/blog/2025/area-man -- Area Man Angry AI Not Silver Bullet/blog/2025/parliament -- Parliament of Rag & Bone (AI co-authored)/blog/2025/good-ai-samples -- Adding Bits Beats AI Slop/blog/2025/sake -- Review: The Birth of Sake/blog/2026/questions-henrich -- 10 Questions for Joseph Henrich/blog/2025/large-files -- Gwern.net Large File Support/blog/2025/llms-can-be-faster -- LLMs Can Be Faster Than You Think/blog/2025/plumbing-vs-internet -- Plumbing vs Internet, Revisited/blog/2024/trajectoid-words, /blog/2024/tlon, /blog/2024/virtual-comments, /blog/2024/winning-arms-races, /blog/2024/china-dl, /blog/2024/diminishing-returns, /blog/2024/program-reliability, /blog/2024/e-positive, /blog/2024/pemmican, /blog/2024/llm-acceptable-use-policy, /blog/2024/tail-collapse, /blog/2024/hardware-hedging, /blog/2024/writing-online, /blog/2024/marvel, /blog/2024/sydney, /blog/2024/rss, /blog/2024/gamblers-verity, /blog/2024/personalization, /blog/2024/regexp-ai, /blog/2024/openai-mojo, /blog/2024/semantic-derealization, /blog/2024/tools-for-thought-failure, /blog/2024/sick-kids, /blog/2024/tolkien, /blog/2024/multiuser-wiki, /blog/2024/marx-brothers/blog/2023/good-writing, /blog/2023/mode-collapse, /blog/2023/littles-law, /blog/2023/subtle-poisons/blog/2022/active-learning, /blog/2022/cleanup-when, /blog/2022/pdf-forgery, /blog/2022/fastest-human/blog/2021/dream-morality, /blog/2021/cat-tail/blog/2020/mlscaling, /blog/2020/smallcaps-filter, /blog/2020/gpt2-poetry-collaboration/blog/2019/twdne-website/blog/2018/brain-imitation-learning/blog/2015/dmt-factoring/blog/2013/weirdness-points/blog/2011/i-zombies