From 19c2efc15c0d8fcf4e6dea73c7f24f1149099116 Mon Sep 17 00:00:00 2001
From: Kris Yotam <75515498+krisyotam@users.noreply.github.com>
Date: Thu, 14 May 2026 16:25:18 -0500
Subject: [PATCH] convert README.md to README.txt

---
 README.md  | 229 -----------------------------------------------------
 README.txt | 228 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 228 insertions(+), 229 deletions(-)
 delete mode 100644 README.md
 create mode 100644 README.txt

diff --git a/README.md b/README.md
deleted file mode 100644
index 49089cc90ed3e010bf8c178d4c7a63887354c97c..0000000000000000000000000000000000000000
--- a/README.md
+++ /dev/null
@@ -1,229 +0,0 @@
-# sbot
-
-**Simple Archiver Bot** -- a suckless web archiver written in C.
-
-sbot creates self-contained archives of web pages and entire websites.
-Every resource -- CSS, images, fonts, scripts -- is fetched and inlined
-directly into the HTML as base64 data URIs. The result is a single file
-(or directory of files) that renders perfectly offline, with no external
-dependencies, forever.
-
-## Why
-
-Web pages disappear. Link rot is real. The average web page has a
-half-life of about two years. Bookmarks break, articles vanish,
-references evaporate.
-
-sbot solves this by creating archives that are:
-
-- **Self-contained.** Everything is inlined. No external requests needed.
-- **Human-readable.** Output is standard HTML. Open it in any browser.
-- **Permanent.** No database, no server, no special viewer. Just files.
-- **Metadata-rich.** GWTAR headers record provenance, date, and source.
-
-## Modes
-
-### Single Page Archive
-
-```sh
-sbot https://example.com/article
-```
-
-Archives a single page in **GWTAR format** (Gwern Web Tar Archive). This
-is the default mode and the most common use case. The output is one
-`.gwtar.html` file containing:
-
-- A GWTAR metadata header (HTML comment) with title, source URL, domain,
-  author, archive date, and generator version
-- The full HTML with all CSS stylesheets inlined as `<style>` blocks
-- All images, fonts, and media encoded as `data:` URIs
-- A completely self-contained document that renders identically to the
-  original
-
-GWTAR format is ideal for:
-
-- Archiving individual articles, blog posts, and essays
-- Preserving references and citations
-- Building a personal web archive / digital library
-- Saving pages before they disappear behind paywalls or get deleted
-
-### Whole Site Archive
-
-```sh
-sbot -r https://example.com
-```
-
-Recursively crawls an entire website and archives every page. The output
-is a directory tree that mirrors the site structure, with each page saved
-as a self-contained HTML file. Internal links are rewritten to relative
-paths so navigation works offline.
-
-Features:
-
-- **BFS crawl order.** Breadth-first traversal ensures important pages
-  (closer to root) are archived first.
-- **Same-domain only.** Never follows links to external sites.
-- **robots.txt compliance.** Respects Disallow rules and Crawl-delay
-  directives by default. Override with `-R`.
-- **Depth control.** Set maximum crawl depth with `-d` to limit scope.
-- **Rate limiting.** Configurable delay between requests to be polite to
-  servers (default: 1 second).
-- **Progress reporting.** Periodic status lines showing pages archived,
-  queue depth, and elapsed time.
-- **Graceful degradation.** Failed resources are skipped; the crawl
-  continues.
-
-This mode is ideal for:
-
-- Archiving entire blogs or documentation sites
-- Creating offline mirrors of reference material
-- Preserving small-to-medium websites wholesale
-- Building browseable offline copies of sites you depend on
-
-## Usage
-
-```
-usage: sbot [-vrR] [-d depth] [-o dir] [-a author] url
-
-  -v          verbose output
-  -r          recursive (crawl entire site)
-  -R          ignore robots.txt
-  -d depth    max crawl depth (default: 5)
-  -o dir      output directory
-  -a author   site author name
-```
-
-### Examples
-
-```sh
-# Archive a single article
-sbot https://example.com/blog/post
-
-# Archive with author metadata
-sbot -a "John Doe" https://example.com/article
-
-# Crawl a blog, max depth 3
-sbot -r -d 3 https://blog.example.com
-
-# Verbose crawl to custom directory
-sbot -v -r -o ./my-archive https://docs.example.com
-
-# Crawl ignoring robots.txt restrictions
-sbot -r -R https://example.com
-```
-
-## GWTAR Format
-
-Every archived page includes a GWTAR (Gwern Web Tar Archive) metadata
-header as an HTML comment at the top of the file:
-
-```
-<!--
-================================================================
-  GWTAR ARCHIVE
-================================================================
-
-  Title:        Example Article
-  Source URL:   https://example.com/article
-  Domain:       example.com
-  Author:       John Doe
-
-  Archived by:  Kris Yotam
-  Archived on:  krisyotam.com
-  Archive date: 2026-02-14
-
-  Generator:    sbot/0.3.0
-  Format:       GWTAR (Gwern Web Tar Archive)
-
-================================================================
--->
-```
-
-This header provides full provenance tracking: what was archived, where
-it came from, who archived it, and when.
-
-## Resource Inlining
-
-sbot inlines all resources to create truly self-contained archives:
-
-| Resource Type | Inlining Method |
-|---------------|-----------------|
-| CSS stylesheets | Fetched and inserted as `<style>` blocks |
-| Images | Base64-encoded as `data:image/*` URIs |
-| Fonts | Base64-encoded as `data:font/*` URIs |
-| Other media | Base64-encoded with appropriate MIME type |
-
-Resources that fail to fetch are silently skipped -- the archive
-degrades gracefully rather than failing entirely.
-
-## Build
-
-Requires `libcurl` development headers.
-
-```sh
-# Arch Linux
-sudo pacman -S curl
-
-# Debian/Ubuntu
-sudo apt install libcurl4-openssl-dev
-
-# Build
-make
-
-# Install to /usr/local/bin
-sudo make install
-
-# Clean
-make clean
-```
-
-## Configuration
-
-All configuration is compile-time via `config.h`:
-
-| Setting | Default | Description |
-|---------|---------|-------------|
-| `USER_AGENT` | `sbot/0.3` | HTTP User-Agent string |
-| `CONNECT_TIMEOUT` | 30s | Connection timeout |
-| `REQUEST_TIMEOUT` | 60s | Total request timeout |
-| `MAX_REDIRECTS` | 10 | Maximum HTTP redirects to follow |
-| `MAX_DEPTH` | 5 | Default recursive crawl depth |
-| `RATE_LIMIT_MS` | 1000ms | Delay between requests |
-| `MAX_FILE_SIZE` | 50 MB | Maximum size per resource |
-| `OUTPUT_EXT` | `.gwtar.html` | File extension for archives |
-
-Edit `config.h` and rebuild to change any setting. This is the suckless
-way -- no runtime configuration files, no environment variables, no
-hidden defaults.
-
-## Architecture
-
-```
-archiver.c   Main entry, page archiving, CSS inlining, link rewriting
-crawl.c      URL queue (BFS), visited set, URL normalization
-fetch.c      HTTP fetching via libcurl
-parse.c      HTML parsing, resource extraction, image inlining
-robots.c     robots.txt fetching, parsing, rule matching
-util.c       Memory wrappers, string ops, base64, MIME types
-config.h     Compile-time constants
-```
-
-Single external dependency: libcurl. No XML parsers, no HTML5 parsers,
-no JavaScript engines. The HTML parsing is deliberately simple --
-regex-based extraction of `src`, `href`, and `url()` references. This
-handles the vast majority of real-world pages and keeps the codebase
-small and auditable.
-
-## Philosophy
-
-sbot follows the [suckless](https://suckless.org) philosophy:
-
-- Written in C99 with POSIX.1-2008
-- Minimal dependencies (libcurl only)
-- Configuration through `config.h` (edit and recompile)
-- Small, readable codebase
-- Does one thing well
-
-## License
-
-MIT/X Consortium License. See [LICENSE](LICENSE) for details.
diff --git a/README.txt b/README.txt
new file mode 100644
index 0000000000000000000000000000000000000000..766fac9d226ac19f98479c7b77fee582310d21b6
--- /dev/null
+++ b/README.txt
@@ -0,0 +1,228 @@
+sbot
+
+
+Simple Archiver Bot -- a suckless web archiver written in C.
+
+sbot creates self-contained archives of web pages and entire websites.
+Every resource -- CSS, images, fonts, scripts -- is fetched and inlined
+directly into the HTML as base64 data URIs. The result is a single file
+(or directory of files) that renders perfectly offline, with no external
+dependencies, forever.
+
+
+WHY
+
+  Web pages disappear. Link rot is real. The average web page has a
+  half-life of about two years. Bookmarks break, articles vanish,
+  references evaporate.
+
+  sbot solves this by creating archives that are:
+
+  - Self-contained. Everything is inlined. No external requests needed.
+  - Human-readable. Output is standard HTML. Open it in any browser.
+  - Permanent. No database, no server, no special viewer. Just files.
+  - Metadata-rich. GWTAR headers record provenance, date, and source.
+
+
+MODES
+
+  SINGLE PAGE ARCHIVE
+
+    sbot https://example.com/article
+
+    Archives a single page in GWTAR format (Gwern Web Tar Archive). This
+    is the default mode and the most common use case. The output is one
+    .gwtar.html file containing:
+
+    - A GWTAR metadata header (HTML comment) with title, source URL,
+      domain, author, archive date, and generator version
+    - The full HTML with all CSS stylesheets inlined as style blocks
+    - All images, fonts, and media encoded as data: URIs
+    - A completely self-contained document that renders identically to
+      the original
+
+    GWTAR format is ideal for:
+
+    - Archiving individual articles, blog posts, and essays
+    - Preserving references and citations
+    - Building a personal web archive / digital library
+    - Saving pages before they disappear behind paywalls or get deleted
+
+
+  WHOLE SITE ARCHIVE
+
+    sbot -r https://example.com
+
+    Recursively crawls an entire website and archives every page. The
+    output is a directory tree that mirrors the site structure, with each
+    page saved as a self-contained HTML file. Internal links are rewritten
+    to relative paths so navigation works offline.
+
+    Features:
+
+    - BFS crawl order. Breadth-first traversal ensures important pages
+      (closer to root) are archived first.
+    - Same-domain only. Never follows links to external sites.
+    - robots.txt compliance. Respects Disallow rules and Crawl-delay
+      directives by default. Override with -R.
+    - Depth control. Set maximum crawl depth with -d to limit scope.
+    - Rate limiting. Configurable delay between requests to be polite to
+      servers (default: 1 second).
+    - Progress reporting. Periodic status lines showing pages archived,
+      queue depth, and elapsed time.
+    - Graceful degradation. Failed resources are skipped; the crawl
+      continues.
+
+    This mode is ideal for:
+
+    - Archiving entire blogs or documentation sites
+    - Creating offline mirrors of reference material
+    - Preserving small-to-medium websites wholesale
+    - Building browseable offline copies of sites you depend on
+
+
+USAGE
+
+  usage: sbot [-vrR] [-d depth] [-o dir] [-a author] url
+
+    -v          verbose output
+    -r          recursive (crawl entire site)
+    -R          ignore robots.txt
+    -d depth    max crawl depth (default: 5)
+    -o dir      output directory
+    -a author   site author name
+
+
+  EXAMPLES
+
+    Archive a single article:
+      sbot https://example.com/blog/post
+
+    Archive with author metadata:
+      sbot -a "John Doe" https://example.com/article
+
+    Crawl a blog, max depth 3:
+      sbot -r -d 3 https://blog.example.com
+
+    Verbose crawl to custom directory:
+      sbot -v -r -o ./my-archive https://docs.example.com
+
+    Crawl ignoring robots.txt restrictions:
+      sbot -r -R https://example.com
+
+
+GWTAR FORMAT
+
+  Every archived page includes a GWTAR (Gwern Web Tar Archive) metadata
+  header as an HTML comment at the top of the file:
+
+    <!--
+    ================================================================
+      GWTAR ARCHIVE
+    ================================================================
+
+      Title:        Example Article
+      Source URL:   https://example.com/article
+      Domain:       example.com
+      Author:       John Doe
+
+      Archived by:  Kris Yotam
+      Archived on:  krisyotam.com
+      Archive date: 2026-02-14
+
+      Generator:    sbot/0.3.0
+      Format:       GWTAR (Gwern Web Tar Archive)
+
+    ================================================================
+    -->
+
+  This header provides full provenance tracking: what was archived,
+  where it came from, who archived it, and when.
+
+
+RESOURCE INLINING
+
+  sbot inlines all resources to create truly self-contained archives:
+
+  Resource Type       Inlining Method
+  ------------------  ------------------------------------------------
+  CSS stylesheets     Fetched and inserted as style blocks
+  Images              Base64-encoded as data:image/* URIs
+  Fonts               Base64-encoded as data:font/* URIs
+  Other media         Base64-encoded with appropriate MIME type
+
+  Resources that fail to fetch are silently skipped -- the archive
+  degrades gracefully rather than failing entirely.
+
+
+BUILD
+
+  Requires libcurl development headers.
+
+  Arch Linux:
+    sudo pacman -S curl
+
+  Debian/Ubuntu:
+    sudo apt install libcurl4-openssl-dev
+
+  Build:
+    make
+
+  Install to /usr/local/bin:
+    sudo make install
+
+  Clean:
+    make clean
+
+
+CONFIGURATION
+
+  All configuration is compile-time via config.h:
+
+  Setting            Default       Description
+  -----------------  ------------  -----------------------------------
+  USER_AGENT         sbot/0.3      HTTP User-Agent string
+  CONNECT_TIMEOUT    30s           Connection timeout
+  REQUEST_TIMEOUT    60s           Total request timeout
+  MAX_REDIRECTS      10            Maximum HTTP redirects to follow
+  MAX_DEPTH          5             Default recursive crawl depth
+  RATE_LIMIT_MS      1000ms        Delay between requests
+  MAX_FILE_SIZE      50 MB         Maximum size per resource
+  OUTPUT_EXT         .gwtar.html   File extension for archives
+
+  Edit config.h and rebuild to change any setting. This is the suckless
+  way -- no runtime configuration files, no environment variables, no
+  hidden defaults.
+
+
+ARCHITECTURE
+
+  archiver.c   Main entry, page archiving, CSS inlining, link rewriting
+  crawl.c      URL queue (BFS), visited set, URL normalization
+  fetch.c      HTTP fetching via libcurl
+  parse.c      HTML parsing, resource extraction, image inlining
+  robots.c     robots.txt fetching, parsing, rule matching
+  util.c       Memory wrappers, string ops, base64, MIME types
+  config.h     Compile-time constants
+
+  Single external dependency: libcurl. No XML parsers, no HTML5 parsers,
+  no JavaScript engines. The HTML parsing is deliberately simple --
+  regex-based extraction of src, href, and url() references. This handles
+  the vast majority of real-world pages and keeps the codebase small and
+  auditable.
+
+
+PHILOSOPHY
+
+  sbot follows the suckless philosophy:
+
+  - Written in C99 with POSIX.1-2008
+  - Minimal dependencies (libcurl only)
+  - Configuration through config.h (edit and recompile)
+  - Small, readable codebase
+  - Does one thing well
+
+
+LICENSE
+
+  MIT/X Consortium License. See LICENSE for details.