<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Large File Storage | Kris Yotam</title>
<link rel="stylesheet" href="../../c/site.css">
<meta name="description" content="Implementation details of large file storage solutions for the website">
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
<div class="mode-container">
<input type="radio" name="theme" id="theme-auto" checked>
<label for="theme-auto" class="auto" title="Auto theme"></label>
<input type="radio" name="theme" id="theme-light">
<label for="theme-light" class="light" title="Light theme"></label>
<input type="radio" name="theme" id="theme-dark">
<label for="theme-dark" class="dark" title="Dark theme"></label>
</div>
<div class="top-bar" role="navigation">
<a href="../../index.html" title="Main page"><img src="../../c/sigma.svg" alt="Main page"></a>
<a href="../../about.html" title="About"><img src="../../c/news.svg" alt="About"></a>
<a href="#" title="Code"><img src="../../c/code.svg" alt="Code"></a>
<a href="../../pics.html" title="Photography"><img src="../../c/camera.svg" alt="Photography"></a>
<a href="#" title="Pictures"><img src="../../c/picture.svg" alt="Pictures"></a>
<a href="../../vids.html" title="Videos"><img src="../../c/video.svg" alt="Videos"></a>
<a href="#" title="Donations"><img src="../../c/beer.svg" alt="Donations"></a>
<a href="#" title="Social"><img src="../../c/elephant.svg" alt="Social"></a>
</div>
<h1>Large File Storage</h1>
<div class="keepws">
(<span class="bold">notes</span>
(date 2025-06-06)
(category <a href="../../notes.html">website</a>)
(status <span class="status-evergreen">evergreen</span>)
(confidence certain)
(importance 8)
(tags methodology tools storage))
</div>
<p class="preview">Implementation details of large file storage solutions for the website</p>
<div class="prose">
<p>As of (4/28/2025) krisyotam.com now has support for Large Files (>200MB). The post here from <a href="https://gwern.net/blog/2025-large-files">gwern.net</a> details much of the
necessity of this for a "long-now" focused site. For those who would rather not read I will provide a synopsis of such here.</p>
<h2>Why Large File Storage</h2>
<p>Lots of people make use of services like OneDrive, Google Drive, iCloud Drive, and Mega for cloud storage services. These platforms tend to be failry straightforward,
designed for ease of use, and []. This does come with tons of draw backs however. With these platforms there is a lot of High Level Abstraction (you interact with apps, not servers),
Freemium Models (Small free amounts that scale prices terribly), Shared Hosting (your files live with a million others), Integrated Ecosystems (Locked into apple, microsoft, etc.), and severe Privacy Concerns.
So how is <a href="https://www.hetzner.com/storage/storage-box/">Hetzner Storage Boxes</a> different? HSB is Infrastructure Focused (designed for developers, and power users), and provides Low-Level Control (Access via FTP, SFTP, SCP, rsync, etc.).
It can also act as a system hardrive (otehr cloud providers can use this specific feature). It has no Ecosystem Lock-in (works with any os, tool, or client).</p>
<h2>Need for Large-File Support</h2>
<p>It's no secret that this site will have tons expository work done over the years. That comes with thousands of references, resources, and citations. With this number easily multiplying itself due to the ease of implementing tools like perplexity
to help myself discover new sources I would have otherwise not found. I find with sources the more the better. Why lean towards a scarcity mindset? Would it not be more convicning to see 10, 20, or even 30 studies with reproducable resutls on a certain topic.
Then say being motivatied to make a otherwise significant change in your life based off a one of source. So for this first reason it is less a need for the store of singular "Large Files" but the need for elphantine storage sizes caused by the consistent accumulation of
small pdfs, mp3s, mp4s, etc. See also my method of countering this via my <a href="/blog/2025/automated-url-archiving">automated url archving</a> script. Beyond the simple things my main need for this implementation of the preservation of sites. Such as that of
geologist, and Independant Researcher <a href="https://independent.academia.edu/LeurenKMoret">Leuren Moret</a>. Her site recently when down between the last time I had visited it in mid 2024. It's no telling when valuable assets of information may go down. I have more thoughts on the extended preservation of acess in situations like these.
Such as the retaining of eth domains such as leurenmoret.eth and pinning her content via IPFS, or even using off-shore hosting providers which I have been deliberating of which to switch to before the content here starts to get more serious. For now at least using a viable LFS provider allows me to take my time weighing my options and retain
offline access to such information while my doing so. I am still weighing the value of mp4s for long-term storage. They might be pleasing and add visual stimuli but for most content with great information density it is just not necessary. I would rather sacrifice the visual fiedlity of a mp4 for the extra stroage gained by using mp3s. In such situations
I am also thinking about ideas for the transcription of videos through things like OpenAI whisper. Maybe getting a extra Hetzner server for such purposes would not be a bad idea.</p>
<blockquote>
The opposite of courage in our society is not cowardice... it is conformity.<br>
<br>
-- Rollo May<br>
</blockquote>
<h2>Drawbacks of Implementation</h2>
<p>I have deliberated for a while about tons of alternative solutions to this problem as mentioned above in (#Why Large File Storage?). There where several drawbacks to consider such as privacy, security, ease of access,
pricing over the long-term, portability, etc. a number of these factors out right eliminated the more traditional options such as OneDrive, Google Drive (G-Suite), iCloud Drive, and even Mega as a main source for LFS. Even though I am still particularly fond of Mega, and still retain
my subscription due to the not infrequent use of receiving massive amounts of data from people via the platform. The biggest drawback that most of these platforms have was either the inabillity to download via HTTPS. I was also drastically influenced by the massive pricing difference which we will dicuss next.</p>
<h2>Competitor Price Comparison</h2>
<p>Unlike the popular consumer cloud providers above there are a number of <em>viable</em> alternatives to Hetzner. Those include Wasabi, Scaleway, Storj, and my own system backup provider BackBlaze B2.
As you can see below however there is simply no comparison when it comes to pricing models.</p>
<h2>Use Cases</h2>
<p>The immediate use case that comes to mind for the newly integrated Large File Storage (LFS) is the completion of the formerly delayed
archive page. It should be of no suprise that the future of this blog will discuss at length a variety of topics that must be heavily supported with documentation. Documentation like
Panama Papers, Paradise Papers, Offshore Leaks, Silk Road Archives, RaidForums Dumps, and many more data sets. It will also include the storage of youtube videos I think are in danger of being silenced.
Things like this as well as research papers, historical documents, etc. are to be stored for reference here on the site and the archive page to be made available to people at my discretion.</p>
<p>More fun things to store include the GeoCities Archives, which represent an important piece of early internet history that would otherwise be lost to time.</p>
<h2>Implementation</h2>
<p>The site runs as a Next.js app behind nginx on a Hetzner server. Large files don’t go through Next.js at all. They sit on a 1TB drive mounted at <code>/mnt/storage</code> on the same box, and nginx serves them straight from disk. Two directories do all the work: <a href="https://krisyotam.com/doc">/doc</a> and <a href="https://krisyotam.com/cdn">/cdn</a>.</p>
<h3>/doc</h3>
<p>This is the document archive. Mostly research papers and PDFs. Some leaked datasets, some transcripts. If I cite something in a post, I keep a copy here so the link can’t rot out from under me. The files are sorted by subject: <code>/doc/mathematics/</code>, <code>/doc/psychology/</code>, <code>/doc/history/</code>, etc. When someone requests a file like <code>/doc/philosophy/some-paper.pdf</code>, nginx matches the extension and hands it back with a 7-day cache header. Browsing <code>/doc/</code> without a specific file falls through to the Next.js app, which renders a listing page.</p>
<h3>/cdn</h3>
<p>This is the media side. Cover images, portraits, audio files. Anything the site needs to display but that would be insane to commit to the git repo. The directory structure follows the site’s content categories, so you get paths like <code>/cdn/images/people/authors/</code> and <code>/cdn/audio/</code>. nginx has autoindex turned on here so you can browse it as a file listing. I inject a custom theme into those listing pages through nginx’s <code>sub_filter</code> module.</p>
<p>Both directories are symlinks from the home directory to the storage drive:</p>
<pre><code>~/cdn -> /mnt/storage/cdn
~/doc -> /mnt/storage/doc
</code></pre>
<h3>How the routing works</h3>
<p>nginx does all of it. For <code>/doc</code>, a regex location block catches requests ending in known file extensions (<code>.pdf</code>, <code>.epub</code>, <code>.mp3</code>, etc.) and serves them from <code>/home/krisyotam/doc/</code> using <code>root</code>. That follows the symlink down to <code>/mnt/storage/doc/</code>. Anything that doesn’t match a file extension passes through to the Next.js app on <code>127.0.0.1:3080</code>.</p>
<p>For <code>/cdn</code>, an <code>alias</code> directive points the URL path to the storage directory. Autoindex handles the file browser. <code>sub_filter</code> injects the theme files from <code>/doc-theme/</code>.</p>
<p>Uploads happen over SSH:</p>
<pre><code>scp paper.pdf server:/mnt/storage/doc/mathematics/
scp portrait.jpg server:/mnt/storage/cdn/images/people/authors/
</code></pre>
<p>For bigger transfers I use <code>rsync</code>. TLS comes from Let’s Encrypt via certbot, with nginx terminating SSL on 443.</p>
<h3>Why I did it this way</h3>
<p>No S3. No object storage API. No monthly metering. The files sit on a drive I own, served by nginx which is already running for the site. The 1TB drive was a one-time cost. If I run out of space I add another one. If I move servers I <code>rsync</code> the whole thing. The URLs are just paths on my own domain, so there’s no external service involved that could change its pricing or disappear.</p>
</div>
</body>
</html>