~kris/sbot: robots.h main

~kris/hacks

sbot/robots.h -rw-r--r-- 754 bytes

e3abbc9a — Kris Yotam 7 months ago

archiver-bot v0.2: hash table, robots.txt, retry logic