~kris/dots

srice

ref: e9b48d06a8541f3eda5c4db90382ab3c77183afb srice/doc/tools-advanced.md -rw-r--r-- 10.0 KiB
e9b48d06 — Kris Yotam xprofile: systemd-aware pipewire start + blueman-applet; sb-internet: tolerate missing /proc/net/wireless 2 months ago

#CLI Tools -- advanced and obscure

These tools are not in tools.md. Each one covers a blind spot that standard tooling handles badly. If you have the basics down, these are the next tier.


#File Watching

#entr

What it does: Runs a command whenever watched files change.
Killer use case: ls *.go | entr -cr go run . -- restarts your server on every save. -c clears the screen, -r kills and restarts the process.
Install: pacman -S entr

#watchexec

What it does: Language-agnostic file watcher with gitignore support and sane defaults.
Killer use case: watchexec -e rs cargo test -- reruns tests on any .rs change, without listing files manually.
Install: pacman -S watchexec


#Environment and Version Management

#direnv

What it does: Loads and unloads .envrc files automatically when you enter or leave a directory.
Killer use case: Store export DATABASE_URL=... in .envrc; it activates on cd and disappears when you leave. No more forgotten env vars polluting other projects.
Install: pacman -S direnv
Note: Already listed in tools.md under Navigation but omitted from the install line. Details added here.

#mise

What it does: Universal version manager (the actively maintained asdf successor). Handles Node, Python, Ruby, Go, Rust toolchains, and 500+ more.
Killer use case: mise use node@22 python@3.12 in a project root writes a .mise.toml that any teammate with mise will honor automatically.
Install: pacman -S mise


#zoxide

What it does: Frecency-based cd -- tracks which directories you visit and weights by frequency and recency.
Killer use case: z proj jumps to ~/dev/some-deep/project/path without typing the full path. Already in tools.md; included here for the workflow detail.
Install: pacman -S zoxide

#atuin

What it does: Replaces shell history with a SQLite database; adds fuzzy search, timestamps, exit codes, and optional cross-machine sync.
Killer use case: atuin search --exit 1 -- find every command that ever failed. Already in tools.md; included here for the exit-code filter detail.
Install: pacman -S atuin

What it does: Interactive cheatsheet browser. You write .cheat files in a simple syntax and navi gives you a fuzzy-searchable menu that fills in placeholders.
Killer use case: Store your ffmpeg and openssl incantations as cheat entries. navi surfaces them and drops the filled command into your prompt.
Install: pacman -S navi


#JSON / YAML / Structured Data

#gron

What it does: Flattens JSON into grep-able assignments (path.to.key = "value"), then reverses the process.
Killer use case: gron api.json | grep '"error"' | gron --ungron -- find and extract any nested key without knowing the schema.
Install: pacman -S gron

#dasel

What it does: Single binary that queries and mutates JSON, YAML, TOML, XML, and CSV with one unified selector syntax.
Killer use case: dasel -f config.yaml 'server.port' works the same whether the file is YAML or TOML. Stop context-switching between jq/yq/tomlq.
Install: pacman -S dasel

#yq

What it does: jq for YAML (and JSON, TOML, XML, CSV).
Killer use case: yq '.services.web.image' docker-compose.yml -- read a Compose file without writing a Python script.
Install: pacman -S yq

#miller (mlr)

What it does: awk/sed/cut/join/sort for CSV, TSV, JSON, and other structured formats in a single tool.
Killer use case: mlr --csv sort -f name then filter '$age > 30' data.csv -- SQL-style operations on flat files without spinning up a database.
Install: pacman -S miller

#pup

What it does: jq for HTML. Parses HTML and lets you select elements with CSS selectors.
Killer use case: curl -s https://example.com | pup 'a[href] attr{href}' -- extract all links from a page without BeautifulSoup.
Install: pacman -S pup

#htmlq

What it does: Like pup but closer to the jq interface. More actively maintained.
Killer use case: curl -s https://news.ycombinator.com | htmlq '.storylink' -- scrape headlines from any site.
Install: pacman -S htmlq

#xsv / qsv

What it does: Fast CSV toolkit. xsv is the original; qsv is the maintained fork with more commands.
Killer use case: xsv stats data.csv prints min/max/mean/stddev for every column instantly on a million-row file where Excel would crash.
Install: pacman -S xsv (or build qsv from AUR)


#Diffing and Patching

#difftastic

What it does: Syntax-aware structural diff. Understands the grammar of 30+ languages and ignores formatting-only changes.
Killer use case: Rename a variable across a file. Regular diff shows every line changed; difftastic shows only the semantic change.
Install: pacman -S difftastic
Wire it into git: git config diff.external difft


#Text Manipulation

#sad

What it does: Batch find-and-replace across files with a live preview before writing.
Killer use case: sad 'OldName' 'NewName' -- **/*.go shows a side-by-side diff of every planned change; you confirm before it touches disk.
Install: yay -S sad (AUR)

#sd

What it does: Simpler sed. Uses regex by default, supports literal strings without escaping, reads from file or stdin.
Killer use case: sd 'foo(\w+)' 'bar$1' file.txt -- no doubled backslashes, no cryptic flags.
Install: pacman -S sd

#choose

What it does: Simpler cut and awk field selector. Zero-indexed, supports negative indices, handles any whitespace delimiter.
Killer use case: ps aux | choose 10 -- grab the last column. choose 0 2 for first and third.
Install: pacman -S choose


#Networking

#bandwhich

What it does: Shows real-time network bandwidth broken down by process and remote host.
Killer use case: Run it during a slow deploy to see which process is saturating the link and where the traffic is going.
Install: pacman -S bandwhich

#doggo

What it does: Modern dig replacement. Human-readable output, supports DoH and DoT, colored tables.
Killer use case: doggo krisyotam.com MX @1.1.1.1 -- readable DNS debug without decoding dig's wire-format output.
Install: pacman -S doggo

#socat

What it does: Bidirectional data relay between almost any two endpoints (TCP, UDP, Unix sockets, files, serial ports, TLS).
Killer use case: socat TCP-LISTEN:8080,fork TCP:localhost:3000 -- forward a port without iptables. Also useful for poking Unix sockets directly.
Install: pacman -S socat


#Pipes and Process Control

#pv

What it does: Monitors data passing through a pipe and shows throughput, elapsed time, ETA, and a progress bar.
Killer use case: cat huge.sql | pv | mysql mydb -- finally know how far along that import is.
Install: pacman -S pv

#task-spooler (tsp)

What it does: A command queue. Submit jobs; they run one at a time (or N at a time). Jobs persist even if you close the terminal.
Killer use case: Queue five long ffmpeg encodes with tsp ffmpeg ... each. They run sequentially; tsp -l shows the queue.
Install: pacman -S task-spooler

#rlwrap

What it does: Wraps any program's stdin with readline editing, history, and tab completion.
Killer use case: rlwrap sqlite3 mydb gives you arrow-key history in sqlite3 without patching sqlite3 itself. Works on any REPL that lacks readline.
Install: pacman -S rlwrap

#expect

What it does: Scripted automation for interactive programs. Watches for patterns in output and sends input in response.
Killer use case: Automate an SSH login to a device that requires password auth and then runs a fixed sequence of commands. Where shell scripts give up, expect continues.
Install: pacman -S expect


#moreutils

A collection of small Unix tools that fill gaps in coreutils. Install the whole package: pacman -S moreutils

Tool What it does Killer use case
sponge Soaks up stdin, then writes to file `grep -v bad file sponge file` -- edit a file in-place without a temp file
ts Prepends timestamps to each line of stdin `tail -f app.log ts '[%Y-%m-%d %H:%M:%S]'` -- timestamped live logs
chronic Runs a command silently, but shows output if it fails chronic rsync -av src/ dst/ -- cron jobs that email only on failure
vipe Inserts your editor into a pipeline `curl api vipe jq .` -- edit the JSON before passing it downstream
vidir Opens a directory listing in your editor for batch rename/delete vidir ~/downloads -- rename 40 files as text editing
ifne Runs a command only if stdin is non-empty `find . -name '*.tmp' ifne mail -s "Stale files" me@me.com`
pee Sends stdin to multiple commands (tee for pipelines) pee 'wc -l' 'grep error' < log.txt
combine Combines two files line-by-line with a set operation combine list-a.txt and list-b.txt -- intersection without awk
errno Looks up errno values by name or number errno ENOENT -- readable error reference without man pages

#If you install nothing else from this list

These ten cover the highest-leverage gaps:

  1. entr -- live reload during development
  2. mise -- one tool for all language version pinning
  3. difftastic -- structural diff that actually shows what changed
  4. miller -- SQL-like queries on CSV/TSV/JSON without a database
  5. gron -- make any JSON greppable in one command
  6. sd -- sed that you can actually remember the syntax for
  7. pv -- progress bars in pipes, invaluable for long data operations
  8. socat -- port forwarding and socket debugging without ceremony
  9. moreutils -- sponge, ts, chronic, and vidir alone are worth the install
  10. navi -- your own interactive cheatsheet for the commands you always Google

Install: pacman -S entr mise difftastic miller gron sd pv socat moreutils navi