# Unix Pipeline Mastery A dense reference for pipelines, text processing, and system introspection. Each section gives working one-liners with enough explanation to adapt them. --- ## 1. awk One-Liners **Deduplicate lines (preserving order)** ```bash awk '!seen[$0]++' file.txt ``` `seen[$0]` is zero (false) on first encounter, incremented to 1 after the `!` test passes. Subsequent duplicates fail the test and are skipped. Unlike `sort | uniq`, order is preserved. **Frequency count of a field** ```bash awk '{count[$1]++} END {for (k in count) print count[k], k}' file.txt | sort -rn ``` Counts occurrences of the first field, prints freq then value, sorts descending. **Group-by sum (sum field 3 grouped by field 1)** ```bash awk '{sum[$1] += $3} END {for (k in sum) print k, sum[k]}' file.txt ``` **Inline CSV to TSV** ```bash awk 'BEGIN {FPAT="([^,]*)|(\"[^\"]+\")"; OFS="\t"} {for(i=1;i<=NF;i++) gsub(/^"|"$/, "", $i); $1=$1; print}' file.csv ``` `FPAT` makes awk parse quoted CSV fields correctly. `$1=$1` forces field reconstruction with `OFS`. **Filter rows where a field count differs (ragged lines)** ```bash awk 'NF != 5' file.txt # lines that do NOT have exactly 5 fields awk 'NF >= 3 && NF <= 7' file.txt ``` **Print lines between two patterns (inclusive)** ```bash awk '/START/,/END/' file.txt ``` **Column math: add two fields, print result** ```bash awk '{print $1, $2 + $3}' file.txt ``` **Print unique lines in field 2 only (deduplicate on specific key)** ```bash awk '!seen[$2]++' file.txt ``` --- ## 2. sed Tricks **Alternate delimiters (avoids escaping slashes in paths)** ```bash sed 's|/usr/local/bin|/opt/bin|g' file sed 's#https://old.example.com#https://new.example.com#g' file ``` Any character after `s` becomes the delimiter. Useful with paths and URLs. **Section editing: delete lines between markers** ```bash sed '/^# BEGIN/,/^# END/d' file ``` **Section editing: replace content inside markers** ```bash sed '/^# BEGIN/,/^# END/ s/foo/bar/g' file ``` **Section editing: extract a block** ```bash sed -n '/^# BEGIN/,/^# END/p' file ``` **DOS to Unix (strip carriage returns)** ```bash sed 's/\r$//' file.txt # or in-place: sed -i 's/\r$//' file.txt ``` **Commify numbers (add thousand separators)** ```bash echo "1234567" | sed ':a; s/\B[0-9]\{3\}\>/,&/; ta' ``` `:a` defines label. `\B` matches non-word-boundary before groups of 3 digits from the right. `ta` loops until no substitution occurs. **Delete blank lines** ```bash sed '/^[[:space:]]*$/d' file.txt ``` **Print line number N** ```bash sed -n '42p' file.txt ``` **Insert a line after a match** ```bash sed '/^MATCH/a\inserted line here' file.txt ``` **In-place edit with backup** ```bash sed -i.bak 's/old/new/g' file.txt ``` `.bak` suffix creates `file.txt.bak` before editing. GNU sed accepts `-i` without space; BSD sed requires `-i ''`. --- ## 3. xargs Advanced **Handle filenames with spaces/newlines using null delimiter** ```bash find . -name "*.log" -print0 | xargs -0 rm -f ``` `-print0` outputs NUL-separated names; `-0` reads them. Never use bare `xargs` with filenames. **Parallel execution with -P** ```bash cat urls.txt | xargs -P 8 -I{} curl -sO {} ``` `-P 8` runs up to 8 processes concurrently. `-I{}` sets the placeholder for the argument. **Placeholder substitution** ```bash ls *.tar.gz | xargs -I{} tar -xzf {} -C /tmp/{}_extracted ``` `-I{}` replaces `{}` with the argument everywhere in the command. Note: `-I` implies `-L 1` (one argument per invocation). **Batch multiple args per invocation with -n** ```bash cat ids.txt | xargs -n 5 ./process_batch.sh ``` Calls `process_batch.sh arg1 arg2 arg3 arg4 arg5`, then the next 5, etc. Useful for APIs with batch endpoints. **Combine -n and -P for parallel batching** ```bash cat hosts.txt | xargs -n 1 -P 20 -I{} ssh {} 'hostname && uptime' ``` **Dry run: echo what would execute** ```bash cat files.txt | xargs -I{} echo rm {} ``` **xargs with shell logic (requires sh -c)** ```bash find . -name "*.md" -print0 | xargs -0 -I{} sh -c 'wc -l "{}" && echo "---"' ``` --- ## 4. GNU parallel vs xargs -P Both parallelize, but GNU parallel is richer. **Basic parallel (same as xargs -P)** ```bash cat hosts.txt | parallel ssh {} hostname ``` **Ordered output (parallel collects and prints in input order)** ```bash cat urls.txt | parallel --keep-order curl -s {} ``` `xargs -P` interleaves output unpredictably. `--keep-order` buffers until earlier jobs finish. **File extension replacement with {.}** ```bash parallel ffmpeg -i {} {.}.mp3 ::: *.wav ``` `{.}` strips extension. `{/}` is basename, `{//}` is dirname, `{/.}` is basename without extension. **Resume a failed run** ```bash parallel --joblog /tmp/jobs.log --resume my_command ::: input*.txt # Re-run failures only: parallel --joblog /tmp/jobs.log --resume-failed my_command ::: input*.txt ``` **--pipe: split stdin across workers** ```bash cat big.csv | parallel --pipe --block 10M grep "pattern" ``` Splits stdin into 10M chunks and runs grep on each in parallel. Faster than single grep on huge files. **Limit rate and add delay** ```bash cat api_ids.txt | parallel --delay 0.5 --jobs 4 curl "https://api.example.com/item/{}" ``` **Build command strings with multiple inputs** ```bash parallel echo {1} {2} ::: a b c ::: x y z ``` Computes the Cartesian product: a x, a y, a z, b x, ... --- ## 5. Forgotten Text Tools ### comm: Set Operations on Sorted Files ```bash sort file1 > a.sorted sort file2 > b.sorted comm -12 a.sorted b.sorted # intersection (lines in both) comm -23 a.sorted b.sorted # lines only in file1 comm -13 a.sorted b.sorted # lines only in file2 comm -3 a.sorted b.sorted # symmetric difference (lines in exactly one) ``` Files must be sorted. Output has three columns: only-in-file1, only-in-file2, in-both. Suppressing columns with -1/-2/-3. ### join: Relational Join on a Common Field ```bash join -t, -1 1 -2 1 users.csv orders.csv ``` Joins on field 1 of each file, comma-delimited. Like SQL `INNER JOIN`. Files must be sorted on the join field. ```bash join -a 1 file1 file2 # left outer join (keep unmatched from file1) join -a 2 file1 file2 # right outer join join -v 1 file1 file2 # only lines from file1 with NO match (anti-join) ``` ### paste: Merge Files Side by Side ```bash paste file1 file2 # tab-separated columns paste -d, file1 file2 # comma-delimited paste -s file.txt # transpose: one file, all lines become one row paste -d'\n' file1 file2 # interleave lines (alternate) ``` Transpose a matrix (rows become columns): ```bash paste -s file.txt | awk '{for(i=1;i<=NF;i++) col[i]=col[i] (col[i]?"\t":"") $i} END {for(i=1;i<=NF;i++) print col[i]}' ``` ### cut ```bash cut -d, -f2,4 file.csv # extract fields 2 and 4 cut -c1-10 file.txt # extract characters 1-10 cut -d: -f1,6 /etc/passwd # username and shell ``` ### tr: Translate or Delete Characters ```bash tr '[:lower:]' '[:upper:]' < file.txt # uppercase tr -d '\r' < dos.txt # strip CR tr -s ' ' < file.txt # squeeze multiple spaces tr -dc '[:alnum:]\n' < file.txt # delete non-alphanumeric echo "hello world" | tr ' ' '\n' # one word per line ``` ### fold: Wrap Long Lines ```bash fold -w 72 -s file.txt # wrap at 72 chars, break at spaces ``` ### fmt: Reformat Paragraphs ```bash fmt -w 72 essay.txt # reflow paragraphs to 72 char width fmt -u file.txt # uniform spacing (one space after period) ``` --- ## 6. column -t for Instant Tables ```bash mount | column -t ``` ```bash { echo "NAME AGE CITY"; echo "Alice 30 NYC"; echo "Bob 25 LA"; } | column -t ``` With a custom delimiter: ```bash cat /etc/passwd | column -t -s: ``` Output alignment only (no column separator): ```bash ps aux | column -t | head ``` Fixed separator in output: ```bash column -t -s, file.csv ``` Column with header and borders (newer util-linux): ```bash column -t -N "Host,Port,Service" -s, services.csv ``` --- ## 7. sort Advanced **Multi-key sort: primary by field 3 numeric, secondary by field 1 alphabetic** ```bash sort -k3,3n -k1,1 file.txt ``` `-k3,3n` means sort on field 3 only (start,end with same number) numerically. Without the end field, sort uses the rest of the line. **Human-readable size sort (10M before 2G)** ```bash du -sh * | sort -h ``` **Version sort (1.9 before 1.10)** ```bash ls v*.tar.gz | sort -V ``` **Debug sort keys (shows what key was used per line)** ```bash sort -k2,2n --debug file.txt ``` Underlines the portion of each line used as the sort key. Invaluable for debugging unexpected order. **Limit RAM usage** ```bash sort -S 2G large.txt ``` Default is often 10% of RAM. Explicit `-S` avoids hitting swap. **Parallel sort** ```bash sort --parallel=8 large.txt ``` **Stable sort (preserve original order of equal elements)** ```bash sort -s -k1,1 file.txt ``` **Sort CSV by second field, ignoring header** ```bash (head -1 file.csv; tail -n +2 file.csv | sort -t, -k2,2) > sorted.csv ``` --- ## 8. tee Fan-Out and Mid-Pipeline Debugging **Fan output to multiple files and stdout** ```bash pipeline | tee file1.txt file2.txt | next_command ``` **Fan-out to two different commands using process substitution** ```bash cat data.txt | tee >(gzip > data.gz) >(wc -l > count.txt) | grep "ERROR" ``` Both subshells receive every line. Stdout of `tee` continues to the next pipe stage. **Debug mid-pipeline without breaking the flow** ```bash cat data.txt | grep "foo" | tee /dev/stderr | awk '{print $2}' > result.txt ``` `/dev/stderr` lets you see the intermediate stream in the terminal while the pipe continues. **Log mid-pipeline with timestamp** ```bash pipeline | tee >(ts '%Y-%m-%d %H:%M:%S' >> pipeline.log) | next_stage ``` (`ts` from `moreutils`) **Duplicate to multiple named pipes** ```bash mkfifo /tmp/p1 /tmp/p2 tee /tmp/p1 /tmp/p2 < input.txt & consumer1 < /tmp/p1 & consumer2 < /tmp/p2 & wait ``` --- ## 9. jq Advanced **Select / filter objects** ```bash jq '.[] | select(.status == "active")' users.json jq '.[] | select(.age > 30 and .city == "NYC")' users.json ``` **group_by and count** ```bash jq 'group_by(.status) | map({status: .[0].status, count: length})' data.json ``` **to_entries: iterate over object keys** ```bash jq 'to_entries | map(.key + "=" + (.value | tostring)) | .[]' config.json ``` Useful for converting JSON objects to env-var format. **@csv and @tsv output** ```bash jq -r '.[] | [.name, .age, .city] | @csv' users.json jq -r '.[] | [.name, .age, .city] | @tsv' users.json ``` **reduce: fold over an array** ```bash jq 'reduce .[] as $x (0; . + $x.amount)' transactions.json ``` **// for defaults (alternative operator)** ```bash jq '.[] | {name: .name, role: (.role // "user")}' users.json ``` If `.role` is null or missing, substitutes "user". **-s slurp: read multiple JSON docs into an array** ```bash cat *.json | jq -s '.' cat *.json | jq -s 'map(select(.active)) | length' ``` **Compact output + raw strings** ```bash jq -rc '.[] | .id' data.json # -r raw (no quotes), -c compact (no pretty print) ``` **Update a field in place** ```bash jq '(.[] | select(.id == 42)).status = "done"' data.json ``` **Build new objects from arrays** ```bash jq '[.[] | {(.id | tostring): .name}] | add' users.json ``` **Extract nested paths** ```bash jq '[path(..| numbers)] ' data.json # all paths to numeric values ``` --- ## 10. curl Tricks **Timing breakdown with -w write-out** ```bash curl -o /dev/null -s -w "\ DNS: %{time_namelookup}s\n\ Connect: %{time_connect}s\n\ TLS: %{time_appconnect}s\n\ TTFB: %{time_starttransfer}s\n\ Total: %{time_total}s\n\ HTTP: %{http_code}\n\ Size: %{size_download} bytes\n\ " https://example.com ``` **Check HTTP status in scripts** ```bash curl --fail -sS https://api.example.com/health || echo "DOWN" ``` `--fail` exits non-zero on 4xx/5xx. `-sS` is silent but shows errors. **Retry with backoff** ```bash curl --retry 5 --retry-delay 2 --retry-max-time 30 https://flaky.api/endpoint ``` **Cookie jar (session persistence)** ```bash curl -c cookies.txt -b cookies.txt -d "user=admin&pass=secret" https://site.com/login curl -b cookies.txt https://site.com/protected ``` **Follow redirects and get final URL** ```bash curl -Ls -o /dev/null -w "%{url_effective}" https://short.url/abc ``` **Download with resume** ```bash curl -C - -O https://example.com/big-file.iso ``` **Multiple URLs in parallel (xargs)** ```bash cat urls.txt | xargs -P 10 -I{} curl -sO {} ``` **POST JSON** ```bash curl -X POST -H "Content-Type: application/json" -d '{"key":"value"}' https://api/endpoint ``` **Upload file** ```bash curl -F "file=@/path/to/file.csv" https://upload.example.com/ ``` --- ## 11. netcat Patterns **Port check (TCP)** ```bash nc -zv host 443 # verbose, exit after check nc -z -w 2 host 8080 # 2s timeout ``` **Banner grab** ```bash echo "" | nc -w 3 host 22 # SSH banner echo "HEAD / HTTP/1.0\r\n\r\n" | nc -w 3 host 80 ``` **Simple file transfer** ```bash # receiver: nc -l 9999 > received_file # sender: nc receiver_host 9999 < file_to_send ``` **With pv for progress** ```bash pv file.tar.gz | nc receiver 9999 ``` **TCP proxy with mkfifo (bidirectional)** ```bash mkfifo /tmp/proxy_pipe nc -l 8080 < /tmp/proxy_pipe | nc real_backend 80 > /tmp/proxy_pipe ``` Traffic to localhost:8080 is forwarded to real_backend:80. **One-shot HTTP server (serve a file)** ```bash { echo -e "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\n"; cat file.txt; } | nc -l 8080 ``` **UDP listener** ```bash nc -u -l 5140 # listen on UDP 5140 (syslog) ``` **Scan a range of ports** ```bash nc -zv host 20-1024 2>&1 | grep "succeeded" ``` --- ## 12. find -exec Patterns **{} + vs {} \; (batching vs per-file)** ```bash find . -name "*.log" -exec rm {} + # rm file1 file2 ... (one invocation) find . -name "*.log" -exec rm {} \; # rm file1; rm file2; ... (one per file) ``` `{} +` is like `xargs`; faster for many files. `{} \;` is needed when the command cannot take multiple arguments. **Shell logic inside -exec** ```bash find . -name "*.txt" -exec sh -c 'wc -l "$1" >> /tmp/counts.txt' _ {} \; ``` The `_` is `$0` (script name placeholder); `{}` becomes `$1`. **Process multiple files with shell logic** ```bash find . -name "*.jpg" -print0 | xargs -0 -I{} sh -c ' base=$(basename "{}" .jpg) convert "{}" -resize 800x /tmp/resized/${base}_sm.jpg ' ``` **Chain conditions as AND** ```bash find . -name "*.py" -newer setup.py -exec grep -l "TODO" {} \; ``` Multiple predicates are AND by default. Use `-o` for OR, `\( \)` for grouping. **Delete empty directories** ```bash find . -type d -empty -delete ``` **Find and compress files older than 30 days** ```bash find /var/log -name "*.log" -mtime +30 -exec gzip {} \; ``` **Find files by size range** ```bash find . -size +1M -size -100M -type f ``` **Exclude a directory** ```bash find . -path ./node_modules -prune -o -name "*.js" -print ``` --- ## 13. strace Patterns **Trace only specific syscall categories** ```bash strace -e trace=file command # open, stat, read, write on files strace -e trace=network command # socket, connect, bind, send, recv strace -e trace=memory command # mmap, mprotect, brk strace -e trace=process command # fork, exec, wait, clone ``` **Summary of syscall counts and time** ```bash strace -c -p PID # attach to running process, print summary on exit strace -c command ``` Shows number of calls, errors, and total time per syscall. **Per-syscall timing** ```bash strace -T command 2>&1 | grep "open" # shows time in <0.000123> after each call ``` **Resolve file descriptor paths** ```bash strace -yy command # annotates fds with full path in every syscall ``` Without `-yy`, you see `read(4, ...)`. With `-yy`, you see `read(4, ...)`. **Follow child processes** ```bash strace -f command # trace forked children too strace -ff -o /tmp/trace command # one file per pid: /tmp/trace.PID ``` **Filter output** ```bash strace -e trace=openat command 2>&1 | grep -v "ENOENT" # hide missing files ``` **Attach to existing process** ```bash strace -p $(pgrep nginx) ``` --- ## 14. /proc/$PID Tricks ```bash PID=$(pgrep myapp) ``` **Full command line (null-separated args)** ```bash cat /proc/$PID/cmdline | tr '\0' ' ' ``` **Environment variables** ```bash cat /proc/$PID/environ | tr '\0' '\n' cat /proc/$PID/environ | tr '\0' '\n' | grep PATH ``` **Open file descriptors** ```bash ls -la /proc/$PID/fd # symlinks to open files, sockets, pipes ``` **Memory map (which libraries are loaded)** ```bash cat /proc/$PID/maps ``` Shows address ranges, permissions, and file backing. Useful for detecting injected libraries or diagnosing memory layout. **Process status (state, memory, threads)** ```bash cat /proc/$PID/status ``` Contains `VmRSS` (resident set), `VmSwap`, `Threads`, `State`. **I/O statistics** ```bash cat /proc/$PID/io ``` `rchar`, `wchar` are bytes read/written. `syscr`, `syscw` are syscall counts. `read_bytes`, `write_bytes` are actual disk I/O. **Stack trace (kernel stack)** ```bash cat /proc/$PID/stack # kernel stack frames (requires CAP_SYS_PTRACE or root) ``` **Limits** ```bash cat /proc/$PID/limits # ulimits currently in effect ``` **Live CPU/memory for all processes** ```bash paste <(ps -eo pid,comm) <(cat /proc/*/status 2>/dev/null | grep VmRSS | awk '{print $2}') | sort -k3 -rn | head ``` --- ## 15. coproc: Bidirectional Pipe Communication `coproc` creates a background process with two-way pipes. Unlike `cmd | read`, you can write and read repeatedly. **Basic usage** ```bash coproc BC { bc -l; } echo "3.14159 * 2" >&${BC[1]} read result <&${BC[0]} echo "Result: $result" ``` `BC[1]` is stdin of the coprocess; `BC[0]` is its stdout. **Named coprocess (bash 4+)** ```bash coproc CALC { python3 -c "import sys; [print(eval(l)) for l in sys.stdin]"; } echo "2**10" >&${CALC[1]} read answer <&${CALC[0]} echo "$answer" # 1024 ``` **Interactive reuse across loop iterations** ```bash coproc DB { sqlite3 mydb.sqlite; } for table in users orders products; do echo ".schema $table" >&${DB[1]} read schema <&${DB[0]} echo "$table: $schema" done echo ".quit" >&${DB[1]} ``` **Difference from stdin pipe** With a regular pipe you get one write/one read: `echo "input" | cmd`. With coproc you maintain a persistent session and can write multiple times without restarting the process. This matters when process startup is expensive (interpreters, DB connections, TLS negotiation). **Capture stderr separately** ```bash coproc CMD { my_tool 2>/tmp/cmd_errors; } ``` --- ## 10 Composition Patterns ### 1. Filter-Map-Reduce Filter down to relevant records, transform each record, then aggregate. ```bash awk -F, '$3 == "US"' orders.csv \ | awk -F, '{print $5}' \ | awk '{sum += $1} END {print sum}' ``` ### 2. Fan-Out: Same Data, Two Different Analyses ```bash cat access.log | tee \ >(awk '{print $1}' | sort | uniq -c | sort -rn | head > top_ips.txt) \ >(awk '{print $7}' | sort | uniq -c | sort -rn | head > top_paths.txt) \ > /dev/null ``` ### 3. Diff Two Commands Compare outputs from two command variants without temp files. ```bash diff <(curl -s https://api.example.com/v1/users | jq -r '.[] | .id' | sort) \ <(curl -s https://api.example.com/v2/users | jq -r '.[] | .id' | sort) ``` ### 4. Set Operations on Live Data ```bash # Which hosts are in inventory but not responding to ping? comm -23 \ <(sort inventory.txt) \ <(cat inventory.txt | parallel -j50 "ping -c1 -W1 {} &>/dev/null && echo {}" | sort) ``` ### 5. Safe Iteration Over File Paths ```bash find . -name "*.csv" -print0 \ | xargs -0 -I{} sh -c ' lines=$(wc -l < "{}") echo "{}: $lines lines" ' ``` ### 6. Parallel Transform with Ordered Output ```bash cat image_ids.txt | parallel --keep-order \ 'curl -s "https://api/image/{}" | jq -r ".url"' \ > ordered_urls.txt ``` ### 7. Column Extraction and Table Formatting ```bash ps aux \ | awk 'NR==1 || $3 > 1.0 {print $1, $2, $3, $4, $11}' \ | column -t ``` ### 8. API to Table ```bash curl -s "https://api.github.com/repos/torvalds/linux/releases" \ | jq -r '.[] | [.tag_name, .published_at, (.assets | length | tostring)] | @tsv' \ | column -t -s $'\t' ``` ### 9. Live Monitoring Pipeline ```bash watch -n1 'ss -tnp | awk "NR>1 {print \$1,\$4,\$5,\$6}" | column -t' ``` Or with continuous output: ```bash tail -f /var/log/nginx/access.log \ | awk '{print $1, $7, $9}' \ | grep --line-buffered -v "200" \ | ts '%H:%M:%S' ``` ### 10. Config Surgery: Extract, Edit, Re-embed ```bash # Extract a JSON block from a config, update a field, put it back value=$(jq '.database.pool_size' config.json) new_value=$((value * 2)) tmp=$(mktemp) jq --argjson n "$new_value" '.database.pool_size = $n' config.json > "$tmp" mv "$tmp" config.json ``` For INI-style configs: ```bash sed -i 's/^max_connections\s*=.*/max_connections = 200/' /etc/myapp/app.conf ``` --- ## Quick Reference Card | Tool | Key flag | Use | |------|----------|-----| | awk | `!seen[$0]++` | ordered dedup | | awk | `NF` | field count filter | | sed | `s\|/\|:\|g` | alternate delimiter | | sed | `/a/,/b/` | range address | | xargs | `-0 -P N -I{}` | null-safe parallel | | parallel | `--keep-order {.}` | ordered + extension strip | | comm | `-12 -23 -13` | intersection / diff / diff | | join | `-a 1 -v 1` | outer / anti join | | sort | `-k3,3n -h -V -s --debug` | multi-key / human / version | | tee | `>(cmd)` | fan-out via process sub | | jq | `select group_by reduce //` | filter / aggregate / default | | curl | `-w "%{time_total}" --fail --retry` | timing / scripts / retry | | nc | `-zv -l` | port check / listen | | find | `-exec {} + -print0` | batch / null-safe | | strace | `-e trace=file -c -T -yy -f` | category / summary / timing / fds | | /proc | `cmdline environ fd maps io` | process introspection | | coproc | `${NAME[0]} ${NAME[1]}` | bidirectional IPC |