A dense reference for pipelines, text processing, and system introspection. Each section gives working one-liners with enough explanation to adapt them.
Deduplicate lines (preserving order)
awk '!seen[$0]++' file.txt
seen[$0] is zero (false) on first encounter, incremented to 1 after the ! test passes. Subsequent duplicates fail the test and are skipped. Unlike sort | uniq, order is preserved.
Frequency count of a field
awk '{count[$1]++} END {for (k in count) print count[k], k}' file.txt | sort -rn
Counts occurrences of the first field, prints freq then value, sorts descending.
Group-by sum (sum field 3 grouped by field 1)
awk '{sum[$1] += $3} END {for (k in sum) print k, sum[k]}' file.txt
Inline CSV to TSV
awk 'BEGIN {FPAT="([^,]*)|(\"[^\"]+\")"; OFS="\t"} {for(i=1;i<=NF;i++) gsub(/^"|"$/, "", $i); $1=$1; print}' file.csv
FPAT makes awk parse quoted CSV fields correctly. $1=$1 forces field reconstruction with OFS.
Filter rows where a field count differs (ragged lines)
awk 'NF != 5' file.txt # lines that do NOT have exactly 5 fields
awk 'NF >= 3 && NF <= 7' file.txt
Print lines between two patterns (inclusive)
awk '/START/,/END/' file.txt
Column math: add two fields, print result
awk '{print $1, $2 + $3}' file.txt
Print unique lines in field 2 only (deduplicate on specific key)
awk '!seen[$2]++' file.txt
Alternate delimiters (avoids escaping slashes in paths)
sed 's|/usr/local/bin|/opt/bin|g' file
sed 's#https://old.example.com#https://new.example.com#g' file
Any character after s becomes the delimiter. Useful with paths and URLs.
Section editing: delete lines between markers
sed '/^# BEGIN/,/^# END/d' file
Section editing: replace content inside markers
sed '/^# BEGIN/,/^# END/ s/foo/bar/g' file
Section editing: extract a block
sed -n '/^# BEGIN/,/^# END/p' file
DOS to Unix (strip carriage returns)
sed 's/\r$//' file.txt
# or in-place:
sed -i 's/\r$//' file.txt
Commify numbers (add thousand separators)
echo "1234567" | sed ':a; s/\B[0-9]\{3\}\>/,&/; ta'
:a defines label. \B matches non-word-boundary before groups of 3 digits from the right. ta loops until no substitution occurs.
Delete blank lines
sed '/^[[:space:]]*$/d' file.txt
Print line number N
sed -n '42p' file.txt
Insert a line after a match
sed '/^MATCH/a\inserted line here' file.txt
In-place edit with backup
sed -i.bak 's/old/new/g' file.txt
.bak suffix creates file.txt.bak before editing. GNU sed accepts -i without space; BSD sed requires -i ''.
Handle filenames with spaces/newlines using null delimiter
find . -name "*.log" -print0 | xargs -0 rm -f
-print0 outputs NUL-separated names; -0 reads them. Never use bare xargs with filenames.
Parallel execution with -P
cat urls.txt | xargs -P 8 -I{} curl -sO {}
-P 8 runs up to 8 processes concurrently. -I{} sets the placeholder for the argument.
Placeholder substitution
ls *.tar.gz | xargs -I{} tar -xzf {} -C /tmp/{}_extracted
-I{} replaces {} with the argument everywhere in the command. Note: -I implies -L 1 (one argument per invocation).
Batch multiple args per invocation with -n
cat ids.txt | xargs -n 5 ./process_batch.sh
Calls process_batch.sh arg1 arg2 arg3 arg4 arg5, then the next 5, etc. Useful for APIs with batch endpoints.
Combine -n and -P for parallel batching
cat hosts.txt | xargs -n 1 -P 20 -I{} ssh {} 'hostname && uptime'
Dry run: echo what would execute
cat files.txt | xargs -I{} echo rm {}
xargs with shell logic (requires sh -c)
find . -name "*.md" -print0 | xargs -0 -I{} sh -c 'wc -l "{}" && echo "---"'
Both parallelize, but GNU parallel is richer.
Basic parallel (same as xargs -P)
cat hosts.txt | parallel ssh {} hostname
Ordered output (parallel collects and prints in input order)
cat urls.txt | parallel --keep-order curl -s {}
xargs -P interleaves output unpredictably. --keep-order buffers until earlier jobs finish.
File extension replacement with {.}
parallel ffmpeg -i {} {.}.mp3 ::: *.wav
{.} strips extension. {/} is basename, {//} is dirname, {/.} is basename without extension.
Resume a failed run
parallel --joblog /tmp/jobs.log --resume my_command ::: input*.txt
# Re-run failures only:
parallel --joblog /tmp/jobs.log --resume-failed my_command ::: input*.txt
--pipe: split stdin across workers
cat big.csv | parallel --pipe --block 10M grep "pattern"
Splits stdin into 10M chunks and runs grep on each in parallel. Faster than single grep on huge files.
Limit rate and add delay
cat api_ids.txt | parallel --delay 0.5 --jobs 4 curl "https://api.example.com/item/{}"
Build command strings with multiple inputs
parallel echo {1} {2} ::: a b c ::: x y z
Computes the Cartesian product: a x, a y, a z, b x, ...
sort file1 > a.sorted
sort file2 > b.sorted
comm -12 a.sorted b.sorted # intersection (lines in both)
comm -23 a.sorted b.sorted # lines only in file1
comm -13 a.sorted b.sorted # lines only in file2
comm -3 a.sorted b.sorted # symmetric difference (lines in exactly one)
Files must be sorted. Output has three columns: only-in-file1, only-in-file2, in-both. Suppressing columns with -1/-2/-3.
join -t, -1 1 -2 1 users.csv orders.csv
Joins on field 1 of each file, comma-delimited. Like SQL INNER JOIN. Files must be sorted on the join field.
join -a 1 file1 file2 # left outer join (keep unmatched from file1)
join -a 2 file1 file2 # right outer join
join -v 1 file1 file2 # only lines from file1 with NO match (anti-join)
paste file1 file2 # tab-separated columns
paste -d, file1 file2 # comma-delimited
paste -s file.txt # transpose: one file, all lines become one row
paste -d'\n' file1 file2 # interleave lines (alternate)
Transpose a matrix (rows become columns):
paste -s file.txt | awk '{for(i=1;i<=NF;i++) col[i]=col[i] (col[i]?"\t":"") $i} END {for(i=1;i<=NF;i++) print col[i]}'
cut -d, -f2,4 file.csv # extract fields 2 and 4
cut -c1-10 file.txt # extract characters 1-10
cut -d: -f1,6 /etc/passwd # username and shell
tr '[:lower:]' '[:upper:]' < file.txt # uppercase
tr -d '\r' < dos.txt # strip CR
tr -s ' ' < file.txt # squeeze multiple spaces
tr -dc '[:alnum:]\n' < file.txt # delete non-alphanumeric
echo "hello world" | tr ' ' '\n' # one word per line
fold -w 72 -s file.txt # wrap at 72 chars, break at spaces
fmt -w 72 essay.txt # reflow paragraphs to 72 char width
fmt -u file.txt # uniform spacing (one space after period)
mount | column -t
{ echo "NAME AGE CITY"; echo "Alice 30 NYC"; echo "Bob 25 LA"; } | column -t
With a custom delimiter:
cat /etc/passwd | column -t -s:
Output alignment only (no column separator):
ps aux | column -t | head
Fixed separator in output:
column -t -s, file.csv
Column with header and borders (newer util-linux):
column -t -N "Host,Port,Service" -s, services.csv
Multi-key sort: primary by field 3 numeric, secondary by field 1 alphabetic
sort -k3,3n -k1,1 file.txt
-k3,3n means sort on field 3 only (start,end with same number) numerically. Without the end field, sort uses the rest of the line.
Human-readable size sort (10M before 2G)
du -sh * | sort -h
Version sort (1.9 before 1.10)
ls v*.tar.gz | sort -V
Debug sort keys (shows what key was used per line)
sort -k2,2n --debug file.txt
Underlines the portion of each line used as the sort key. Invaluable for debugging unexpected order.
Limit RAM usage
sort -S 2G large.txt
Default is often 10% of RAM. Explicit -S avoids hitting swap.
Parallel sort
sort --parallel=8 large.txt
Stable sort (preserve original order of equal elements)
sort -s -k1,1 file.txt
Sort CSV by second field, ignoring header
(head -1 file.csv; tail -n +2 file.csv | sort -t, -k2,2) > sorted.csv
Fan output to multiple files and stdout
pipeline | tee file1.txt file2.txt | next_command
Fan-out to two different commands using process substitution
cat data.txt | tee >(gzip > data.gz) >(wc -l > count.txt) | grep "ERROR"
Both subshells receive every line. Stdout of tee continues to the next pipe stage.
Debug mid-pipeline without breaking the flow
cat data.txt | grep "foo" | tee /dev/stderr | awk '{print $2}' > result.txt
/dev/stderr lets you see the intermediate stream in the terminal while the pipe continues.
Log mid-pipeline with timestamp
pipeline | tee >(ts '%Y-%m-%d %H:%M:%S' >> pipeline.log) | next_stage
(ts from moreutils)
Duplicate to multiple named pipes
mkfifo /tmp/p1 /tmp/p2
tee /tmp/p1 /tmp/p2 < input.txt &
consumer1 < /tmp/p1 &
consumer2 < /tmp/p2 &
wait
Select / filter objects
jq '.[] | select(.status == "active")' users.json
jq '.[] | select(.age > 30 and .city == "NYC")' users.json
group_by and count
jq 'group_by(.status) | map({status: .[0].status, count: length})' data.json
to_entries: iterate over object keys
jq 'to_entries | map(.key + "=" + (.value | tostring)) | .[]' config.json
Useful for converting JSON objects to env-var format.
@csv and @tsv output
jq -r '.[] | [.name, .age, .city] | @csv' users.json
jq -r '.[] | [.name, .age, .city] | @tsv' users.json
reduce: fold over an array
jq 'reduce .[] as $x (0; . + $x.amount)' transactions.json
// for defaults (alternative operator)
jq '.[] | {name: .name, role: (.role // "user")}' users.json
If .role is null or missing, substitutes "user".
-s slurp: read multiple JSON docs into an array
cat *.json | jq -s '.'
cat *.json | jq -s 'map(select(.active)) | length'
Compact output + raw strings
jq -rc '.[] | .id' data.json # -r raw (no quotes), -c compact (no pretty print)
Update a field in place
jq '(.[] | select(.id == 42)).status = "done"' data.json
Build new objects from arrays
jq '[.[] | {(.id | tostring): .name}] | add' users.json
Extract nested paths
jq '[path(..| numbers)] ' data.json # all paths to numeric values
Timing breakdown with -w write-out
curl -o /dev/null -s -w "\
DNS: %{time_namelookup}s\n\
Connect: %{time_connect}s\n\
TLS: %{time_appconnect}s\n\
TTFB: %{time_starttransfer}s\n\
Total: %{time_total}s\n\
HTTP: %{http_code}\n\
Size: %{size_download} bytes\n\
" https://example.com
Check HTTP status in scripts
curl --fail -sS https://api.example.com/health || echo "DOWN"
--fail exits non-zero on 4xx/5xx. -sS is silent but shows errors.
Retry with backoff
curl --retry 5 --retry-delay 2 --retry-max-time 30 https://flaky.api/endpoint
Cookie jar (session persistence)
curl -c cookies.txt -b cookies.txt -d "user=admin&pass=secret" https://site.com/login
curl -b cookies.txt https://site.com/protected
Follow redirects and get final URL
curl -Ls -o /dev/null -w "%{url_effective}" https://short.url/abc
Download with resume
curl -C - -O https://example.com/big-file.iso
Multiple URLs in parallel (xargs)
cat urls.txt | xargs -P 10 -I{} curl -sO {}
POST JSON
curl -X POST -H "Content-Type: application/json" -d '{"key":"value"}' https://api/endpoint
Upload file
curl -F "file=@/path/to/file.csv" https://upload.example.com/
Port check (TCP)
nc -zv host 443 # verbose, exit after check
nc -z -w 2 host 8080 # 2s timeout
Banner grab
echo "" | nc -w 3 host 22 # SSH banner
echo "HEAD / HTTP/1.0\r\n\r\n" | nc -w 3 host 80
Simple file transfer
# receiver:
nc -l 9999 > received_file
# sender:
nc receiver_host 9999 < file_to_send
With pv for progress
pv file.tar.gz | nc receiver 9999
TCP proxy with mkfifo (bidirectional)
mkfifo /tmp/proxy_pipe
nc -l 8080 < /tmp/proxy_pipe | nc real_backend 80 > /tmp/proxy_pipe
Traffic to localhost:8080 is forwarded to real_backend:80.
One-shot HTTP server (serve a file)
{ echo -e "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\n"; cat file.txt; } | nc -l 8080
UDP listener
nc -u -l 5140 # listen on UDP 5140 (syslog)
Scan a range of ports
nc -zv host 20-1024 2>&1 | grep "succeeded"
{} + vs {} ; (batching vs per-file)
find . -name "*.log" -exec rm {} + # rm file1 file2 ... (one invocation)
find . -name "*.log" -exec rm {} \; # rm file1; rm file2; ... (one per file)
{} + is like xargs; faster for many files. {} \; is needed when the command cannot take multiple arguments.
Shell logic inside -exec
find . -name "*.txt" -exec sh -c 'wc -l "$1" >> /tmp/counts.txt' _ {} \;
The _ is $0 (script name placeholder); {} becomes $1.
Process multiple files with shell logic
find . -name "*.jpg" -print0 | xargs -0 -I{} sh -c '
base=$(basename "{}" .jpg)
convert "{}" -resize 800x /tmp/resized/${base}_sm.jpg
'
Chain conditions as AND
find . -name "*.py" -newer setup.py -exec grep -l "TODO" {} \;
Multiple predicates are AND by default. Use -o for OR, \( \) for grouping.
Delete empty directories
find . -type d -empty -delete
Find and compress files older than 30 days
find /var/log -name "*.log" -mtime +30 -exec gzip {} \;
Find files by size range
find . -size +1M -size -100M -type f
Exclude a directory
find . -path ./node_modules -prune -o -name "*.js" -print
Trace only specific syscall categories
strace -e trace=file command # open, stat, read, write on files
strace -e trace=network command # socket, connect, bind, send, recv
strace -e trace=memory command # mmap, mprotect, brk
strace -e trace=process command # fork, exec, wait, clone
Summary of syscall counts and time
strace -c -p PID # attach to running process, print summary on exit
strace -c command
Shows number of calls, errors, and total time per syscall.
Per-syscall timing
strace -T command 2>&1 | grep "open" # shows time in <0.000123> after each call
Resolve file descriptor paths
strace -yy command # annotates fds with full path in every syscall
Without -yy, you see read(4, ...). With -yy, you see read(4</var/log/app.log>, ...).
Follow child processes
strace -f command # trace forked children too
strace -ff -o /tmp/trace command # one file per pid: /tmp/trace.PID
Filter output
strace -e trace=openat command 2>&1 | grep -v "ENOENT" # hide missing files
Attach to existing process
strace -p $(pgrep nginx)
PID=$(pgrep myapp)
Full command line (null-separated args)
cat /proc/$PID/cmdline | tr '\0' ' '
Environment variables
cat /proc/$PID/environ | tr '\0' '\n'
cat /proc/$PID/environ | tr '\0' '\n' | grep PATH
Open file descriptors
ls -la /proc/$PID/fd # symlinks to open files, sockets, pipes
Memory map (which libraries are loaded)
cat /proc/$PID/maps
Shows address ranges, permissions, and file backing. Useful for detecting injected libraries or diagnosing memory layout.
Process status (state, memory, threads)
cat /proc/$PID/status
Contains VmRSS (resident set), VmSwap, Threads, State.
I/O statistics
cat /proc/$PID/io
rchar, wchar are bytes read/written. syscr, syscw are syscall counts. read_bytes, write_bytes are actual disk I/O.
Stack trace (kernel stack)
cat /proc/$PID/stack # kernel stack frames (requires CAP_SYS_PTRACE or root)
Limits
cat /proc/$PID/limits # ulimits currently in effect
Live CPU/memory for all processes
paste <(ps -eo pid,comm) <(cat /proc/*/status 2>/dev/null | grep VmRSS | awk '{print $2}') | sort -k3 -rn | head
coproc creates a background process with two-way pipes. Unlike cmd | read, you can write and read repeatedly.
Basic usage
coproc BC { bc -l; }
echo "3.14159 * 2" >&${BC[1]}
read result <&${BC[0]}
echo "Result: $result"
BC[1] is stdin of the coprocess; BC[0] is its stdout.
Named coprocess (bash 4+)
coproc CALC { python3 -c "import sys; [print(eval(l)) for l in sys.stdin]"; }
echo "2**10" >&${CALC[1]}
read answer <&${CALC[0]}
echo "$answer" # 1024
Interactive reuse across loop iterations
coproc DB { sqlite3 mydb.sqlite; }
for table in users orders products; do
echo ".schema $table" >&${DB[1]}
read schema <&${DB[0]}
echo "$table: $schema"
done
echo ".quit" >&${DB[1]}
Difference from stdin pipe
With a regular pipe you get one write/one read: echo "input" | cmd. With coproc you maintain a persistent session and can write multiple times without restarting the process. This matters when process startup is expensive (interpreters, DB connections, TLS negotiation).
Capture stderr separately
coproc CMD { my_tool 2>/tmp/cmd_errors; }
Filter down to relevant records, transform each record, then aggregate.
awk -F, '$3 == "US"' orders.csv \
| awk -F, '{print $5}' \
| awk '{sum += $1} END {print sum}'
cat access.log | tee \
>(awk '{print $1}' | sort | uniq -c | sort -rn | head > top_ips.txt) \
>(awk '{print $7}' | sort | uniq -c | sort -rn | head > top_paths.txt) \
> /dev/null
Compare outputs from two command variants without temp files.
diff <(curl -s https://api.example.com/v1/users | jq -r '.[] | .id' | sort) \
<(curl -s https://api.example.com/v2/users | jq -r '.[] | .id' | sort)
# Which hosts are in inventory but not responding to ping?
comm -23 \
<(sort inventory.txt) \
<(cat inventory.txt | parallel -j50 "ping -c1 -W1 {} &>/dev/null && echo {}" | sort)
find . -name "*.csv" -print0 \
| xargs -0 -I{} sh -c '
lines=$(wc -l < "{}")
echo "{}: $lines lines"
'
cat image_ids.txt | parallel --keep-order \
'curl -s "https://api/image/{}" | jq -r ".url"' \
> ordered_urls.txt
ps aux \
| awk 'NR==1 || $3 > 1.0 {print $1, $2, $3, $4, $11}' \
| column -t
curl -s "https://api.github.com/repos/torvalds/linux/releases" \
| jq -r '.[] | [.tag_name, .published_at, (.assets | length | tostring)] | @tsv' \
| column -t -s $'\t'
watch -n1 'ss -tnp | awk "NR>1 {print \$1,\$4,\$5,\$6}" | column -t'
Or with continuous output:
tail -f /var/log/nginx/access.log \
| awk '{print $1, $7, $9}' \
| grep --line-buffered -v "200" \
| ts '%H:%M:%S'
# Extract a JSON block from a config, update a field, put it back
value=$(jq '.database.pool_size' config.json)
new_value=$((value * 2))
tmp=$(mktemp)
jq --argjson n "$new_value" '.database.pool_size = $n' config.json > "$tmp"
mv "$tmp" config.json
For INI-style configs:
sed -i 's/^max_connections\s*=.*/max_connections = 200/' /etc/myapp/app.conf
| Tool | Key flag | Use |
|---|---|---|
| awk | !seen[$0]++ |
ordered dedup |
| awk | NF |
field count filter |
| sed | s|/|:|g |
alternate delimiter |
| sed | /a/,/b/ |
range address |
| xargs | -0 -P N -I{} |
null-safe parallel |
| parallel | --keep-order {.} |
ordered + extension strip |
| comm | -12 -23 -13 |
intersection / diff / diff |
| join | -a 1 -v 1 |
outer / anti join |
| sort | -k3,3n -h -V -s --debug |
multi-key / human / version |
| tee | >(cmd) |
fan-out via process sub |
| jq | select group_by reduce // |
filter / aggregate / default |
| curl | -w "%{time_total}" --fail --retry |
timing / scripts / retry |
| nc | -zv -l |
port check / listen |
| find | -exec {} + -print0 |
batch / null-safe |
| strace | -e trace=file -c -T -yy -f |
category / summary / timing / fds |
| /proc | cmdline environ fd maps io |
process introspection |
| coproc | ${NAME[0]} ${NAME[1]} |
bidirectional IPC |