🐧 Mastering Linux Pipelines: The Complete Guide to Text Processing with sed, awk, grep and more

🐧 Mastering Linux Pipelines: The Complete Guide to Text Processing with sed, awk, grep and more

In the world of Linux, text is data. From configuration files to logs and automation scripts, mastering tools like |, sed, awk, and grep allows you to transform, analyze, and automate tasks — all from the command line, often with a single line of code.

📘 1. The Power of the Pipeline (|)

The pipeline symbol (|) connects commands together, sending the standard output (stdout) of one command as the standard input (stdin) of another.

# Count active login sessions
who | wc -l

# Find and rank kernel error messages
dmesg | grep "error" | sort | uniq -c | sort -nr | head

🧱 2. Using cat + EOF to Create Multi-line Files

The EOF (End Of File) syntax is a Here Document, letting you generate configuration files or scripts inline.

cat < /etc/nginx/conf.d/blog.conf
server {
    listen 80;
    server_name blog.wwfandy.com;
    root /var/www/html;
}
EOF

You can also combine it with tee to write system files safely:

cat </dev/null
Welcome to WWFandy Linux Server
Today is $(date)
EOF

🧩 3. grep — The Text Search Engine

grep "root" /etc/passwd             # Search for keyword
grep -i "server" nginx.conf         # Case-insensitive
grep -n "error" /var/log/syslog     # Show line numbers
grep -r "Listen" /etc/httpd/        # Recursive search

✂️ 4. cut — Extracting Fields and Characters

cut -d: -f1 /etc/passwd             # Extract usernames
echo "WWFandyBlog" | cut -c1-8

⚙️ 5. awk — Field Processing and Reporting

awk is ideal for structured data analysis.

awk -F: '{printf "%-15s → %s\n",$1,$7}' /etc/passwd
grep "ERROR" /var/log/syslog | awk '{count++} END {print "Error Count:", count}'

🔧 6. sed — Search and Replace at Scale

sed is perfect for automatic editing and substitutions.

sed -i 's/Listen 80/Listen 8080/' /etc/httpd/conf/httpd.conf
echo "Welcome to Linux" | sed 's/Linux/WWFandy Blog/'

📊 7. wc — Counting Lines, Words, and Characters

wc -l file.txt   # Count lines
wc -w file.txt   # Count words
wc -c file.txt   # Count characters
grep "http" access.log | wc -l

📑 8. sort + uniq — Sorting and Deduplication

cat list.txt | sort | uniq
cat list.txt | sort | uniq -c | sort -nr | head

🔡 9. tr — Character Transformation and Cleanup

echo "Hello World" | tr 'a-z' 'A-Z'     # Convert to uppercase
cat file.txt | tr -d '\r'               # Remove CR (Windows line endings)
cat file.txt | tr -s ' '                # Merge multiple spaces

📎 10. paste — Combine Files Horizontally

paste names.txt phones.txt > contacts.txt

🚀 11. xargs — Execute Commands in Batches

find . -name "*.tmp" | xargs rm -f
grep -rl "localhost" . | xargs sed -i 's/localhost/blog.wwfandy.com/g'

🧭 12. tee — Write and Display Simultaneously

df -h | tee ~/disk_report.txt | grep "root"
echo "Backup done at $(date)" | tee -a /var/log/backup.log

🧮 13. End-to-End Example: Generate and Analyze Configs

# 1️⃣ Create a config file
cat < /tmp/service.conf
service=nginx
port=80
status=active
EOF

# 2️⃣ Modify content
sed -i 's/80/8080/' /tmp/service.conf

# 3️⃣ Pretty-print output
awk -F= '{printf "%-10s : %s\n",$1,$2}' /tmp/service.conf

# 4️⃣ Count lines
wc -l /tmp/service.conf

📘 Conclusion

These tools together form the Linux text-processing pipeline ecosystem: grep searches, cut extracts, sed edits, awk analyzes, tee writes, and xargs automates. By mastering them, you can automate system tasks, create reports, and transform data — all in a single command line.

---

🔗 Related Articles

• Linux sed Basics: Text Replacement and Editing Examples
• Linux tail and Stream Processing Techniques
• Linux Proxy Server Setup Guide (Squid Configuration)

— WWFandy · Linux Automation Notes