Skip to content

Latest commit

 

History

History
469 lines (433 loc) · 11.4 KB

hol-grep-awk-sed.md

File metadata and controls

469 lines (433 loc) · 11.4 KB
Raw

grep (global regular expression print)

A regular expression may be followed by one of several repetition operators:

  • ? The preceding item is optional and matched at most once.
  • * The preceding item will be matched zero or more times.
  • + The preceding item will be matched one or more times.
  • {n} The preceding item is matched exactly n times.
  • {n,} The preceding item is matched n or more times.
  • {,m} The preceding item is matched at most m times.
  • {n,m} The preceding item is matched at least n times, but not more than m times.

Practice Sets

  1. Create Demo File
    cat > demo.txt
    THIS LINE IS THE 1ST UPPER CASE LINE IN THIS FILE.
    this line is the 1st lower case line    in this file.
    This Line Has All Its First Character   Of The Word With Upper Case.

    Two lines above this line is empty.
    And this is the last line.
  1. Search "literal string"
    grep "this" demo_file
  1. Check the given string in multiple files
    grep "this" *
  1. Case insensitive search
    grep -i "this" demo.txt
  1. Check full word, not sub-strings
    grep -iw "is" demo.txt
  1. Display lines before/after/around the match.
    1. N lines after match 
      grep -A 3 -i "example" demo.txt
    2. N lines before match 
      grep -B 2 "single WORD" demo.txt
    3. N lines around match 
      grep -C 2 "Example" demo.txt
  2. Highlight the search using GREP_OPTIONS 
    export GREP_OPTIONS='--color=auto' GREP_COLOR='100;8'
  3. Search recursively 
    grep -r "error" *
  4. Invert match 
    grep -v "go" demo.txt
  5. display the lines which does not matches all the given pattern. 
    $ cat test-file.txt
a
b
c
d

$ grep -v -e "a" -e "b" -e "c"  test-file.txt
d
  6. Counting the number of matches 
    grep -c "go" demo.txt
grep -v -c "this" demo.txt
  7. disploy the file names 
    grep -l "this" demo.txt
  8. Show only the matched string 
    grep -o "is.*line" demo.txt
  9. Show the position of match in the line 
    cat > temp-file.txt
12345
12345

grep -o -b "3" temp-file.txt

Note: The output of the grep command above is not the position in the line, it is byte offset of the whole file. 15. Show line number while displaying the output using grep -n

    grep -n "go" demo.txt

Stream Editor SED Examples

The three basic operations supported by the sed command are:

  1. Insertion
  2. Deletion
  3. Substitution (Find and replace)

Create sample file

cat > sed_example.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
unix linux which one you choose.
unix is easy to learn.unix is a multiuser os.Learn unix .unix is a powerful.

cat > sed_examle2.txt
Hello World!
This is a test file
Computer Science is fun!
  1. find and replace 
    sed 's/Hello/Goodbye/' testfile
  2. In-place file editing Without backup 
    cat > fruits.txt
banana
papaya
mango
    sed -i 's/an/AN/g' fruits.txt
cat fruits.txt
    With Backup 
    cat > colors.txt
deep blue
light orange
blue delight
    sed -i.bkp 's/blue/green/' colors.txt
    cat colors.txt.bkp
    Multiple files 
    cat > f1.txt
have a nice day
bad morning

cat > f2.txt
worse than ever
too bad
    sed -i.bkp 's/bad/good/' f1.txt f2.txt
    Prefix backup name 
    sed -i'bkp.*' 's/green/yellow/' colors.txt
    Place backups in a different directory 
    mkdir backups
sed -i'backups/*' 's/good/nice/' f1.txt f2.txt
  3. Reading Lines 
    cat > lyrics.txt
Thundercats are on the move
Thundercats are loose
Feel the magic, hear the roar
Thundercats are loose
Thunder, thunder, thunder, Thundercats
Thunder, thunder, thunder, Thundercats
Thunder, thunder, thunder, Thundercats
Thunder, thunder, thunder, Thundercats!
Thundercats!
    1. Printing only lines 5-8 of lyrics.txt
    sed -n '10p' lyrics.txt # Print only one line
sed -n '5,8p' lyrics.txt
sed -n '5,+3p' lyrics.txt
    1. Deleting lines
    sed '5,8d' lyrics.txt
sed '5,+3d' lyrics.txt
    1. delete lines that contain specific patterns
    sed '/thunder/ d' lyrics.txt
    1. run multiple commands at the same time
    sed -e '/thunder/ d' -e '/magic/ d' lyrics.txt  
    sed '
/thunder/ d;
/magic/ d
' lyrics.txt
    sed '/thunder/ d; /magic/ d' lyrics.txt
    1. display only the lines we changed
    sed -n '
s/thunder/lightning/gp;
s/Thunder,/Lightning,/gp
' lyrics.txt

Real world journey

https://github.com/Anant/example-introduction-to-sed/blob/main/spacecraft_journey_catalog.csv

```bash
sed 's/^,/Missing Summary,/' spacecraft_journey_catalog.csv > updated_items.csv
```
```bash
sed '/^,/ d' spacecraft_journey_catalog.csv > removed_items.csv
```
  1. append, change, insert Basic Usage
    • a appends the given string after the end of line for each matching address
    • c changes the entire matching address contents to the given string
    • i inserts the given string before the start of line for each matching address
        cat > file.txt
    # start
    This line is between the start and end keywords.
    Hello foo!!!
    Hello bar
    # end

    # End of file.txt

    sed '10p;5i\"INSERTED BEFORE LINE 5" file.txt
  2. replacement on all lines except line 5 
    sed '5!/s/foo/bar/' file.txt
  3. Remove comments between lines starting with these two keywords. Empty lines will be put there instead 
    sed -E '/start/,/end/ s/#.*//' file.txt
  4. Delete comments starting with # (no empty lines left behind) 
    sed -E '/^#/d' f1
  5. Insert an empty line after pattern (after each line containing comment in this case) 
    sed '/^#/G' file.txt
  6. View lines minus lines between line starting with pattern and end of file 
    sed  '/start/,$ d' file.txt
  7. View lines except lines between line starting with pattern and line ending with pattern 
    sed -rn '/start/,/end/ !p' file.txt
  8. Print until you encounter pattern then quit 
    sed  '/start/q' file.txt
  9. Insert contents of file after a certain line 
    sed  '5 r newfile.txt' file.txt
  10. Append text after lines containing regex (AFTER FOO) 
    sed '/foo/a\AFTER FOO' file.txt
  11. Insert text after lines containing regex (BEFORE FOO) 
    sed '/foo/i\BEFORE FOO' file.txt
  12. Change line containing regex match 
    sed '/foo/c\FOO IS CHANGED' file.txt

AWK Example

Example files

  1. Input record separator

    printf 'this,is\na,sample,text' | awk -v RS=, '{print NR ")", $0}'

  2. String anchor and sub built in function

    printf 'spared no one\npar\nspar\n' | awk '{sub(/^spar$/, "123")} 1'

  3. OFS

    echo 'goal:amazing:whistle:kwality' | awk -F: -v OFS=: '{print $2, $NF}'
echo 'goal:amazing:whistle:kwality' | awk 'BEGIN{FS=OFS=":"} {print $2, $NF}'

  4. Changing field content

    echo 'goal:amazing:whistle:kwality' | awk -F: -v OFS=: '{$2 = 42} 1'
echo 'goal:amazing:whistle:kwality' | awk -F: -v OFS=, '{$2 = 42} 1'

  5. assign a field value to itself to force the reconstruction of $0

    echo 'Sample123string42with777numbers' | awk -F'[0-9]+' -v OFS=, '1'
echo 'Sample123string42with777numbers' | awk -F'[0-9]+' -v OFS=, '{$1=$1} 1'

  6. In place file editing

    1. Without backup

      cat > greet.txt
Hi there
Have a nice day
Good bye
awk -i inplace '{print NR ". " $0}' greet.txt
      cat > f1.txt
I ate 3 apples
cat > f2.txt
I bought two balls and 3 bats

awk -i inplace '{gsub(/\<3\>/, "three")} 1' f1.txt f2.txt

    2. With backup

      cat > f3.txt
 Name    Physics  Maths
Moe  76  82
Raj  56  64
awk -i inplace -v inplace::suffix='.bkp' -v OFS=, '{$1=$1} 1' f3.txt

  7. Using shell variables

    s='cake'
awk -v word="$s" '$2==word' table.txt

  8. Shell environment

    awk 'BEGIN{print ENVIRON["HOME"]}'

  9. Control Structures

    1. if-else 
      awk '/^b/{print; if($NF>0) print "------"}' table.txt
awk '/^b/{print; if($NF>0) print "------"; else print "======"}' table.txt
    2. loops 
      awk 'BEGIN{for(i=2; i<7; i+=2) print i}'
awk -v OFS=, '{for(i=1; i<=NF; i++) if($i ~ /^[bm]/) $i="["$i"]"} 1' table.txt
awk 'NR>1{d[$1]+=$3; c[$1]++} END{for(k in d) print k, d[k]/c[k]}' marks.txt

  10. Built-in functions

    1. length 
      awk 'BEGIN{print length("road"); print length(123456)}'
awk 'length($1) < 6' table.txt
    2. split 
      s='Joe,1996-10-25,64,78'
echo "$s" | awk -F, '{split($2, d, "-"); print $1 " was born in " d[1]}'
s='air,water,12:42:3'
echo "$s" | awk -F, '{n=split($NF, a, ":");
               for(i=1; i<=n; i++) print $1, $2, a[i]}'
    3. substr 
      echo 'abcdefghij' | awk '{print substr($0, 1, 5)}'
echo 'abcdefghij' | awk -v OFS=: '{print substr($0, 2, 3), substr($0, 6, 3)}'
    4. match 
      s='051 035 154 12 26 98234 3'
echo "$s" | awk 'match($0, /[0-9]{4,}/){print substr($0, RSTART, RLENGTH)}'

Example with BEGIN and END

awk 'BEGIN {
    sum = 0
}

{
    # Process each line
    for (i = 1; i <= NF; i++) {
        # Assuming the fields are numbers
        sum += $i
    }
}

END {
    # Actions to be performed after processing all lines
    print "Sum of Numbers:", sum
}' numbers.txt
awk 'BEGIN {
    # Initialize variables and categories
    count_low = 0
    count_medium = 0
    count_high = 0
}

{
    # Process each line
    for (i = 1; i <= NF; i++) {
        # Assuming the fields are numbers
        value = $i

        # Categorize into different ranges
        if (value < 50) {
            count_low++
        } else if (value >= 50 && value < 100) {
            count_medium++
        } else {
            count_high++
        }
    }
}

END {
    # Actions to be performed after processing all lines
    print "Number of values in different ranges:"
    print "Low (less than 50):", count_low
    print "Medium (50 to 99):", count_medium
    print "High (100 and above):", count_high
}' numbers.txt

cat > number.txt
25 72 48 95 102
60 35 80 110 45
90 55 75 40 105

Let's check one awk game

https://github.com/TheMozg/awk-raycaster/tree/master
gawk -f awkaster.awk