103.7 Using Regular Expressions

Candidates should be able to manipulate files and text data using regular expressions. This objective includes creating simple regular expressions containing several notational elements. It also includes using regular expression tools to perform searches through a filesystem or file content.

Key Knowledge Areas

Create simple regular expressions containing several notational elements.
Use regular expression tools to perform searches through a filesystem or file content.

Overview

Finding a word or multiple words in a text is achieved using grep, fgrep or egrep. The keywords used during a search are a combination of letters called regular expressions. Regular expressions are recognised by many other applications such as sed, and vi.

Regular Expressions

Traditional Regular Expressions (regex)

A regular expression is a sequence of characters (or atoms) used to match a pattern. Characters are either constants (treated literally) or metacharacters.

Table1: Main metacharacters

Characters	Search Match
\<KEY	Words beginning with ‘KEY’
WORD\>	Words ending with ‘WORD’
^	Beginning of a line
$	End of a line
[ Range ]	Range of ASCII characters enclosed
[^c ]	Not the character ‘c’
\[	Interpret character ‘[‘ literally
“ca*t”	Strings containing ‘c’ followed by no 'a' or any number of the letter 'a' followed by a 't'
“.”	Match any single character

Extended regex:

The main eregex’s are: +,?,() and |

Table2: List of main eregex

Characters	Search Match
"A1\|A2\|A3"	Strings containing ‘A1’ or ‘A2’ or ‘A3’
"ca+t"	Strings containing a 'ca' followed by any number of the letter 'a' followed by a 't'
"ca?t"	Strings containing ‘c’ followed by no 'a' or exactly one 'a' followed by a 't'

The grep family

The grep utility supports regular expressions regex such as those listed in Table1.

Working with basic grep

Syntax for grep:

grep PATTERN FILE

Options for grep include:

grep	Main Options
-c	count the number of lines matching PATTERN
-f	obtain PATTERN from a file
-i	ignore case sensitivity
-n	Include the line number of matching lines
-v	output all lines except those containing PATTERN
-w	Select lines only if the pattern matches a whole word.

For example list all non blank lines in /etc/lilo.conf:

$ grep –v “^$” /etc/lilo.conf

egrep

The egrep tool supports extended regular expressions eregex such as those listed in Table2.

The egrep utility will handle any modern regular expressions. It can also search for several keywords if they are entered at the command line, separated by the vertical bar character.

For example:

$ egrep 'linux|^image' /etc/lilo.conf

fgrep

fgrep stands for fast grep and fgrep interprets strings literally (no regex or eregex support). The fgrep utility does not recognise the special meaning of the regular expressions.

For example:

$ fgrep 'cat*' FILE

will only match words containing ‘cat*’. The main improvement came from fgrep’s ability to search from a list of keywords entered line by line in a file, say LIST. The syntax would be

$ fgrep –f LIST FILE

The Stream Editor - sed

sed performs automatic, non-interactive editing of files. It is often used in scripts to search and replace patterns in text. It supports most regular expressions.

Syntax for sed:

sed [options] 'command' [INPUTFILE]

The input file is optional since sed also works on file redirections and pipes. Here are a few examples assuming we are working on a file called MODIF.

Delete all commented lines:

$ sed '/^#/ d ' MODIF

Notice that the search pattern is between the double slashes.

Substitute /dev/hda1 by /dev/sdb3:

$ sed 's/\/dev\/hda1/\/dev\/sdb3/g' MODIF

The s in the command stands for ‘substitute’. The g stands for “globally” and forces the substitution to take place throughout each line. You can also specify which line numbers the substitutions should occur on, either using line numbers or regular expression match.

If the line contains the keyword KEY then substitute ‘:’ with ‘;’ globally:

$ sed '/KEY/ s/:/;/g' MODIF

More Advanced sed

You can issue several commands each starting with –e at the command line. For example, (1) delete all blank lines then (2) substitute ‘OLD’ by ‘NEW’ in the file MODIF

$ sed –e '/^$/ d’ -e ‘s/OLD/NEW/g' MODIF

These commands can also be written to a file, say COMMANDS. Then each line is interpreted as a new command to execute (no quotes are needed).

An example COMMANDS file

1 s/old/new/

/keyword/ s/old/new/g

23,25 d

The syntax to use this COMMANDS file is:

sed -f COMMANDS MODIF

This is much more compact than a very long command line !

Summary of options for sed

Command line flags

-e Execute the following command

-f Read commands from a file

-n Do not printout unedited lines

sed commands

d Delete an entire line

r Read a file and append to output

s Substitute

w Write output to a file

Used files, terms and utilities:

grep
egrep
fgrep
sed
regex(7)

Previous Chapter | Next Chapter