LPI Linux Certification/LPIC1 Exam 101/GNU & Unix Commands

Section Overview

1.103.1 Work on the command line
1.103.2 Process text streams using filters
1.103.3 Perform basic file management
1.103.4 Use streams, pipes, and redirects
1.103.5 Create, monitor, and kill processes
1.103.6 Modify process execution priorities
1.103.7 Search text files using regular expressions
1.103.8 Perform basic file editing operations using VI

Unix and Linux

Unix and Linux

Overview

Unix is the original Internet Operating System (NFS, TCP/IP, RPC, ...). Linux is a Unix clone written from scratch by Linus Torvalds with assistance from a world wide team of developers across the Net. Properties of both systems are:

Portable, multi-user, multiprocessing
Device-independent input/output
Shared resources, high performance and highly secure
Line commands
GUI

Linux aims toward POSIX compliance and supports also other standard API's such as BSD and SVR4. Linux has all the features you would expect in a modern Unix, including true multitasking/multiple-users, virtual memory, shared libraries, demand loading, shared copy-on-write executables, proper memory management and TCP/IP networking.

More and more companies and organizations are selecting Linux because of its performance, low cost, and royalty free license.

Unix and Linux

Unix is the original Internet Operating System written in 1970 developed at A&T Bell Labs. Distributed freely to government and universities. Different entities did support different distributions.

System V
BSD
SCO

Most varieties of Unix carry copyright licenses.

Linux was released in 1991 by Linus Torvalds. The model of Linux licence is copy left from GPL. Linux implement most of the System V and BSD Unix commands. Use the man pages to determine the version.

Logging In and Out

The logging In session is the first step to access a Linux system. You need a login ID and a password to be able to start using the system. The login ID root is the super user ID that has all privileges, normally assigned to the System Administrator. When a log in session is successful it will set some environment variables and start the shell that has been assigned to the login ID account.

Examples of variables set at log in time: $HOME, $SHELL, $PATH, and more. By changing the file /etc/login.defs the administrator can customize some login parameters and variables for all the users.

Examples of some of them:

UID_MIN 500
UID_MAX 60000
UMASK 022
ENV_PATH /usr/local/bin:/usr/bin:/bin

To logout use the exit or logout command.

Password

Nobody can see your password, not even the administrator. To change passwords for users or groups, use passwd.

passwd [options]

Common options:

-l, -u : Lock and unlock an account.
-g [| r]: Used for group.
-g set a password for group and -r removes it.
-x|n : Maximum and minimum number of days a password remain valid.

As a general rule a password must:

Have a minimum length
Not contain any English language words
Contain non-alphanumeric characters
Contain a mixture of uppercase and lower case letters.

User information

For every user account, there is a line defined in the file /etc/passwd. The encrypted password is stored in /etc/shadow. The format of /etc/passwd is:

root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/bin/bash
daemon:x:2:2:Daemon:/sbin:/bin/bash
yann:x:500:100:Yann Forget:/home/yann:/bin/bash

The x field is the old location for the password.

The format of /etc/shadow is:

root:IMXweyiV816gg:11996:0:10000::::
bin:!*:8902:0:10000::::
daemon:*:8902:0:10000::::
yann:GoIM8j1S.IuTY:11996:0:99999:7:::

The * for the encrypted password means there is no password defined yet. The ! Before the encrypted password means the account is locked.

Group information

For every group of users there is a line in file /etc/group. The encrypted password is stored in /etc/gshadow. The format of /etc/group is:

root:x:0:root
bin:x:1:root,bin,daemon
daemon:x:2:
video:x:33:yann

The format of /etc/gshadow is:

root:*:root:root
bin:*:root:root
daemon:*:root:root
video:*:root:root,yann

Passwords for group are not often implemented since you have to pass to members of a group the password.

Misc. Commands

To get information on a logging In session, use id.

id [options] [username]

Common options: -g : Print only the group ID. -u : Print only the user ID.

Example:

$ id
uid=0(root) gid=0(root) groups=0(root), 1(bin), 14(uucp), 15(shadow),16(dialout)

To run a shell with another user ID and group ID, use su.

su [options] [username]

Common options: -s : Select another shell.

Example:

$ su toto
passwd:

Shell

A Shell is a command line interpreter and:

Prompts for commands.
Interprets command which can include:
I/O redirection, pipes, filename generation, etc.
Executes commands.
Executes scripting language.

There are several popular shells:

sh: Bourne shell, the standard scripting shell.
csh: Berkeley shell, job control, command history, alias.
ksh: Korn shell, Bourne syntax with Berkeley features.
bash: Bourne Again shell, free, Korn-like shell.
t csh: Free, Berkeley clone with many extensions.

Most shells let you customize your prompt and environment. For bash the prompt variable is PS1. For bash the file that can be used for the customization of the environment are:

.bashrc Executed for each new bash session
.bash_profile Executed at logging In time
.bash_logout Executed at logging Out time

Exercises

Exercises results

Process text streams using filters

Overview

Description: Candidates should should be able to apply filters to text streams. Tasks include sending text files and output streams through text utility filters to modify the output, and using standard UNIX commands found in the GNU textutils package.

Key files terms and utilities include:
cat cut expand fmt head join nl od paste pr sed sort split tac tail tr unexpand uniq wc

Pattern matching and wildcards

Wildcard is a pattern matching mechanism for filenames. The purpose of wildcard is to increase productivity: Locate files that you don't fully remember. Locate files that have something in common. Work with a group of files rather than individual.

The shell interprets these special characters:

! @ # $ % ^ & * ( ) { } [ ] | \ ; ~ ' " ` ?

The characters used for wildcard are:

?  *  [  ]  ~

If you use the wildcard characters the shell will try to generate a file from them. Try the following:

echo all files *

Special wildcard characters

? match any one character.
* Any string
[abcfghz] One char set
[a-z] One char in range
[!x-z] Not in set
~ Home directory
~user User home directory

Examples:

? One character filenames only
[aA]??? Four characters, starting with a or A.
~toto pathname of toto home directory
[!0-9]* All string not starting with a number.

What about these commands?

ls [a-z][A-Z]??.[uk]
ls big*
ls a???a
ls ??*

Shell and wildcards

A shell command line can be a simple command or more complex.

ls -l [fF]*
ls *.c | more
ls -l [a-s]* | mail `users`

The first event in the shell is to interpret wildcards. Only the shell interprets unquoted wildcards.

Quoting and Comments

Quoting

Do quote to prevent the shell to interpret the special characters and to transform multiple words into one shell word.

'string' Nearly everything within the quote is literal:

echo 'He did it, "Why?"'
echo 'Because "#@&^:-)"'
echo '$VAR='Me

"string" Like 'string' but interprets $, \, !:

echo "What's happening?"
echo "I don't know but check this $ANSWER"

The backslash \ treats the following character as literal:

echo \$VAR=Me
echo What\'s happening\?

How could we display the backslash? With the following line:

echo \\

Comments

You can add comments in a command line or a script. Use the character #. A white space must immediately precede #.

Examples:

echo $HOME # Print my Home directory
echo "### PASSED ###" # Only this part is a comment
echo The key h#, not g was pressed.

Commands

cat, tac: Concatenate files and print on the standard output.
head, tail: Output the first and last part of files.
nl: Number lines of files.
wc: Print the number of bytes, words, and lines in files.
cut: Remove sections from each line of files.
tr: Translate or delete character.
expand, unexpand: Convert tabs to spaces and space to tabs.
paste: Merge lines of files.
join: Join lines of two files on a common field.
uniq: Remove duplicate lines from a sorted file.
split: Split a file into pieces.
fmt: Simple optimal text formatter.
pr: Convert text files for printing.
sort: Sort lines of text files.
od: Dump files in octal and other formats.

Concatenate files

To concatenate files, use cat.

cat [options] [files...]
tac [options] [files...]

The results are displayed to the standard output.

Common options:

-s: never more than one single blank line.
-n: number all output line.

Examples:

cat file  # Display file to the stdout.
cat chapter* # Display all chapters to stdout.
cat -n -s file # Display file with line number with single blank line.

To concatenate files in reverse order, use tac.

View the beginning and the end of a file

To view only few lines at the beginning or at the end of a file, use head or tail.

head [options] [files...]
tail [options] [files...]

The results are displayed to the standard output.

Common options:

-n: number of lines to be displayed. (head and tail)
-c: number of bytes to be displayed. (head and tail)
-f: append output. (tail)
-s #: iteration for new data every # sec. (tail)

Examples:

head file # Display the first 10 lines of file.
head -n 2 file # Display the first 2 lines of file.
tail -c 10 file # Display the last 10 bytes of file.
tail -f -s 1 /var/log/messages 
Display the last 10 lines of messages, block and check for new data every second.

Numbering file lines

To add the line number to a file, use nl.

nl [options] [files...]

The results are displayed to the standard output.

Common options:

-i #: increment line number by #.
-b: numbering style:
   a: number all lines
   t: non empty line
   n: number no line
-n: numbering format:
   rz: right justified
   ln: left justified.

Examples:

nl file # Add the line number in each line in the file.
nl -b t -n rz file # Add the line number to each non-empty line with zero-completed format.

Counting items in a file

To print the number of lines, words and characters of a file, use wc.

wc [options] [files...]

The results are displayed to the standard output.

Common options:

-c: the byte size.
-m: the number of character.
-w: the number of word.
-l: the number of line.
-L: length of the longest line.

Examples:

wc *.[ch] # Display the number of lines, words, and characters for all files .c or .h.
wc -L file # Display the size of the longest line.
wc -w file # Display the number of words.

Cutting fields in files

To remove sections from each line of files, use cut.

cut [options] [files...]

The results are displayed to the standard output.

Common options:

-b #: Extract the byte at position #.
-f #: Extract the field number #.

Examples:

cut -b 4 file # Extract and display the 4th byte of each line of file. 
cut -b 4,7 file # Extract and display the 4th and 7th byte of each line.
cut -b -2,4-6, 20- file
# Extract character 1 and 2, 4 to 6 and 20 to the end of the line for each line of file.
cut -f 1,3 -d: /etc/passwd # Extract the username and ID of each line in /etc/passwd.

The default delimiter is TAB but can be specified with -d.

Characters conversion

To translate the stdin to stdout, use tr.

tr [options] SET1 SET2

Common options: -d: delete character in SET1. -s: replace sequence of character in SET1 by one.

Examples:

tr ‘a‘ 'A'  # Translate lower a with A
tr ‘[A-Z]’ ‘[a-z]’ # Translate uppercase to lowercase
tr -d ‘ ‘ # Delete all spaces from file

To convert tabs to spaces, use expand and to convert spaces to tabs, use unexpand.

expand  file
unexpand file

Lines manipulation

To paste multiple lines of files, use paste.

paste [options] [files...]

Common options:

-d #: delimiter: Use # for the delimiter.
-s: serial: paste one file at the time.

Examples:

paste f1 f2 # Display line of f1 followed by f2.
paste -d: file1 file2 # Use ':' for the delimiter.

To join multiple lines of files, use join.

join file1 file2

To remove duplicated line, use uniq.

uniq [options] [files...]

Common options:

-d: only print duplicated lines.
-u: only print unique lines.

Examples:

uniq -cd file # Display the number of duplicated line.

Splitting files

To split big files, use split.

split [options] file

Common options:

-l #: split every # lines.
-b #: split file in bytes or b for 512 bytes, k for 1Kbytes, m for 1 Mbytes.

Examples:

split -l 25 file  # Split file into 25-line files.
split -b 512 file # Split file into 512-byte files.
split -b 2b file  # Split file into 2*512-byte files.

Formatting for printing

To format a file, use fmt.

fmt [options] [files...]

Common options: -w #: maximum line width.

Examples:

$ fmt -w 35 file # Display 35-character lines width.

To format a file for a printer, use pr.

pr [options] [files...]

Common options: -d: double space.

Examples:

$ pr -d file # Format file with double-space.

Sort lines of text files

To sort the lines of the names files, use sort.

sort [options] file

The results are displayed to the standard output.

Common options:

-r : Reverse
-f : Ignore case
-n : Numeric
-o file: Redirect output to file
-u : No duplicate records
-t; : Use ';' as delimiter, rather than tab or space.

Examples:

sort file -r
sort file -ro result

Binary file dump

To dump a binary file, use od.

od [options] file

The results are displayed to the standard output and start with an offset address in octal.

Common options:

-c: each byte as character
-x: 2-byte in hex
-d: 2-byte in decimal
-X: 4-byte in hex.
-D: 4-byte in decimal.

Examples:

$ od -cx /bin/ls
0000000 177   E   L   F 001 001 001  \0  \0  \0  \0  \0  \0  \0  \0  \0
       457f 464c 0101 0001 0000 0000 0000 0000
0000020 002  \0 003  \0 001  \0  \0  \0     224 004  \b   4  \0  \0  \0
       0002 0003 0001 0000 9420 0804 0034 0000
0000040   °   ²  \0  \0  \0  \0  \0  \0   4  \0      \0 006  \0   (  \0
       b2b0 0000 0000 0000 0034 0020 0006 0028
0000060 032  \0 031  \0 006  \0  \0  \0   4  \0  \0  \0   4 200 004  \b
       001a 0019 0006 0000 0034 0000 8034 0804

Exercises

Use wildcard characters and list all filenames that contain 'x' followed by 'in' in the /etc directory.
Use wildcard characters and list all filenames that start with any character between 'a' and 'e' that have at least two more characters and do not end with a number.
Use wildcard characters and list all filenames of exactly 4 characters and all filenames starting with an uppercase letter. Do not descend into any directory found.
Use wildcard characters and list all files that contains 'sh' in /bin.
Display you environment variable HOME preceded by the string "$HOME value is:"
Display the contents of $SHELL with two asterisk characters before and after it.
How would you display the following string of characters as is with echo using double quote and \.
- @ # $ % ^ & * ( ) ' " \
Compose echo commands to display the following two strings:
- That's what he said!
- 'Never Again!' he replied.
Display the number of words in all files that begin with the letter h in the /etc directory.
How would you send a 2M file with two 1.44M floppy. How would you put back together the pieces of the split file?
What is the command to translate the : delimiter in /etc/password by #?

Exercises results

Perform basic file management

Overview

Description: Candidates should be able to use the basic UNIX commands to copy, move, and remove files and directories. Tasks include advanced file management operations such as copying multiple files recursively, removing directories recursively, and moving files that meet a wildcard pattern. This includes using simple and advanced wildcard specifications to refer to files, as well as using find to locate and act on files based on type, size, or time.

Key files terms and utilities include:
cp find mkdir mv ls rm rmdir touch
file globbing

cp: Copy files and directories. ls: List directory contents.

Create and Remove directories

To create a directory, use mkdir.

mkdir [options] dir

Common options:

-m  mode: set permission mode. Default use umask.
-p  parent: create parent directory as needed.

Examples:

mkdir -m 0700 bin
mkdir -p bin/system/x86

To delete an empty directory, use rmdir.

rmdir [options] dir

Common options:

-p  parent: remove empty subdirectories.

Examples:

rmdir  tmp
rmdir -p bin/system/x86

Copy files and directories

To copy one file to another, or to a directory, use cp.

cp [options] source target

Source and target can be a file or a directory.

Common options:

-i  interactive: prompt to overwrite
-r  recursive: copy the subdirectories and contents. Use -R for special files.
-f  force: force the overwriting

The default is to silently clobber the target file. Does not alter the source.

Examples:

cp *.[a-z] /tmp
cp readme readme.orig
cp  ls /bin
cp -ri bin/* /bin

Move & Rename files

To rename a file or directory or to move a file or directory to another location, use mv.

mv [options] source target

Source and target can be a file or a directory.

Common options:

-i  interactive: prompt to overwrite
-f  force: force the overwriting
-v verbose

The default is to silently clobber the target file.

Examples:

mv *.[a-z] /tmp
mv readme readme.orig
mv ls /bin
mv -fi bin/* /bin

Listing filenames and information

The command to list files in the current directory is ls.

ls [options] [filenames]

Common options are:

-l for a long format
-F Append a file type character
-a All files, including hidden
-R Recursive listing of subtree
-d Do not descend into directory

The ls is equivalent to the dir command on DOS.

Examples of ls output:

$ ls -l /bin/ls
-rwxr-xr-x    1   root  root  46784 mar 23  2002 /bin/ls
$ ls -ld /bin
drwxr-xr-x    2 root   root   2144 nov  5 11:55 /bin
$ ls -a .
.bash_history .bash_profile .bashrc ...
$ ls -dF /etc .bashrc /bin/ls
.bashrc  /bin/ls*  /etc/

File types

The long format means:

$ ls -l /etc/hosts    #List a long format of the file hosts
-rw-r—r-- 1 root root 677 Jul 5 22:18 /etc/hosts

File content and location Linux/Unix does not distinguish file by filename extension, like Windows. To determine the file content use file.

$ file /etc .bashrc /bin/ls /dev/cdrom
/etc:       directory
.bashrc:    ASCII English text
/bin/ls:    ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), stripped
/dev/cdrom: symbolic link to /dev/hdc

To determine if a command is a built-in shell command or a program, use type, and use which to find its location.

$ type cp cd ls which type
cp is /bin/cp
cd is a shell builtin
ls is aliased to `ls $LS_OPTIONS'
which is aliased to `type -p'
type is a shell builtin
$ which cut
/usr/bin/cut

Creating and using filenames

Filenames can be created with:

I/O redirection

cat chapter1 chapter2 > book

An editor, such as vi.

vi mynewfile

Many of the Unix utilities

cp file newfile

An application

netscape

The touch command, which creates empty files (or updates the "date modified" of existing files)

touch memo

The valid filename may have:

Maximum 255 characters per filename
Case sensitive 'A' and 'a' are different
Any character except forward '/'
Recommended alphanumeric characters as well as plus, minus, and underscore characters.
Characters to avoid
Hypen character.
touch my-file -lt
White space.

touch more drink
touch "more drink"

Most other special characters !@#$%^&*():;"'}{|\<,>.?~`

touch memo*

Remove files or directories

To remove files or subtree directories, use rm.

rm [options] files

Files can be a file or a directory.

Common options:

-i  interactive: prompt to for each removal
-f  force: force the overwriting
-r recursive: remove subtree directories and contents

There is no unremove or undelete command.

Examples:

rm *.[a-z]
rm readme readme.orig
rm  ls /bin
rm -rfi /bin
cd; rm -rf *  .* # !! this removes all files of in the home directory of the current user, as well as those in the subdirectories !!

Locating files in a subtree directory

To search for a file in a subtree directory, use find.

find [subtrees] [conditions] [actions]

The command can take multiple conditions and will search recursively in the subtree.

Some possible conditions are:

-name [FNG] # Search fo the FNG name
-type c  # Type of file [bcdfl]
-size [+-]# # Has a +- size in blocks (c:bytes,k:kilo)
-user [name] # Own by user
-atime [+-]# # Accessed days ago +n means not been accessed for the last n days -n means been accessed the last ndays.
-mtime [+-]# # Modified days ago
-perm nnn # Has permision flags nnn

Some possible actions are:

-print  # Print the pathname
-exec cmd {} \; # Execute cmd on the file
-ok cmd {} \; # Same as -exec but ask first

Examples

find . -name '*.[ch]' -print
find /var /tmp . -size +20 -print
find ~ -type c -name '*sys*' -print
find / -type f -size +2c -exec rm -i {} \;
find / -atime -3 -print
find ~jo ~toto -user chloe -exec mv {} /tmp \;

To locate a binary, source file, or man pages, use whereis.

Whereis [options]

Common options:

-b: Search only for binaries.
-m: Search only for manual sections.
-s: Search only for sources.

Examples:

$ whereis host
host: /usr/bin/host /etc/host.conf /usr/share/man/man1/host.1.gz
$ whereis -m host
host: /usr/share/man/man1/host.1.gz

To locate a file located somewhere defined by the PATH variable, use which.

$ which -a ls
/bin/ls

The -a will look for all possible matches in PATH, not just for the first one.

Exercises

Compose an interactive command to remove all .tmp file in your home directory. Respond y to every prompt.
List all the files in users home directories ending with .pdf that are bigger than 50 blocks and have not been accessed for a month.
Create a file file.h that will contain all the filenames ending with .h found in the /usr directory.
Do a touch on all the c files found in /usr/src/packages directory.
What are the default permissions when you create a new file and a new directory?
How would you create a new file or directory that contains a space in the filename? (Example: 'new dir')
What is the command to remove all the files of type char and block in your home directory?
How to find the location of the programm find ?
Delete all files in /tmp witch are not owned by root and have not been accessed for a week.

Exercises results.

Use streams, pipes, and redirects

Overview

Description: Candidates should be able to redirect streams and connect them in order to efficiently process textual data. Tasks include redirecting standard input, standard output, and standard error, piping the output of one command to the input of another command, using the output of one command as arguments to another command and sending output to both stdout and a file.

Key files terms and utilities include:
tee xargs < << > >> | ` `

Standard input and standard output

For each command executed in a terminal, there are: a standard input value 0. (default keyboard) a standard output value 1.(default terminal) and a standard output for the errors value 2. (default terminal).

Each channel can also be identified by an address: &0 for input, &1 for output, And &2 for errors.

Each channel [n] can be redirected. [n]< file: Default value of n is 0 and it reads standard input from file. [n]> file: Default value is 1 and it sends standard output to file, overwriting the file if it exists. (clobber) [n]>>file: Default value is 1 and it appends standard output to file. <<word: Read standard input until word is reached. `command`: Substitute the command name by the result.

Examples:

$ pwd > file  # out=file in=none error=terminal.

cat chap* >book # out=book in=none error=terminal.

mv /etc/* . 2>error # out=terminal in=none error=error.

echo end of file >> book # out=book in=none error=terminal.

set -o noclobber # Shell does not clobber existing files.

ls > list 2>&1 # ls and errors are redirected to list.

ls 2>&1 > list # Errors are redirected to standard output and ls output is redirected to list.

cat `ls /etc/*.conf` > conffile 2>>/tmp/errors

Concatenate all the configuration files from /etc dir in conffile and append errors in file /tmp/errors.

Redirecting with pipes

Pipes are an efficient way to apply multiple commandes concurrently.

command1 | command2

The standard output of command1 will be piped to the standard input of command2. The standard error is not piped.

Examples:

ls -l /dev | more
ls -l /etc/*.conf | grep user | grep 500
ls -l /bin | mail `users`

To redirect the standard output to a file and to the terminal at the same time, use tee.

ls -l /dev | tee file
ls -l /etc | tee -a file # Append to the file

Building arguments

The xargs utility constructs an argument list for a command using standard input.

xargs [options] [command]

The xargs command creates an argument list for command from standard input. It is typically used with a pipe.

Common options: -p: prompt the user before executing each command.

Examples:

ls f* | xargs cat # Print to standard output the content of all files starting with f.
find ~ -name 'proj1*' print | xargs cat

Search in the Home directory for files starting with proj1 and send it to the standard input of cat.

Use the /dev/null device file to discard output or error messages.

Try the following:

grep try /etc/*
grep try /etc/* 2> /dev/null
grep try /etc/* > /dev/null 2> /dev/null

Exercises

Create a file list.bin that will contain all the filenames from /bin directory.
Write a command that will append the list of files from /usr/local/bin to the file named list.bin and discard any error output.
Split list.bin file into files that are 50 lines long and remove list.bin.
From the splitted files create back list.bin. but with inversed order.
Simplify the following commands:

ls *.c | xargs rm
ls [aA]* | xargs cat
cat `ls *.v` 2>/dev/null

Use find to do the following command:

more `ls *.c`

Write a command that will create a file list.sbin with the contents of /sbin and at the same time display it to standard output.
Create a file that within the filename you include the creation time.
Create a file that will contain all the filename in reverse order with extension .conf from the /etc directory.

Exercises results.

Create, monitor, and kill processes

Overview

Description: Candidates should be able to manage processes. This includes knowing how to run jobs in the foreground and background, bring a job from the background to the foreground and vice versa, start a process that will run without being connected to a terminal and signal a program to continue running after logout. Tasks also include monitoring active processes, selecting and sorting processes for display, sending signals to processes, killing processes and identifying and killing X applications that did not terminate after the X session closed.

Key files terms and utilities include:
& bg fg jobs kill nohup ps top

Create processes

A running application is a process. Every processes have: A process ID. A parent process ID. A current directory PWD. A file descriptor table. A program which it is executing. Environment variables, inherited from its parent process. Stdin, stdou, and stderr Other

Bash is a program that when it is executed becomes a process. Each time you execute a command in a shell a new process is created. Except for the buil-in shell command. They run in the shell context. Use type to check if a command is a built-in shell command.

Example type cp ls which type Monitor processes Once the system is up and running from a terminal it is possible to see which processes are running with the ps program. To display a long format of all the processes in the system, do:

ps -Al
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
004 S     0     1     0  0  80   0 -   112 do_sel ?        00:00:04 init
004 S     0   381     1  0  80   0 -   332 do_sel ?        00:00:00 dhcpcd
006 S     0  1000     1  0  80   0 -   339 do_sel ?        00:00:00 inetd
044 R     0  1524  1222  0  79   0 -   761 -      pts/3    00:00:00 ps

The ps program will display all the processes running and their PID numbers and other information. To see a long format of the processes in your login session, do a:

ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
000 S   500  1154  1139  0  80   0 -   724 wait4  pts/1    00:00:00 bash
002 S   500  1285  1283  0  77   0 - 24432 wait_f pts/1    00:00:00 soffice.bin
040 R   500  1442  1435  0  79   0 -   768 -      pts/4    00:00:00 ps
F: Process Flags 002: being created, 040: forked but didn't exec, 400: killed by a signal.
S: Process States: R: runnable, S: sleeping, Z: zompbie
UID: User ID, PID: Process ID, PPID: Parent Process ID, C: Scheduler, PRI: priority
NI: Nice value, SZ: size of routine, WCHAN: name of routine

Monitor processes

To monitor the processes in real-time, use top. top

 9:20am  up  2:48,  4 users,  load average: 0.15, 0.13, 0.09

78 processes: 75 sleeping, 3 running, 0 zombie, 0 stopped CPU states: 15.3% user, 0.3% system, 0.0% nice, 84.2% idle Mem: 254896K av, 251204K used, 3692K free, 0K shrd, 27384K buff Swap: 514072K av, 0K used, 514072K free 120488K cached

 PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
1517 rarrigon   0   0 40816  39M 17372 R    15.0 16.0   2:59 mozilla-bin
1727 rarrigon  19   0   988  988   768 R     0.3  0.3   0:00 top
   1 root      20   0   220  220   188 S     0.0  0.0   0:04 init
   2 root      20   0     0    0     0 SW    0.0  0.0   0:00 keventd

RSS: The total amount of physical memory used by the task. SHARE: The amount of shared memory used by the task. %CPU: The task's share of the CPU time. %MEM: The task's share of the physical memory. Once top is running it is also possible to execute interactive commands: Type N to sort tasks by pid. Type A to sort tasks by age (newest first). Type P to sort tasks by CPU usage. Type M to sort tasks by memory usage. Type k to kill a process (prompted for pid).

Kill processes

The ps program will display all the processes running and their PID numbers. Once the PID is known, it is possible to send signals to the process. SIGSTOP to stop a process. SIGCONT to continue a stopped process. SIGKILL to kill a process.

The program to send a signal to a process is called kill.

kill -SIGKILL [pid]
kill -63 [pid]
kill -l

By default a process is started in foreground and it is the only one to receive keyboard input. Use CTRL+Z to suspend it.

To start a process in background use the &.

bash &
xeyes &

Job control

In a bash process it is possible to start multiple jobs. The command to manipulate jobs are:

jobs   # List all the active jobs
bg %job  # Resume job in background
fg %job  # Resume job in foreground
kill %job # Kill background job

When bash is terminated all processes that have been started from the session will receive the SIGHUP signal. This will by default terminate the process.

To prevent the termination of a process, the program can be started with the nohup command.

nohup mydaemon

Exercises

Exercises results.

Modify process execution priorities

Overview

Description: Candidates should should be able to manage process execution priorities. Tasks include running a program with higher or lower priority, determining the priority of a process and changing the priority of a running process.

Key files terms and utilities include:
nice ps renice top

Priorities

To start a command with an adjusted priority, use nice.
nice -n +2 [command]
nice -n -19 [command]
The program nice changes the base time quantum of the scheduler.
i.e. it informs the scheduler how important a process is, which is used as a guide to how much CPU time to give it.

For example if you wanted to listen to music whilst ripping another CD, you could use nice -n +5 oggenc, you shouldn’t get any “hops” in the music playback, as the scheduler “knows” the oggenc process is less important.

The value can go from -19 (highest priority) to +20 (lowest priority).
The default value is 0. Only root can set a value below zero.

To modify the priority of a running program, use renice.
renice +1 -u root
# Change the priority for all root processes.
renice +2 -p 193
# Change the priority for PID 193

Exercises

Exercises results

Which user and root processes are using most of the memory?
Open two terminals. In one terminal type the following:

while [ 1 ]
do
echo -n The date is:;
date;
done

From the other terminal see that you can stop and continue the print out.

3) Same start as 2) but, make the print out to stop for 3[s] and to continue for 1[s] repeatedly. 4) Make a shell script to renice all process called apache to a 19 value. 5) Do a print from ps formated as: “username”, “command”, “nice value” 6) Kill all the process called “bash” and owned by user polto..

Regular Expression

Overview

Description: Candidates should be able to manipulate files and text data using regular expressions. This objective includes creating simple regular expressions containing several notational elements. It also includes using regular expression tools to perform searches through a filesystem or file content.

Key files terms and utilities include:
grep regexp sed

Pattern matching

There are two kinds of pattern matching:

Wildcards (File Name Generation)
Regexp (Regular Expression)

Wildcard characters are mainly applied when they are used in the current directory or subdirectories. When wildcard characters *, ?, [ - ], ~, and ! are used in regexp they no longer generate filenames.

Some of the utilities that use regexp are:

grep, egrep
vi
more
sed
Perl

Limited regexp search pattern Used by all utilities using regexp.

Any 1 char . Ab.a Abla or Abca
1 char set [ ] Ab[sd]a Absa or Abda only
1 char range [ - ] Ab[a-z]a Abaa or Abba or ...
Not in set [^ ] Ab[^0-9]a Abaa or Abba or ...
0 or more * Ab*a Absala or Aba or ...
Begin line ^ ^Aba Line starts>Aba
End line $ Aba$ Aba<line ends
Literal \ Aba\$ Aba$

Example:

Ab[0-3]s
^Ab\^bA
[01]bin$
^..\\
[^zZ]oro

Combinations of limited regexp Combinations used by all utilities using regexp.

Any string .* Ab.*a Abrahma or Abaa or ...
String from [ ]* th[aersti]* There or This or ...
Multi range [ - - ] Ab[0-2][a-c]a Ab0aa or Ab1aa or ...
Match \ \\ \\[a-zA-Z]* \Bethoven

Examples:

Ab[0-3][a-z]s
...$
^[01]\^2
[0-9][a-z] \$
[a-zA-Z]*
^[^c-zC-Z]*
^[a-zA-Z0-9]$

Modifier patterns Replace strings matched by regexp patterns

Match m \{m\} b[0-9]\{3\} b911
One or more \{m,\} b[0-9]\{2,\} b52
Up to n \{m,n\} b[0-9]\{2,4\} b1234
Beginning of word \< \<wh where
End of word \> [0-9]\> bin01

grep

To find text in a file, use grep.

grep [options] [string] [files]

Best to quote the string to prevent interpretation.

Common options: -i: Ignore case -l: List filename only if at least one matches -c: Display only count of matched lines -n: Also display line number -v: Must not match.

Examples:

grep host /etc/*.conf
grep -l '\<mai' /usr/include/*.h
grep -n toto /etc/group
grep -vc root /etc/passwd
grep '^user' /etc/passwd
grep '[rR].*' /etc/passwd
grep '\<[rR].*' /etc/passwd

sed

To apply a command on a stream, use sed.

sed [address1][,address2][!]command[options] [files...]

The program sed will apply a command from address1 to address2 in a file. The address1 and address2 format are regular expressions. The sed program is a noninteractive editing tool.

Examples:

sed '1,3s/aa/bb/g' file # Replace in file from line 1 to 3 aa by bb.
sed '/here/,$d' file # Delete line from here to the end.
sed '/here/d' file # Delete lines with word here.
sed '1,/xxx/p' file # Print lines from 1 to xxx.
sed '/ll/,/ff/!s/maison/house/g' file # In file replace words maison by house excluding lines from ll to ff.

Exercises

Exercises results

Process your bookmarks.html file to produces a list containing just the web site titles in a file called mywebsites.txt
Copy all the files from /etc in your home direcory etc/. Display the contents of all the *.conf files by replacing the word host by machine.
Display the contents of all the *.conf files that don't contain the root word. What is the command using grep and sed?
Print out all the group names where root belongs to.
List all the group names that are 4 or 5 characters long.
List all the files that contain character lines without spaces (blank).
List in etc/ directory all the files that contain numerical characters.
Print with ls only the directory names in /.
Do “ps -aux” and replace user r_polto by root and print it in to a file called new_process.txt
List all process called apache and owned by a usernames starting by “p” or “P”..

Perform basic file editing using vi

Overview

Description: Candidates should be able to edit text files using vi. This objective includes vi navigation, basic vi nodes, inserting, editing, deleting, copying, and finding text.

Key files terms and utilities include:
vi
/, ?
h,j,k,l
G, H, L
i, c, d, dd, p, o, a
ZZ, :w!, :q!, :e!
:!

vi

When using X-Windows, you can use mouse-oriented editors such as xedit. In a cross-development environment, users use their favorite editor. On a non-windowing system, you only need a keyboard editor such as vi. The vi editor on Linux is the same as on any Unix systems. vi has two modes:

Command mode: Anything you type will be interpreted as a command
Input mode: Anything you type will be inserted into the file

Transition from one mode to another Command mode to Input mode: i, I, a, A, o, O keys Input mode to Command mode: ESC key The default starting mode is the Command mode The file configuration .exrc can be created in your HOME directory to set up some vi behavior.

set ignorecase # No case-sensitive
set tabs=3 # 3 space for tab character

Perform basic file editing using vi

Enter Input mode

i before cursor
I at start of line
a after cursor
A at end of line
o open line below
O open line above

Delete

dw delete word
dd delete line
D delete to end of line
x delete char at cursor

Move cursor

l one space right
h one space left
j one line down
k one line up
$ end of line
^ start of line
w next word
e end of word

Exercices

Use vi from any directory to begin editing an empty buffer.
Enter a few lines of text into this buffer.
Save the contents of the buffer to the directory.
Open the file again with vi.
Create a new line beneath what you typed earlier. (Don't use i from command mode!)
Exit vi without saving these changes.

Exercises results