L120: Linux System Administration II

Linux System Administration

Prerequisites

• 
  None
  

Goals

• 
  Customise system logging system
  

• 
  Configure cron and at
  

• 
  Understand backup tools and strategies
  

• 
  Finding documentation
  

Overview
We will concentrate on the main tasks of system administration such as monitoring log files, scheduling jobs using at and cron. This also includes an overview of the documentation available (manpages and online resources) as well as some backup concepts.

1. Logfiles and configuration files

The /var/log/ directory

This is the directory where most logfiles are kept. Some applications generate their own log files (such as squid or samba). Most of the system logs are managed by the syslogd daemon. Common system files are :
cron keeps track of messages generated when cron executes

mail messages relating to mail

messages logs all messages except private authentication authpriv, cron, mail and news

secure logs all failed authentications, users added/deleted etc

The most important log file is messages where most activities are logged.

The /etc/rsyslog.conf file
When rsyslogd is started it reads the /etc/rsyslog.conf configuration file by default. One can also start rsyslogd with -f and the path to an alternative config file. This file must contain a list of items followed by a priority, followed by the path to the log-file:

item1.priority1 ; item2.priority2 /path-to-log-file

Valid items are :

auth and authpriv user general and private authentication

cron cron daemon messages

kern kernel messages

mail

news

user user processes

uucp

Valid priorities are: (from highest to lowest)

emerg

alert

crit

err

warning

notice

info

debug

*

none

Priorities are minimal! All higher priorities will be logged too. To force a priority to be info only you need to use an '=' sign as in:

user.=info /var/log/user_activity

Listing of /etc/rsyslog.conf

# rsyslog configuration file # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html #### MODULES #### # The imjournal module bellow is now used as a message source instead of imuxsock. $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imjournal # provides access to the systemd journal #$ModLoad imklog # reads kernel messages (the same are read from journald) #$ModLoad immark # provides --MARK-- message capability # Provides UDP syslog reception #$ModLoad imudp #$UDPServerRun 514 # Provides TCP syslog reception #$ModLoad imtcp #$InputTCPServerRun 514 #### GLOBAL DIRECTIVES #### # Where to place auxiliary files $WorkDirectory /var/lib/rsyslog # Use default timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # File syncing capability is disabled by default. This feature is usually not required, # not useful and an extreme performance hit #$ActionFileEnableSync on # Include all config files in /etc/rsyslog.d/ $IncludeConfig /etc/rsyslog.d/*.conf # Turn off message reception via local log socket; # local messages are retrieved through imjournal now. $OmitLocalLogging on # File to store the position in the journal $IMJournalStateFile imjournal.state #### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog mail.* @10.11.12.13 # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg :omusrmsg:* # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log

2. Log Utilities

The logger command
The first utility logger conveniently logs messages to the /var/log/messages file:

If you type the following:

logger program myscipt ERR

The end of /var/log/messages should now have a message similar to this:

Jul 17 19:31:00 localhost penguin: program myscript ERR

local settings
The logger utility logs messages to /var/log/messages by default. There are local items defined that can help you create your own logfiles as follows. local0 to local7 are available items for administrators to use. The availability depends on the system (RedHat local7 logs boot-time information in /var/log/boot.log). Add the following line to /etc/syslog.conf:

local4.* /dev/tty9

Restart the syslogd or force it to re-read its' configuration file as follows:

killall -HUP syslogd

The next command will be logged on the /dev/tty9

logger -p local4.notice "This script is writing to /dev/tty9"

An interesting device is the /dev/speech this is installed with the Festival tools.

logrotate
The log files are updated using logrotate. Usually logrotate is run daily as a cron job. The configuration file /etc/logrotate.conf contains commands to create or compress files.

Listing of logrotate.conf

# rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # send errors to root errors root # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed compress # RPM packages drop log rotation information into this directory include /etc/logrotate.d # no packages own lastlog or wtmp -- we'll rotate them here /var/log/wtmp { monthly create 0664 root utmp rotate 1 }

3. Automatic Tasks

Using cron
The program responsible for running crons is called crond. Every minute the crond will read specific files containing command to be executed. These files are called crontabs.
User crontabs are in /var/spool/cron/. These files should not be edited directly by non-root users and need to be edited using the crontab tool (see below).
The system crontab is /etc/crontab. This file will periodically exectute all the scripts in /etc/cron.* this includes any symbolic link pointing to scripts or binaries on the system.
To manipulate cron entries one uses the crontab utility. Scheduled tasks are view with the -l option as seen below:

crontab –l

0 * * 07 2 /usr/bin/find /home/penguin -name core -exec rm {} \;

Does the user root have any crontabs?

Similarly the -e option will open your default editor and lets you enter a cron entry.

User root can use the -u to view and change any user's cron entries

To delete your crontab file use crontab -r.
This is the format for crontabs :

Minutes(0-59) Hours(0-23) Day of Month(1-31) Month(1-12) Day of Week(0-6) command

Permissions:
By default only the root user can use crontab. The files /etc/cron.deny and /etc/cron.allow

are available to allow or disallow the creation of crontabs for users listed in /etc/passwd.

Scheduling with “at”
The at jobs are run by the atd daemon. At jobs are spooled in /var/spool/at/
The at command is used to schedule a one off task with the syntax
at [time]
Where time can be expressed as:
now

3am + 2days

midnight

10:15 Apr 12

teatime
For a complete list of valid time formats see /usr/share/doc/at-xxx/timespec.

You can list commands that are scheduled with atq or at -l. The at jobs are saved in /var/spool/at/:

ls /var/spool/at/

a0000100fd244d spool

When using atq you should have a list of jobs proceeded by a number. You can use this number to dequeue it:

atq

1 2001-07-17 18:21 a root

From the atq listing we see that the job number is 1, so we can remove the job from the spool as follows:

atrm 1

Permissions:
By default at is restricted to the root user. To override this you must either have an empty /etc/at.deny

or have a /etc/at.allow with the appropriate names.

4. Backups and Compressions

Backup strategies
There are three main strategies to back up a system:
Full: copy all files

Incremental: The first incremental copies all files added or changed since the last full backup, and subsequently copies all the files added or changed since the last incremental backup

Differential: Copies all files added or changed since the last full backup
Example: If you made a full backup and 3 differential backups before a crash, how many tapes would you need to restore ?

Creating archives with tar
The main option to create an archive with tar is -c. You can also specify the name of the archive as the first argument if you use the -f flag.

tar -cf home.tar /home/

If you don't specify the file as an argument tar -c will simply output the archive as standard output:

tar -c /home/ > home.tar

Extracting archives with tar
Extracting is straight forward. Replace the -c flag with an -x. This will cause the archive file to create directories if necessary and copy the archived files in your current directory. To redirect the output of the extracted archive into the directory /usr/share/doc, for example, you can do:

tar xf backeddocs.tar -C /usr/share/doc

Compressions
All archives can be compressed using different compression utilities. These flags are available when creating, testing or extracting an archive:

tar option	compression type
Z	compress
z	gzip
j	bzip2.

The cpio utility

The cpio utility is used to copy files to and from archives. List of files must be given to cpio either through a pipe (as when used with find) or via a file redirection such as with;
- Extract an archive on a tape:

cpio -i < /dev/tape

- Create an archive for the /etc directory:

find /etc | cpio -o > etc.cpio

The dump and restore utilities
Finally, it is also possible to perform backups using dump. Remember that the field after the options in /etc/fstab is used to specify if a device should be backed up or not using dump. An entire device can be backed up this way. However dump can also back directories
When backing up an entire device (not a directory) Information about the previous full or incremental backups is stored in /etc/dumpdates. Dump can automatically do up to 9 incremental backups.
By default dump will save the archive to /dev/st0. Backups are recovered with the restore utility.

dump -0 -f /tmp/etc.dump /etc

You can test this archive with

restore -t -a -f /tmp/etc.dump

Extract all the files with

restore -x -a -f /tmp/etc.dump

or you can interactively extract a list of files (that gets interactively created too):

restore -i -a -f /tmp/etc.dump

restore > add etc/passwd etc/group

restore > extract

restoring ./etc/group

restoring ./etc/passwd

set owner/mode for '.'? [yn] y

restore > ^ D

Backing up with dd

Remember from LPI 101 that the dd tool can make an image of a device preserving everything including:

• 
  the underlying filesystem
  
• 
  the boot sector (first 512 kB)
  

The image can be saved to a file or a device. The same is true retrieving the image.

Syntax:
dd if=FILE/DEVICE of=FILE/DEVICE

What to backup
The following table extracted from the FHS document is used to determine how often specific directories need to be backed up:

	shareable	unshareable
static	/usr, /opt	/etc, /boot
variable	/var/mail	/var/run, /var/spool/mail

5. Documentation

Manpages and the whatis database

The manpages are organised in sections
NAME	the name of the item followed by a short one line description.
SYNOPSYS	the syntax for the command
DESCRIPTION	a longer description
OPTIONS	a review of all possible options and their function
FILES	files that are related to the current item (configuration files etc)
SEE ALSO	other manpages related to the current topic

These are the main sections one can expect to find in a manpage.

The whatis database stores the NAME section of all the manpages on the system. This is done through a daily cron. The whatis database has the following two entries:

name(key) – one line description

The syntax for whatis is:

whatis
The output is the full NAME section of the manpages where string matched named(key)
One can also use the man command to query the whatis database. The syntax is
man -k
This command is similar to apropos. Unlike whatis this will query both the “name” and the “one line description” entries of the database. If the string matches a word in any of these fields the above query will return the full NAME section.

Example: (the matching string has been highlighted)

whatis mkdir

mkdir (8) - make directories

man -k mkdir

mkdir (8) - make directories

The FHS recommends manpages to be kept in /usr/share/man. However additional locations can be searched using the MANPATH environment variable set in /etc/man.config. Each directory is further divided into subdirectories corresponding to manpage sections.

Manpage Sections
Section 1	Information on executables
Section 2	System calls, e.g mkdir(2)
Section 3	Library calls, e.g stdio(3)
Section 4	Devices (files in /dev)
Section 5	Configuration files and formats
Section 6	Games
Section 7	Macro packages
Section 8	Administration commands

To access a specific section N one has to enter:

man N command
Examples:

man mkdir
man 2 mkdir

man crontab

man 5 crontab

Info pages
The FHS recommends info pages be kept in /usr/share/info. These pages are compressed files that can be read with the info tool.
The original GNU tools used info pages rather than manpages. Since then most info pages have been rewritten as manpages. However information about GNU projects such as gcc or glibc is still more extensive in the info pages compared to the manpages.
Installed documents
GNU projects include documents such as a FAQ, README, CHANGELOG and sometimes user/admin guides. The formats can either be ASCII text, HTML, LateX or postscript.
These documents are kept in the /usr/share/doc/ directory.


HOWTOs and The Linux Documentation Project
The Linux Documentation Project provides many detailed documents on specific topics. These are structured guides explaining concepts and implementations. The website URL is www.tldp.org.

The LDP documents are freely redistributable and can be contributed too using a GPL type licence.

Usenet News Groups
The main newsgroups for Linux are the comp.os.linux.* groups (e.g comp.os.linux.networking, comp.os.linux.security ...). Once you have setup a news reader to connect to a news server (usually available through an ISP or a University campus) one downloads a list of all existing discussion groups and subscribes/unsubscribes to a given group.

There are many experienced as well as new users which rely on the newsgroups to get information on specific tasks or projects. Take the time to answer some of these questions if you feel you have the relevant experience.

Notifying Users about the System
It is possible to print information for users login onto the system such as the sysadmin's contact details or the state of the system using either /etc/issue (/etc/issue.net for telnet users) or /etc/motd.
The issue file is printed on the login terminals (ttys) by mingetty and can be used to publish the companies warning regarding the usage of the computer equipment, contact details or even some ASCII art. The same information can be made available through a display manager (see LPI 101). The issue.net file is visible at a telnet login prompt, it should generally not contain information about the system (OS type, kernel version, etc)
The filename motd stand for "message of the day" and is only visible after a successful login.

6. Exercises and Summary

FIles

File	Description
/etc/at.allow, at.deny	at.allow(5) – determine which user can submit commands for later execution via at(1) or batch(1). The format of the files is a list of usernames, one on each line. Whitespace is not permitted. The superuser may always use at. If the file /etc/at.allow exists, only usernames mentioned in it are allowed to use at. If /etc/at.allow does not exist, /etc/at.deny is checked
/etc/cron.allow, cron.deny	crontab(1) – If the cron.allow file exists, then you must be listed therein in order to be allowed to use this command. If the cron.allow file does not exist but the cron.deny file does exist, then you must not be listed in the cron.deny file in order to use this command. If neither of these files exists, only the super user will be allowed to use this command
/etc/crontab	System crontab file read by the crond daemon whenever its modified time is changed
/etc/dumpdates	Stores information about the last full or incremental dumps
/etc/issue	Message printed by the mingetty program at the login prompt on a tty
/etc/issue.net	Message printed by the telnet daemon at the login prompt
/etc/logrotate.conf	Configuration file for logrotate
/etc/motd	Message displayed by login after a successful login
/etc/syslog.conf	Configuration file for syslogd
/usr/share/info	Directory where info pages are stored
/usr/share/man	Directory where the various sections of the manpages are stored
/var/spool/at/	Directory containing spooled at and batch jobs
/var/spool/cron/	Directory containing user defined crontabs. The crontab file has the name of the user that created it and can only be edited with the crontab -e command

Commands

Command	Description
apropos	apropos(1) – searches a set of database files containing short descriptions of system commands for keywords and displays the result on the standard output
at	at(1) – read commands from standard input or a specified file which are to be executed at a later time
atd	atd(8) – run jobs queued by at for later execution
atq	atq(1) - lists the user's pending jobs, unless the user is the superuser; in that case, everybody's jobs are listed. The format of the output lines (one for each job) is: Job number, date, hour, job class
atrm	deletes jobs, identified by their job number
cron or crond	cron(8) – Cron searches /var/spool/cron for crontab files which are named after accounts in /etc/passwd; crontabs found are loaded into memory. Cron also searches for /etc/crontab and the files in the /etc/cron.d directory, which are in a different format

Command	Description
crontab	file loaded by crond. It is also the name of the program used to edit crontabs created by users in /var/spool/cron
dd	copy files and devices with optional modifications such as block size ( see info coreutils dd)
dump	dump(8) – examines files on an ext2/3 filesystem and determines which files need to be backed up
info	read info documentation stored in /usr/share/info
logger	allows shell scripts to log messages with syslogd
logrotate	logrotate(8) – is designed to ease administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large
man -k	same as apropos
restore	restore files or file systems from backups made with dump
syslogd	The system logger. Programs can send messages to syslogd which include information such as the date and the host name. The configuration file /etc/syslog.conf is used to customise where messages are logged (e.g file, device or remote logger)
tar	tar(1) – an archiving program designed to store and extract files from an archive file known as a tarfile. A tarfile may be made on a tape drive, however, it is also common to write a tarfile to a normal file
whatis	whatis(1) – search the whatis database for complete words