Translate

Thursday, January 3, 2013

Viewing huge files in Linux/Unix

In Linux there are several system generated files which might be very large in size (say upto thousands of lines).An example of this is the system generated logs(syslogs).A system may generate thousands of logs in a day.This makes it very difficult for the user to read the portion of the file which he wants.'Catting' the file is no solution for the problem.Using grep is useful if you know what pattern to search for.

There are several tools/commands available in unix/linux systems which can make life easier for the user.The head and tail command are two such commands which facilitate convenient viewing of log files.

Below are some examples on how to use these commands effectively.

Display the first N lines of a file using head command 

Syntax: head -n N filename(or file path)

Example: head -n 15 /var/log/syslog

This will display the first 15 lines of the syslog file.

Display the last N lines of a file using tail command

Syntax: tail -n N filename(or file path)

Example: tail -n 50 /var/log/syslog

This will display the last 50 lines of the syslog file.

Ignore the first N-1 lines of a file using tail command  

Syntax: tail -n +N filename(or file path)

Example: tail -n +10 /var/log/syslog

This will ignore the first 9 lines of the syslog file and will print from the line number 10th.   

Ignore the last N lines of a file using head command  

Synatx: tail -n -N filename(or file path)

Example: tail -n -10 /var/log/syslog

This will ignore the last 10 lines of the syslog file

View growing log file in real time using tail command  

Syntax: tail -f filename(or file path)

Example: tail -f /var/log/syslog

View the syslogs written in real time.

Display specific portion(based on line number) of a file using head and tail command  

The following example will display line numbers 50 to 60 of the syslog file.

Note: M(=50) indicates the starting line number and N(=60) indicates the ending line number.

Syntax: cat filename|tail -n +M|head -n (N-M+1)   

Example: cat /var/log/syslog|tail -n +50|head -n 11

In this example

cat -> prints the whole file to stdout

tail -n +50 -> ignores lines upto the given line number(i.e till line 49),and then start printing from the given number(from line no.50)

head -n 11 -> prints the first 11 lines ,that is from line number 50 to line number 60 and ignores the remaining lines.

Viewing compressed log file 

By virtue of log rotation log files are generally compressed and archived on the systems.A compressed log file is generally a .gz(gzip) format file.It is therefore,not possible to view the file contents in human readable form using the cat command. To view a compressed log file we have zcat command.

The general syntax of zcat command is 
zcat [options] [filename(or file path)]

Using head and tail command with zcat  

The output of the zcat command can be piped for the head and tail command to view specific portions of the log file.

To display the first N lines of a compressed file use this command syntax
zcat filename.gz |head -N

To display the last N lines of a compressed file use this command syntax 
zcat filename.gz |tail -N

To ignore the last N lines of a compressed file use this command syntax
zcat filename.gz|head -n -N

To ignore the first N-1 lines of a compressed file use this command syntax
zcat filename.gz|tail -n +N 


 

No comments: