Wednesday, 5 February 2014

A Short Guide to AWK

AWK is built to process column-oriented text data, such as tables. In which a file is considered to consist of N records (rows) by M fields (columns)


# awk < file ‘{print $2}’

# awk ‘{print $2}’ file

file = input file
print $2 = 2nd field of the line, awk has whitespace as the default delimiter


# awk -F: ‘{print $2}’ file

-F: = use ‘:’ as delimiter

# awk -F ‘[:;]’ ‘{print $2}’ file

-F ‘[:;]’ = use multiple delimiter, and parse using EITHER ‘:’ OR ‘;’

# awk -F ‘:;’ ‘{print $2}’ file

 -F ‘:;’ = use multiple delimiter, and parse using ':;’ as THE delimiter

# awk -F ‘:’+ ‘{print $2}’ file

 -F ‘:’+ = use multiple delimiter, to match any number ':'


# echo 5 4 | awk '{print $1 + $2}'

output is '9', the result of ‘+’ (works as addition)

# echo 5 4 | awk '{print $1 $2}'

output is '54', to get string concatenation

echo 5 4 | awk '{print $1, $2}'

output is '5 4', to get value of 1st and 2nd field


# awk -F ‘<FS>’ ‘{print $2}’ file

<FS>, aka field separator variable, can consist of any single character or regular expression 
e.g. awk -F ‘:’ ‘{print $3}’ /etc/passwd

# awk -F ‘<FS>’ ‘BEGIN{OFS=“<OFS>”} {print $3, $4}’ file

<OFS>, aka output field separator variable, is the value inserted input separated output
e.g. awk -F ‘:’ ‘BEGIN{OFS=“|||”} {print $3, $4}’ /etc/passwd

# awk ‘BEGIN {RS=‘<RS>’} {print $1}’ file

<RS>, aka row separator variable, it works the same as FS, but vertically

# awk ‘BEGIN{ORS=“<OFS>”} {print $3, $4}’ file

<ORS>, aka output row separator variable, it works the similar way as OFS

# awk ‘{print NR}’ file

NR, aka number of records, is equivalent to line number
it is quite helpful when calculating average

# awk ‘{print NF}’ file

NF, aka number of fields, uses whitespace as delimiter and returns the field number
the value will change when delimiter is redefined with ‘-F’

Note: awk ‘{print $NF}’ file

this will print out the last field of the line instead of number of fields, the similar usage is $0, which prints out the line

# awk ‘{print FILENAME}’ file

it prints out the file name

# awk ‘{print FILENAME, FNR}’ file1 file2

FNR, aka number of records for each input file, will give number of records depends on file specified

Monday, 3 February 2014

How to: Upgrade SVN to 1.7 on CentOS

After searching through the web, I managed to find bits and pieces about upgrading subversion from 1.6 to 1.7 on CentOS. Unfortunately, it is like unraveling a puzzle, every blog starts from a slightly different corner and came across different blocks. I was strapped in perpetual trying and failing because of this. So I thought I better summarise and share my whole experience, along with the pitfalls while trying to work it out.

The upgrade, in a nutshell, is a process of uninstalling the current version of svn, and download, compile and install the target version.


CentOS, SVN 1.6

Recipes and Tasting

1. Remove current svn

That's right! You heard me, be brave and remove svn using:
# yum remove subversion

2. Download target version of svn

Go to the following location:
# cd /usr/local/src/

And download the the target version of svn using:
# wget

Surely you might come across problem like, you can't download the specific version because it does not exist. Fear not, just go to this mirror, and find the version you want and wget it again.

3. Unzip svn

Unzip the svn using:
# tar zxf subversion-1.7.16.tar.gz 

And navigate into the directory:
# mv subversion-1.7.16 subversion
# cd subversion

4. Download dependencies for svn

There are two packages needed by svn, namely apr and apr-util, before svn can be complied. Download and unzip them in the directory as previously navigated to.

tar zxf apr-1.5.0.tar.gz
tar zxf apr-util-1.5.3.tar.gz

Of course, go to this mirror to find a specific version, or the listed version does not exist anymore.

5. Rename dependencies directory

mv apr-1.5.0 apr
mv apr-util-1.5.3 apr-util

6. Configure svn

By running:


6.1 Oops, you might be missing sqlite dependency

The steps have been straight forward so far, yet configure may fail because of the following reason:

configure: checking sqlite library
checking sqlite amalgamation... no
checking sqlite3.h usability... no
checking sqlite3.h presence... no
checking for sqlite3.h... no
checking sqlite library version (via pkg-config)... no

An appropriate version of sqlite could not be found.  We recommmend, but require at least 3.6.18.
Please either install a newer sqlite on this system


get the sqlite amalgamation from:
unpack the archive using tar/gunzip and copy sqlite3.c from the
resulting directory to:

configure: error: Subversion requires SQLite

Now have a look at the sqlite version:

# sqlite3 -version

Mine comes up as '3.3.6', which is well out of date apparently. If you are sure you've got the correct version, try ./configure again, on success go to 7, otherwise jump to 6.2.

Now the sqlite has to be downloaded and upgraded:

yum --enablerepo=atomic upgrade sqlite

Even though this step does work quite well for me, keep on trying to find the correct package if the link let you down. And check the sqlite version again,, it says. Hooray.

6.2 Yet another Oops, you might need to get sqlite dependency manually

Sometimes, even if everything seems to be correct, it just won't work. Just like my case, which I have got the correct version of sqlite, ./configure just decided not to work! That leaves me the second option: to get the sqlite files manually, and put it in the specific directory just for svn configuration's sake:

mv sqlite-amalgamation-3080200 sqlite-amalgamation

7. Compile and install svn

Until now, ./configure should run smoothly. Otherwise, I seriously have no idea what you are up against, so go cry for help elsewhere!

It is then a matter of compiling and installing it:

make install

Hmm, might as well go grab a cup of tea while waiting for it to compile. What's worse that could happen right :D

8. Check svn version and happy upgrading the project

cd <project directory>
svn upgrade

9. Certificate

If unfortunately, the svn server uses HTTP/HTTPS protocol, a specific module 'serf' has to be added to svn, and possibly a P12 certificate has to be assigned (for HTTPS).

The 'serf' can be downloaded from: 

# wget

Unzip and move it to the default folder, and reinstall the project:

# unzip
# mv serf-1.2.1 serf

Alternatively, specify it when reconfiguring subversion project:

# ./configure --with-serf=/usr/local/src/subversion/serf
# make
# make install

Even more annoyingly, the certificate needs to be configured in:

# cd ~/.subversion/
# vim servers

with the following information attached to the end of the file:

ssl-client-cert-file = <P12 unc>
ssl-client-cert-password = <***>

And that's given you have been given the details by your sysadmin. Therefore, I call it 'unfortunate', and I should follow it up once I manage to crack that part of the world :D