Monday, October 22, 2012

Reformating output using perl and regex

Perl can be very handy when working with strings. once can reformat almost any string using regular expressions. I had to work on MIB values recently and input these values to Cacti. I would parse the MIB on a network device and get a list of values like :
TERADICI-PCOIPv2-MIB::pcoipGenDevicesName.1 = STRING: "pcoip-host-0030040e079e"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesDescription.1 = ""
TERADICI-PCOIPv2-MIB::pcoipGenDevicesGenericTag.1 = ""
TERADICI-PCOIPv2-MIB::pcoipGenDevicesPartNumber.1 = STRING: "TERA1200 revision 1.0 (128 MB)"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesFwPartNumber.1 = STRING: "Leadtek rev M host card with copper"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesSerialNumber.1 = STRING: "L12060000489"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesHardwareVersion.1 = STRING: "62917013120-D"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesFirmwareVersion.1 = STRING: "4.0.2"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesUniqueID.1 = STRING: "00-30-04-0E-07-9E"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesMAC.1 = STRING: "00-30-04-0E-07-9E"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesUptime.1 = Counter64: 264895
TERADICI-PCOIPv2-MIB::pcoipImagingDevicesIndex.1 = INTEGER: 1
TERADICI-PCOIPv2-MIB::pcoipImagingDevicesIndex.2 = INTEGER: 2
TERADICI-PCOIPv2-MIB::pcoipImagingDevicesIndex.3 = INTEGER: 3
TERADICI-PCOIPv2-MIB::pcoipImagingDevicesIndex.4 = INTEGER: 4


To pass these values to Cacti, I needed to reformat these lines so they would display :.

To replace a substring in perl, we use the =~ s///. We can also add an "i" at the end to make it case insensitive : =~ s///i.
A simple exemple would be :
$output = "Characters replacement in perl";
$output =~ s/Characters/String/i;
print $output;

would print out :
String replacement in perl

All the power comes from search operators that we can apply on the search part. These seach operators are :
.   Match any character
\w  Match "word" character (alphanumeric plus "_")
\W  Match non-word character
\s  Match whitespace character
\S  Match non-whitespace character
\d  Match digit character
\D  Match non-digit character
\t  Match tab
\n  Match newline
\r  Match return
\f  Match formfeed
\a  Match alarm (bell, beep, etc)
\e  Match escape
\021  Match octal char ( in this case 21 octal)
\xf0  Match hex char ( in this case f0 hexidecimal)
You can follow any character, wildcard, or series of characters and/or wildcard with a repetiton. Here's where you start getting some power:
*      Match 0 or more times
+      Match 1 or more times
?      Match 1 or 0 times
{n}    Match exactly n times
{n,}   Match at least n times
{n,m}  Match at least n but not more than m times
 
So, if we want to get rid of some text in a line, we could use :
$output = "Today blabla is a 354446 nice day!";
$output =~ s/Today \w+ is a \d+ nice day!/Today is a nice day!/i;
print $output;
 
This would output :
Today is a nice day!
 
Another very powerful elements are (). They store the matching pattern in variables and name them \1 \2 etc...
Let see another exemple :
$output = "I want to keep this value and this other value";
$output =~ s/I want to keep (this value) and this (other value)/\1 AND \2/i;
print $output; 
 
Would print :
this value AND other value
 
so back to our MIB values. In order to replace :
TERADICI-PCOIPv2-MIB::pcoipImagingDevicesDisplayHeight.1 = INTEGER12: 1080
by :
pcoipImagingDevicesDisplayHeight.1:1080
We would use the following statement :
$_ =~ s/^TERADICI-PCOIPv2-MIB::(\w+\.\d) = \w+\d*: (\d+)$/\1 \2/i;

One last process. In the previous line, I wanted to keep only the OID name ( 
pcoipImagingDevicesDisplayHeight
 ) without the .1 for all OIDs ending with .1 and for every other OID name I wanted to append the . to the OID name. This is done with the following 2 lines :
 
$_ =~ s/^(\w+)\.1 (\d*)$/\1 \2/i;
$_ =~ s/^(\w+)\.([234567890]) (.*)$/\1\2 \3/i;
 
So because the first line will only match OID.1, it only affects these lines. Line 2 takes care of the others :
pcoipImagingDevicesDisplayProcessRate.1 -> pcoipImagingDevicesDisplayProcessRate
pcoipImagingDevicesDisplayProcessRate.2 -> pcoipImagingDevicesDisplayProcessRate2 
 

Thursday, September 11, 2008

Parsing a text file line by line in ksh

This is how to parse a file line by line in ksh :


while read line
do
run you commands here
done < text_file.txt

Wednesday, September 3, 2008

Enabling xvid codec in Ubuntu 8.04

I usually convert video using a 2 pass process with ffmpeg. The command line is :
date;ffmpeg -i myinputfile.mpg -pass 1 -passlogfile vts.log -qscale 2 -vcodec xvid VTS_02_temp.avi;date;ffmpeg -i myinputfile.mpg -f avi -vcodec xvid -pass 2 -passlogfile vts.log -b 800 -g 300 -bf 2 -acodec mp3 -ab 128 myoutputfile.avi;date

When I tried to the first time on my new ubuntu 8.04, I got this error message :
Unknown codec 'xvid'

After googling on the web, I found the solution. The xvid codec along other non free codecs are not included in the Ubuntu repository. We need to use a separate repository call Medibuntu. This repository contains everything free and non-free for all the video processing needs. The web site is hosted on http://medibuntu.org.

A complete installation procedure is on the repository how-to section of the web site. In short, if you're using Ubuntu 8.04 on i386 architecture, these are the steps you need to enable the new repository :

sudo wget http://www.medibuntu.org/sources.list.d/hardy.list -O /etc/apt/sources.list.d/medibuntu.list
sudo apt-get update && sudo apt-get install medibuntu-keyring && sudo apt-get update


Then you open up Synaptic package manager and do a search on ffmpeg and xvid. you should see quite a few files that need upgrade (a little yellow star on the package icon). Upgrade all packages and you should now be able to encode using xvid codec.

Thursday, August 28, 2008

Install various linux partition on USB drive

If you want to bring your favorite Linux distro with you, why don't you prepare a USB stick ? UNetBootin will do it for you. There are a windows version and a linux version. Check it out : http://unetbootin.sourceforge.net/

Friday, July 18, 2008

Taking control on a PC stuck at GDM

Have you ever been in the situation where you PC at work have rebooted and is now sitting at the GDM display waiting for you to enter a username and password ? Well this happened few times to me. The solution is rather simple. You only need to have an ssh access to your PC.

1 ) Logon to your PC using your ssh access.
2 ) install (if not already there) x11vnc : sudo apt-get install x11vnc
3 ) The tricky part is that gdm using a different xauth file than X11 server. So you need to find out which and where this file is. A simple ps -ef | grep gdm will give you the answer :

jboismar-desktop ~ # ps -ef | grep gdm
root 5683 1 0 11:12 ? 00:00:00 gdm
root 5684 5683 0 11:12 ? 00:00:00 gdm
root 6300 5684 0 11:31 tty7 00:00:00 /usr/bin/X :0 -br -audit 0 -auth /var/lib/gdm/:0.Xauth -nolisten tcp vt7
gdm 6313 5684 0 11:31 ? 00:00:01 /usr/lib/gdm/gdmgreeter
root 6320 5726 0 11:33 pts/0 00:00:00 grep --colour=auto gdm

The part you need is : -auth /var/lib/gdm/:0.Xauth

4 ) then you launch x11vnc with this argument. I usually use root :
jboismar-desktop ~ # x11vnc -auth /var/lib/gdm/:0.Xauth

The last lines should look like this :
18/07/2008 11:35:18 WARNING: the warning message printed above for more info.
18/07/2008 11:35:18

The VNC desktop is: jboismar-desktop:0
PORT=5900

******************************************************************************
Have you tried the x11vnc '-ncache' VNC client-side pixel caching feature yet?

The scheme stores pixel data offscreen on the VNC viewer side for faster
retrieval. It should work with any VNC viewer. Try it by running:

x11vnc -ncache 10 ...

more info: http://www.karlrunge.com/x11vnc/#faq-client-caching


5 ) You can then run a vnc client to access your remote screen. Once you login, GDM will terminate to start the X server. Your x11vnc program will also terminate at this point.

Tuesday, July 15, 2008

Compiling Synergy on Ubuntu 8.04

I'm using a lot synergy at work to use one keyboard/mouse to control my laptop and my desktop. It's a wonderfull software. Unfortunately, it is buggy in Ubuntu 8.04. So I decided to re-compile and see if it is any better.

First, you need to get the latest cvs tree :
cvs -z3 -d:pserver:anonymous@synergy2.cvs.sourceforge.net:/cvsroot/synergy2 co -P synergy

now, go to synergy/src directory and try a ./configure. In my case it ended up like this :
checking for X... no
checking for XTestQueryExtension in -lXtst... no
configure: error: You must have the XTest library to build synergy

After googling everywhere, I finally found this page. So, to get the XTest library, we do :
sudo apt-get install libxtst-dev

Then we need to specify where synergy can get the X include and library files :
./configure -x-includes /usr/include -x-libraries /usr/lib --prefix=/usr

and voila ! The configure process should now be happy. You can then make and make install.

Tuesday, June 17, 2008

Determining version of Solaris

Ever wonder if you are running Solaris 32 or 64 bits ? You can find out with isainfo command.

[root@xxxxxx:/root]
#isainfo
sparcv9 sparc

sparcv9 means you are running 64bits version of the OS. To make sure, use :
#isainfo -b
64

This gives you the name of the instruction set(s) used by the operating system kernel components such as device drivers and STREAMS modules.

This command also gives you if your system runs 32 and 64 bits apps :
#isainfo -v
64-bit sparcv9 applications
32-bit sparc applications