Disk usage using awk

This command will give you the top 20 files or directories that take the most space in the current directory :

sudo du -k -x --max-depth=1 | sed 's/\.$/Total\n/g' | sort -rn | head -20 | awk 'BEGIN{printf "\n%20s %-30s","Size in Kbytes","file or directory"}{printf "\n%20'\''d %-30s", $1, $2} END{print "\n"}'

Here it is in action :)

> sudo du -k -x --max-depth=1 | sed 's/\.$/Total\n/g' | sort -rn | head -20 | awk 'BEGIN{printf "\n%20s %-30s","Size in Kbytes","file or directory"}{printf "\n%20'\''d %-30s", $1, $2} END{print "\n"}'

      Size in Kbytes file or directory
          12,964,287 Total
             455,377 ./Photos
             238,572 ./realtek-linux-audiopack-5.17
             235,719 ./IPMI
             135,153 ./realtek-linux-audiopack-5.16
             123,731 ./PCoIP_Driver_SW_v3-0-6_Linux(source
             119,884 ./alsa-driver-1.0.25
              97,912 ./pcoip_host_software
              69,968 ./nagios-plugins-1.4.16
              58,653 ./BMC
              53,357 ./PCoIP-Test
              50,866 ./scim-1.4.9
              50,793 ./sunbird
              48,648 ./leostream
              41,160 ./teradici-firmware
              24,886 ./cacti-0.8.8a
              19,548 ./wqy-zenhei-0.9.45
              16,258 ./leolog
              15,888 ./PCoIP_Host_Software_Driver_Linux_r4-0-5
              14,394 ./Leostream

>

Adding console output formatting to blogger

I have been looking for a way to display properly formatted command line output in blogger for quite some times now, and I finally have a solution.
Go to Design page of your blog, then click on the "advanced" link on the top left of the page. Scroll all the way down to "Add css". On the text field right below "Add custom CSS" Type the following :

pre.console {
  background-color: #666666 ;
  border: 1px solid #006600 ;
  color: #FFFFFF ;
  font-size:.8em ;
}

Now each time you want to display command output prepend your text with <pre class="console">
and append your text with </pre>

It's not very pretty at the moment, but I will work on it ! Also this does not seem to work on mobile device....

Cacti : How to use templates

Cacti is a pretty good tool to generate graph. It is developped using php and rrd to store the data collected. Its templates are very usefull and very powerfull.... if you figure out how to create them and in which order ;-)
I recently had to graph a family of devices using Cacti. Using templates was an absolute necessity so I spent time trying to figure out how to use them... It took me few days, so if I may spare that time to somebody else, I decided to write a little documentation on the subject. It's more a flow chart than anything else and it is not official documentation but it might help. Because the size of the flowchart is actually quite big, I only put a thumbsize image here but I included a link to a PDF version of the normal size one.

Link to the original post with PDF version of the document.

LSI MegaRAID failed drive replacement

Installing command line tool

The command line tool to manage raid devices should be located at /opt/MegaRAID/MegaCli/MegaCli64. If not, you can download the RPM from IBM. It is an architecture independant package. I personnaly used it on Redhat and opensuse successfully. The one I'm using for this documentation is ibm_utl_sraidmr_megacli-8.04.08_linux_32-64.zip. So you have to untar, then do rpm -Uvh on both packages ( one for library and one for the actual command line tool ).

Getting the big picture

To start with, the following command queries all adapters and returns information about the virtual drives defined, their status and all physical drives that they are made of. The command output a lot of information so I usually grep some keywords to shorten the text. Here is the command and an extract of the output :

./MegaCli64 -LDPDInfo -aALL | egrep "Adapter|Virtual Disk|Name|RAID|State|^Number|^Span|PD:|^Device|Firmware|^$"
Adapter #0

Number of Virtual Disks: 2
Virtual Disk: 0 (target id: 0)
Name:
RAID Level: Primary-5, Secondary-0, RAID Level Qualifier-3
State: Optimal
Number Of Drives:9
Span Depth:1
Number of Spans: 1
Span: 0 - Number of PDs: 9
PD: 0 Information
Device Id: 15
Firmware state: Online

PD: 1 Information
Device Id: 16
Firmware state: Online

PD: 2 Information
Device Id: 17
Firmware state: Online

...

PD: 8 Information
Device Id: 23
Firmware state: Online

Virtual Disk: 1 (target id: 1)
Name:data2
RAID Level: Primary-5, Secondary-0, RAID Level Qualifier-3
State: Degraded
Number Of Drives:24
Span Depth:1
Number of Spans: 1
Span: 0 - Number of PDs: 24
PD: 0 Information
Device Id: 42
Firmware state: Online

PD: 1 Information
Device Id: 43
Firmware state: Online

...

PD: 22 Information
Device Id: 64
Firmware state: Online

PD: 23 Information
Device Id: 65
Firmware state: Online

Adapter #1

Number of Virtual Disks: 1
Virtual Disk: 0 (target id: 0)
Name:
RAID Level: Primary-5, Secondary-0, RAID Level Qualifier-3
State: Optimal
Number Of Drives:11
Span Depth:1
Number of Spans: 1
Span: 0 - Number of PDs: 11
PD: 0 Information
Device Id: 8
Firmware state: Online

PD: 1 Information
Device Id: 9
Firmware state: Online

...

PD: 10 Information
Device Id: 18
Firmware state: Online

Finding the drive to replace

It's a good idea to start gathering info about the Adapter :

/opt/MegaRAID/MegaCli> sudo ./MegaCli64 -AdpAllInfo -aALL

Adapter #0

==============================================================================
                    Versions
                ================
Product Name    : PERC H700 Integrated
Serial No       : 18P02M3
FW Package Build: 12.10.2-0004

...

                Device Present
                ================
Virtual Drives    : 1
  Degraded        : 0
  Offline         : 0
Physical Devices  : 5
  Disks           : 4
  Critical Disks  : 0
  Failed Disks    : 0
...

So we know now that the current machine has one adapter : Adapter 0. So in the following command, we will specify -a0 for adpater 0. Then we get enclosure information :

/opt/MegaRAID/MegaCli> sudo ./MegaCli64 -EncInfo -a0

    Number of enclosures on adapter 0 -- 1

    Enclosure 0:
    Device ID                     : 32
    Number of Slots               : 6
    Number of Power Supplies      : 0
    Number of Fans                : 0
    Number of Temperature Sensors : 0
    Number of Alarms              : 0
    Number of SIM Modules         : 0
    Number of Physical Drives     : 4
    Status                        : Normal
    Position                      : 0
    Connector Name                : Unavailable
    Enclosure type                : SES
    FRU Part Number               : N/A
    Enclosure Serial Number       : N/A
    ESM Serial Number             : N/A
    Enclosure Zoning Mode         : N/A
    Partner Device Id             : 65535

    Inquiry data                  :
        Vendor Identification     : DP
        Product Identification    : BACKPLANE
        Product Revision Level    : 1.07
        Vendor Specific           : 18NJ5VP


Exit Code: 0x00

So we have one adapter a0 and one enclosure with an id of 32. We now query for the logical drive information :

/opt/MegaRAID/MegaCli> sudo ./MegaCli64 -LDInfo -LALL -a0


Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :server
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 557.75 GB
Mirror Data         : 557.75 GB
State               : Degraded
Strip Size          : 64 KB
Number Of Drives per span:2
Span Depth          : 2
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: Yes
Cache Cade Type : Read Only

This shows us the RAID level used ( 1-0 so a mirror of strippes ) and the status of this raid device : Degraded.
So now we look for the deffective drive with the following command. The field we need to watch is Firmware State : Failed.

/opt/MegaRAID/MegaCli> sudo ./MegaCli64 -PDList -a0

Adapter #0

...

Enclosure Device ID: 32
Slot Number: 3
...

Firmware state: Failed

...

The output has been truncated to show only relevant information. So in our case, it's the drive in slot number 3 of enclosure ID 32 that needs to be replaced.

Replacing the drive

We prepare the drive for replacement :

/opt/MegaRAID/MegaCli> sudo ./MegaCli64 -PDOffline -PhysDrv\[32:3\] -a0

Adapter: 0: EnclId-32 SlotId-3 state changed to OffLine.

Exit Code: 0x00
/opt/MegaRAID/MegaCli> sudo ./MegaCli64 -PDMarkMissing -PhysDrv\[32:3\] -a0

EnclId-32 SlotId-3 is marked Missing.

Exit Code: 0x00
/opt/MegaRAID/MegaCli> sudo ./MegaCli64 -PDPrpRmv -PhysDrv\[32:3\] -a0


Prepare for removal Success

Exit Code: 0x00

Now it's time for the physical replacement.

Then if everything went smoothly, you should see the array being rebuild :

/opt/MegaRAID/MegaCli> sudo ./MegaCli64 -PDInfo -PhysDrv\[32:3\] -a0

Enclosure Device ID: 32
Slot Number: 3
Drive's postion: DiskGroup: 0, Span: 1, Arm: 1
Enclosure position: N/A
Device Id: 3

...
Firmware state: Rebuild

...

We can query the controler to see the actual rebuild progress :

/opt/MegaRAID/MegaCli> sudo ./MegaCli64 -PDRbld -ShowProg -PhysDrv\[32:3\] -a0

Rebuild Progress on Device at Enclosure 32, Slot 3 Completed 7% in 3 Minutes.

Exit Code: 0x00

Checking

Eventually, you should see something like that :

/opt/MegaRAID/MegaCli> sudo ./MegaCli64 -PDRbld -ShowProg -PhysDrv\[32:3\] -a0

Device(Encl-32 Slot-3) is not in rebuild process

Exit Code: 0x00 

/opt/MegaRAID/MegaCli> sudo ./MegaCli64 -LDInfo -LALL -a0


Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :server
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 557.75 GB
Mirror Data         : 557.75 GB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives per span:2
Span Depth          : 2
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: Yes
Cache Cade Type : Read Only



Exit Code: 0x00

Reformating output using perl and regex

Perl can be very handy when working with strings. once can reformat almost any string using regular expressions. I had to work on MIB values recently and input these values to Cacti. I would parse the MIB on a network device and get a list of values like :
TERADICI-PCOIPv2-MIB::pcoipGenDevicesName.1 = STRING: "pcoip-host-0030040e079e"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesDescription.1 = ""
TERADICI-PCOIPv2-MIB::pcoipGenDevicesGenericTag.1 = ""
TERADICI-PCOIPv2-MIB::pcoipGenDevicesPartNumber.1 = STRING: "TERA1200 revision 1.0 (128 MB)"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesFwPartNumber.1 = STRING: "Leadtek rev M host card with copper"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesSerialNumber.1 = STRING: "L12060000489"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesHardwareVersion.1 = STRING: "62917013120-D"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesFirmwareVersion.1 = STRING: "4.0.2"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesUniqueID.1 = STRING: "00-30-04-0E-07-9E"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesMAC.1 = STRING: "00-30-04-0E-07-9E"
TERADICI-PCOIPv2-MIB::pcoipGenDevicesUptime.1 = Counter64: 264895
TERADICI-PCOIPv2-MIB::pcoipImagingDevicesIndex.1 = INTEGER: 1
TERADICI-PCOIPv2-MIB::pcoipImagingDevicesIndex.2 = INTEGER: 2
TERADICI-PCOIPv2-MIB::pcoipImagingDevicesIndex.3 = INTEGER: 3
TERADICI-PCOIPv2-MIB::pcoipImagingDevicesIndex.4 = INTEGER: 4

To pass these values to Cacti, I needed to reformat these lines so they would display :.

To replace a substring in perl, we use the =~ s///. We can also add an "i" at the end to make it case insensitive : =~ s///i.
A simple exemple would be :

$output = "Characters replacement in perl";

$output =~ s/Characters/String/i;

print $output;

would print out :
String replacement in perl

All the power comes from search operators that we can apply on the search part. These seach operators are :

.   Match any character
\w  Match "word" character (alphanumeric plus "_")
\W  Match non-word character
\s  Match whitespace character
\S  Match non-whitespace character
\d  Match digit character
\D  Match non-digit character
\t  Match tab
\n  Match newline
\r  Match return
\f  Match formfeed
\a  Match alarm (bell, beep, etc)
\e  Match escape
\021  Match octal char ( in this case 21 octal)
\xf0  Match hex char ( in this case f0 hexidecimal)

You can follow any character, wildcard, or series of characters and/or wildcard with a repetiton. Here's where you start getting some power:

*      Match 0 or more times
+      Match 1 or more times
?      Match 1 or 0 times
{n}    Match exactly n times
{n,}   Match at least n times
{n,m}  Match at least n but not more than m times

So, if we want to get rid of some text in a line, we could use :

$output = "Today blabla is a 354446 nice day!";

$output =~ s/Today \w+ is a \d+ nice day!/Today is a nice day!/i;

print $output;

This would output :

Today is a nice day!

Another very powerful elements are (). They store the matching pattern in variables and name them \1 \2 etc...

Let see another exemple :

$output = "I want to keep this value and this other value";
$output =~ s/I want to keep (this value) and this (other value)/\1 AND \2/i;
print $output; 

Would print :

this value AND other value

so back to our MIB values. In order to replace :

TERADICI-PCOIPv2-MIB::pcoipImagingDevicesDisplayHeight.1 = INTEGER12: 1080

by :

pcoipImagingDevicesDisplayHeight.1:1080

We would use the following statement :

$_ =~ s/^TERADICI-PCOIPv2-MIB::(\w+\.\d) = \w+\d*: (\d+)$/\1 \2/i;

One last process. In the previous line, I wanted to keep only the OID name ( 
pcoipImagingDevicesDisplayHeight
 ) without the .1 for all OIDs ending with .1 and for every other OID name I wanted to append the . to the OID name. This is done with the following 2 lines :

$_ =~ s/^(\w+)\.1 (\d*)$/\1 \2/i;

$_ =~ s/^(\w+)\.([234567890]) (.*)$/\1\2 \3/i;

So because the first line will only match OID.1, it only affects these lines. Line 2 takes care of the others :

pcoipImagingDevicesDisplayProcessRate.1 -> pcoipImagingDevicesDisplayProcessRate

pcoipImagingDevicesDisplayProcessRate.2 -> pcoipImagingDevicesDisplayProcessRate2 

Parsing a text file line by line in ksh

This is how to parse a file line by line in ksh :


while read line
do
run you commands here
done < text_file.txt


Enabling xvid codec in Ubuntu 8.04

I usually convert video using a 2 pass process with ffmpeg. The command line is :

date;ffmpeg -i myinputfile.mpg -pass 1 -passlogfile vts.log -qscale 2 -vcodec xvid VTS_02_temp.avi;date;ffmpeg -i myinputfile.mpg -f avi -vcodec xvid -pass 2 -passlogfile vts.log -b 800 -g 300 -bf 2 -acodec mp3 -ab 128 myoutputfile.avi;date

When I tried to the first time on my new ubuntu 8.04, I got this error message :
Unknown codec 'xvid'

After googling on the web, I found the solution. The xvid codec along other non free codecs are not included in the Ubuntu repository. We need to use a separate repository call Medibuntu. This repository contains everything free and non-free for all the video processing needs. The web site is hosted on http://medibuntu.org.

A complete installation procedure is on the repository how-to section of the web site. In short, if you're using Ubuntu 8.04 on i386 architecture, these are the steps you need to enable the new repository :

sudo wget http://www.medibuntu.org/sources.list.d/hardy.list -O /etc/apt/sources.list.d/medibuntu.list
sudo apt-get update && sudo apt-get install medibuntu-keyring && sudo apt-get update

Then you open up Synaptic package manager and do a search on ffmpeg and xvid. you should see quite a few files that need upgrade (a little yellow star on the package icon). Upgrade all packages and you should now be able to encode using xvid codec.