Wednesday, March 21, 2012

libs are not available on the server

Sometime Linux server distro will report missing libraries when libs are not available on the server or not in the path.

a) Searching and install packages

Debian/Ubuntu Linux:
apt-cache search lib-name-here
apt-cache search libdl

RHEL and friends:
yum whatprovides lib-name-here
... yum whatprovides

Now you need to install the library set from the DVD/repos using apt-get/yum command.

b) Another recommend option is to go though,, and rhn search feature. Again install missing libs using apt-get/yum.

c) Run ldd on binary:
ldd /path/to/binary

Find missing lib and install it. If it is installed in your home directory or another location (say /usr/local/app dir), set lib path using /etc/$lib.conf file (replace $lib with actual name). An example
cat /etc/

Do not forgot to run ldconfig -v to update lib cache.

Tuesday, March 20, 2012


One-stop performance analysis using atop

Linux system administrators often receive complaints about the performance of their systems. It can be rather difficult to track down these problems and to find why, when, and how often they happen. Being able to zoom in on the processes that are responsible, and to see what has happened in the past, is very valuable. The atop utility was written with just these things in mind.

It is nice tools that will work for all the command (sar,vmstat,iostat,top).

You need to install the rpm for this.


#rpm -ivh atop-1.24-1.i386.rpm

Performance analysis tools :

Linux has a rich set of tools for performance analysis, but each has its own capabilities and limitations. In developing atop, the following list was considered to be desirable features for the tool:

* The tool should obviously be able to show the current situation. However many resource problems don't occur "now". Often complaints will come in about the system performance "last night" or "last week". Therefore the tool must be able to look in the past. Being able to look in the future would be a "nice to have" but was deemed too difficult to implement.

* It should show the load of the four main resources on a system level: CPU, memory, disk I/O, and network usage.

* The four main resources are consumed by or on behalf of processes, so the tool should be able to show which processes (over)load the four resources.

* A monitoring tool takes snapshots of the system, using a certain interval. If a process used resources since the last snapshot but has exited before the current snapshot, the tool should still be able to show which processes loaded which resources. In other words: the sum of resource usage by the processes should be equal to the system wide reported resource usage.

Looking at this list of requirements, none of the existing standard analysis tools meets the bill. sar shows extensive data regarding CPU, memory, disk and network usage from the past and the present. However, it cannot "zoom in" on processes: it only shows resource usage on a system level. vmstat and iostat can only show CPU, memory and disk usage on a system level; they cannot show usage data from the past. Finally top, one of the most used performance monitors, does show CPU and memory usage on a system level and on a process level. However, it only shows the current situation, it cannot show usage data from the past. It also does not show the resource consumption for exited processes, so with top it is possible that on a system level the CPU is shown as 90% busy, while the sum of all CPU consumption on a process level is only 40% (the other 50% might have been used by processes that exited between the previous and the current snapshot).

This chart compares the characteristics of these other analysis tools with atop:

atop is free software, and can be downloaded from the web site, though many Linux distributions include atop in their repositories. After installing atop, the command atopsar is also available. It can be compared to sar but references the same log files that are generated and used by atop.

Characteristics of atop

atop was created mainly because the other tools don't report about processes that exit between snapshots. When using "process accounting", the kernel writes a record to a log file for every process that exits. atop will use these records to make a process activity list that is as complete as possible, including processes that exited since the last snapshot.

atop shows the load of the CPUs, memory, disks, and network on a system level. Apart from the network, atop also shows which processes consume these resources (for network utilization per process, a kernel patch is provided). By default, atop shows generic information about processes (like PID, name, CPU utilization, memory utilization, disk utilization, and status). However, more information about the process's memory usage, disk I/O, and scheduling characteristics is available by using single-character keystrokes (for example, s for scheduling characteristics).

Users can always override the default sorting order that atop uses. For example, for more information about a process's memory usage, the M subcommand sorts the processes in descending order of their resident memory usage. But, these processes can also be sorted on their disk I/O usage by using the D subcommand. Typing A will let atop determine what the most sensible sorting order would be given the most heavily used resource at the moment. In the system overview (the top half of the screen) a line will be highlighted if that particular resource is overloaded.

Obviously, not all data about all resources can be shown on the screen at once. Therefore, if the window is resized, atop will automatically show more (or less) data depending on the room available. Configurable priorities are used to determine what data is no longer shown if there is too little space.
Using atop on a system level

The default screen of atop looks like this:

Wednesday, March 14, 2012

hyper-threading enabled

Is hyper-threading enabled on a Linux system ?

When you need to know whether hyper-threading is enabled without the luxury to reboot the system (and consulting the BIOS), you can simply look at the output of /proc/cpuinfo and compare the siblings with the cpu cores fields.

Even though /proc/cpuinfo shows you all the logical CPUs (processor field) in the system, the siblings field holds the number of logical CPUs for the physical CPU this entry belongs to (including both the cores and the hyper-threaded LCPUs).

In other words, if you see:
processor : 7
physical id : 9
siblings : 4
cpu cores : 2

It means that LCPU #7 (the eight logical CPU in your system) is one of the 4 logical CPUs on the physical CPU that has 2 cores. So yes, hyper-threading is enabled.

If the number of cpu cores is the same as the number of siblings, hyper-threading is disabled.

If you want to make sure how many physical CPUs are in the system, you can use the above information to calculate the number of physical CPUs. However simply counting the different physical id fields is easier.

You could also count the number of sockets that are populated from the dmidecode output, if you really want to make sure the kernel is accounting for all of them.

Update: If you want to disable hyper-threading online (and selectively) on RHEL5/CentOS-5 you can do this by disabling the HT processor by doing:

echo 0 >/sys/devices/system/node/node0/cpu2/online

But in general it is better to disable it in the BIOS if you don't need HT at all.

* centos
* hyper-threading
* linux

Nice tool that shows you what/where cpuX is

./configure && make && ./utils/lstopo

On a dual socket nehalem with HT/SMT on lstopo gives (phys=X is the linux processor number):

$ ./hwloc-1.0/utils/lstopo
Machine (24GB)
NUMANode #0 (phys=0 12GB) + Socket #0 + L3 #0 (8192KB)
L2 #0 (256KB) + L1 #0 (32KB) + Core #0
PU #0 (phys=0)
PU #1 (phys=8)
L2 #1 (256KB) + L1 #1 (32KB) + Core #1
PU #2 (phys=1)
PU #3 (phys=9)
L2 #2 (256KB) + L1 #2 (32KB) + Core #2
PU #4 (phys=2)
PU #5 (phys=10)
L2 #3 (256KB) + L1 #3 (32KB) + Core #3
PU #6 (phys=3)
PU #7 (phys=11)
NUMANode #1 (phys=1 12GB) + Socket #1 + L3 #1 (8192KB)
L2 #4 (256KB) + L1 #4 (32KB) + Core #4
PU #8 (phys=4)
PU #9 (phys=12)
L2 #5 (256KB) + L1 #5 (32KB) + Core #5
PU #10 (phys=5)
PU #11 (phys=13)
L2 #6 (256KB) + L1 #6 (32KB) + Core #6
PU #12 (phys=6)
PU #13 (phys=14)
L2 #7 (256KB) + L1 #7 (32KB) + Core #7
PU #14 (phys=7)
PU #15 (phys=15)