Measuring Performance Data Using Collectl Under Linux

Using the tool collectl it is possible to measure a number of performance indicators under Linux. Particularly interesting are the methods to measure properties like CPU and memory consumption per taks / thread.

Example Code

// Filename: example.c
#include <stdio.h>
#include <pthread.h>
#include <math.h>
#include <stdlib.h>
void *f1(void *arg) {
while (1) {
usleep(1);
printf(“f1\n”);
}
return NULL;
}
void *f2(void *arg) {
while (1) {
usleep(1);
float j = sqrt(cos(sin(1.2)));
printf(“stuff: %f\n”,j);
}
return NULL;
}

int main(void) {
pthread_t p1,p2;
pthread_create(&p1,NULL,f1,”f1…”);
pthread_create(&p2,NULL,f2,”f2…”);

while (1) {
float i = sqrt(1.2);
printf(“main stuff: %f\n”,i);
}
pthread_join(p1, NULL);
pthread_join(p2, NULL);
return 0;
}

Compile the code

gcc -lm -lpthreads example.c

Running Collectl

sh-4.1$ collectl -sZ -i5:10 –procopts t

waiting for 10 second sample…

### RECORD 1 >>> machine <<< (1355908840.001) (Wed Dec 19 10:20:40 2012) ###

# PROCESS SUMMARY (counters are /sec)
# PID User PR PPID THRD S VSZ RSS CP SysT UsrT Pct AccuTime RKB WKB MajF MinF Command
…….
25989 5053 20 17675 2 S 28M 636K 3 1.20 0.41 16 0:02.14 0 0 0 0 ./a.out
25990+ 5053 20 25989 2 S 28M 636K 3 0.43 0.05 4 0:00.63 0 0 0 0 a.out
25991+ 5053 20 25989 2 S 28M 636K 3 0.33 0.13 4 0:00.60 0 0 0 0 a.out
…….

Advertisements

GPS Track Profile Analysis using Perl

In this post I show a small example on how one can easily evaluate GPX track files using Perl and Gnuplot. The idea in brief: Read in GPX track infile.gpx and extract the velocity distribution. In a first approximation I have fitted gaussian distributions…

The calculation of distances between two points in WGS 84 coordinates is done using the approach given by Vincenty. There are methods with an accuracy of up to 10nm (geographiclib, interesting paper on this topic).

Velocity Distribution

Code Snipplets

The code pieces listed below are extracts of what I have done. Do not expect the code to work – it shall only serve as an idea for you…

Perl Script

#!/usr/bin/perl
use Geo::Gpx;
use Data::Dumper;
use DateTime;
use GIS::Distance;
use Statistics::Lite qw(:all);
use POSIX;
my $gis = GIS::Distance->new();
$gis->formula (“Vincenty”);
my $fh = “infile.gpx”;
 
# Extract vlat from GPX file
my $gpx = Geo::Gpx->new( input => $fh );
my $waypoints = $gpx->waypoints();
my $tracks = $gpx->tracks();
my $iter = $gpx->iterate_trackpoints(); #points();
my (@time, @lat, @lon, @ele);
while (my $pt = $iter->())
{
push (@time, $pt->{time}); # Linux Epoch time
push (@lat,$pt->{lat}); # WGS 84
push (@lon, $pt->{lon}); # WGS 84
push (@ele, $pt->{ele}); # meters
}
my %s_time = statshash (@time);
my %s_lat = statshash (@lat);
my %s_lon = statshash (@lon);
my %s_ele = statshash (@ele);
my $n = 1+$#time;
 
# Velocities
my (@vlat, @dlateral_next, @vhor);
my $ms_kmh = 3.6;
for my $i (0..$n-2)
{
my $d = $gis->distance ($lat[$i],$lon[$i] => $lat[$i+1],$lon[$i+1]); 
my $dlateral = $d->meters(); # meters
my $dhorizontal = $ele[$i+1]-$ele[$i]; # meters
my $dt = $time[$i+1]-$time[$i]; # seconds
$dt > 0 or warn;
$dt > 0 or next;
my $vv = sqrt($dlateral**2)/$dt * $ms_kmh; 
my $hh = sqrt($dhorizontal**2)/$dt * $ms_kmh;
push (@vlat, $vv); # km/h
push (@vhor, $hh); # km/h
push (@dlateral_next, $dlateral); # meters
}
my %s_vlat = statshash (@vlat);
my %s_vhor = statshash (@vhor);
my %s_dlateral_next = statshash (@dlateral_next);
 
# Distribution
sub distribution
{
my $nbins = floor(sqrt($n));
my $nsigma = 2;
my ($min, $max) = ($s_vlat{mean}-$nsigma*$s_vlat{stddev}, $s_vlat{mean}+$nsigma*$s_vlat{stddev});
my $d = ($max – $min) / $nbins;
my %hist;
for my $i (0..$nbins-1)
{
$hist{$i}{x} = $min+($i+.5)*$d;
}
for my $i (0..$n-2)
{
my $bin = floor(($vlat[$i]-$min)/$d);
$bin < 0 and warn;
$bin < 0 and $bin = 0;
$bin >= $nbins and warn;
$bin >= $nbins and $bin = $nbins-1;
$hist{$bin}{y} = $hist{$bin}{y}+1;
}
my $f_dist = “dist.txt”;
open (F_DIST, “>”.$f_dist);
for my $j (0..$nbins-1)
{
printf F_DIST “%d %f %f\n”, $j, $hist{$j}{x}, $hist{$j}{y};
}
}
 
distribution();

GnuPlot Fitting

f1(x) = p1*exp(-(x-m1)**2/(2*s1**2))
f2(x) = p2*exp(-(x-m2)**2/(2*s2**2))
f3(x) = p3*exp(-(x-m3)**2/(2*s3**2))
m1=120
m2=80
m3=50
f(x) = f1(x)+f2(x)+f3(x)
fit f(x) “dist.txt” u 2:3 via p1,p2,p3,m1,m2,m3,s1,s2,s3
plot  “dist.txt” u 2:3 w l,f(x)

Network Properties Simulation Using qdisc

If you ever want to simulate bad transmission conditions (packet loss, latency, bandwidth, …) in a network: there is a module built-in to the linux kernel named qdisc. More information can be found on tldp.org.

 

Examples

Add delay 80ms

tc qdisc add dev eth0 root netem delay 80ms

Add delay with +-10ms jitter

tc qdisc add dev eth0 root netem delay 80ms 10ms
NB: For very large jitter, package order will be changed by netemIn order to disable this:
tc qdisc add dev eth0 root handle 1: netem delay 10ms 100ms

tc qdisc add dev eth0 parent 1:1 pfifo limit 1000

Add loss etc

tc qdisc add dev eth0 root netem delay 10ms 1ms distribution normal loss 2% duplicate 0.1%

Show current parameters

tc qdisc show dev eth0tc filter show dev eth0 

More complex example:

Consider you have a machine connected on eth1 and eth2 in your network, which will be used to simulate the network conditions:
d1=eth1

d2=eth2
tc qdisc add dev $d1 root netem delay 50ms 2ms distribution normal loss 0.5% duplicate 0.1%
tc qdisc add dev $d2 root handle 1:0 htb default 15
tc class add dev $d2 parent 1:0 classid 1:1 htb rate 400mbit ceil 400mbit

 

Reset

Once you are done you may want to remove the filters again:
tc qdisc del dev eth0 root
tc qdisc show

Creating an Envelope from Commandline Using LaTeX

This small piece of code will create an envelope from commandline…

#!/bin/zsh

MYADDRESS = “Mr X // Street 1 // ZIP”

usage() {
   echo “$0 – print envelopes using LaTeX”
   echo ”   address:   Mr. A // Street 123 // ZIP City, State // Country”
   echo ”   -f <a4,c4,c5,c6,dl> default: $fomat”
   echo ”         a4: a4              “
   echo ”         c4: a4              “
   echo ”         c5: a4 folded in 1/2″
   echo ”         c6: a4 folded in 1/4″
   echo ”         dl: a4 folded in 1/3″
   exit 0
}

while getopts “f:h” opt; do
   case $opt in
      f) format=$OPTARG; ;;
      h) usage; ;;
   esac
done
shift $(($OPTIND -1))

if [[ $format == c6 ]]; then
   lx=114mm
   ly=162mm
elif [[ $format == c5 ]]; then
   lx=162mm
   ly=229mm
elif [[ $format == a4 ]]; then
   lx=210mm
   ly=397mm
elif [[ $format == c4 ]]; then
   lx=229mm
   ly=324mm
elif [[ $format == dl ]]; then
   lx=100mm
   ly=210mm
else
   exit 1
fi


ADDRESS=${@//\/\//\\newline}
MYADDRESS=${MYADDRESS//\/\//\/}

[[ -z $ADDRESS ]] && usage

TMP=$(mktemp -d /tmp/envelope-XXXXXX)
cd $TMP

cat << eof > envelope.tex
\documentclass{article}
\usepackage[margin=0.2cm,top=0.7cm,papersize={$lx,$ly},landscape,twoside=false]{geometry}
\usepackage{oldgerm}
\usepackage[latin1]{inputenc}

\setlength\parskip{0pt}
\pagestyle{empty}

\def\myaddress{$MYADDRESS}
\def\address{$ADDRESS}

\begin{document}

\begin{minipage}{\textwidth}
{\fontencoding{U}\fontfamily{yswab}\fontseries{m}\fontshape{n}\selectfont
\myaddress 
}
\end{minipage} 
 
\Large
\vspace{$((${lx/mm/}-60))mm}
\setlength\parindent{$((${ly/mm/}/2))mm}
 
\begin{minipage}{\textwidth}
\address
\end{minipage}
 
\end{document}
eof
 
pdflatex envelope.tex
lpr -o Duplex=None -o Manualfeed=On -o InputSlot=Envelope -o PageSize=Custom.${lx/mm/}x${ly} -o PageRegion=Custom.${lx/mm/}x${ly} -Penvelope envelope.pdf
rm -rf $TMP

Make an Image with Email Address

This is a small piece of code to create an image from your Email address:

#!/bin/bash
echo $0 Name Emailaddress
[[ -z $2 ]] && exit 2
NAME=$1
shift
convert -size 250×30 xc:transparent -font Verdana -fill “#003399” -pointsize 16 -draw “text 5,15 ‘Email: $@'” $NAME-mail.png


Scan PDF from Commandline

In the following I show a small piece of code to scan PDF files from commandline:

#!/bin/bash
cat << EOF 
— $0 — Scan Images -> PNG-Archive & PDF
Parameters:
   (1) Name
   -> Scanning: counter_start counter_increment
   -> Postprocessing: 
      -rotate (by 90 deg)
      -gray (keep it grayscale)
EOF

function DIE { echo “$@”>&2; exit -1; }
function is_installed {
   while [[ -n $1 ]]; do
      [[ -z $( type -p $1  ) ]] && DIE “missing: $1”
      shift
   done
}
#is_installed convert pdftk scanimage

[[ -z $1 ]] && DIE “missing name”
NAME=$1
shift

if [[ -n $2 ]]; then
   CNT=$1
   CNT_INC=$2
   shift 2

   MODE=Lineart
   RES=300
   echo “Scanning images”
   scanimage –batch=$NAME”_%d”.tif –batch-increment $CNT_INC –batch-start $CNT –source ADF –mode $MODE –resolution $RES -x 210 -y 297 -v
   exit
fi

#unpaper –layout double –output-pages 2
#http://unpaper.berlios.de/unpaper.html

ROTATE=””
GRAY=”-threshold 50%”
while [[ -n $1 ]]; do
   case $1 in 
      -rotate) ROTATE=”-rotate 90″; shift; ;;
      -gray) GRAY=””; shift; ;;
      *) shift; ;;
   esac
done
 
echo “Processing images:”
for FILE in $NAME*tif; do
   echo “Processing “$FILE”:”
   PREFIX=`echo $FILE | sed ‘s/.tif//g’`
   LIMITS=”-limit memory 512 -limit map 512″
   PAGE=”$ROTATE -page a4 +repage”
   COMPRESSION=”-compress Group4″
   #[[ ! -e $PREFIX.png ]] && convert $LIMITS $GRAY -monitor $FILE $PREFIX.png
   [[ ! -e $PREFIX.pdf ]] && convert $LIMITS $PAGE $GRAY $COMPRESSION -monitor $FILE $PREFIX.pdf
done
 
echo “Creating PDF”
CNT_MAX=`find -name “$NAME*pdf” | tr -cd 0-9″\n” | awk ‘$1>max{max=$1}END{print max}’`
FILES=””
for I in `seq 1 $CNT_MAX`; do FILES=$FILES” ${NAME}_$I.pdf”; done;
pdftk $FILES cat output $NAME.pdf
 
echo “Creating archive”
#tar cvjf ${NAME}_png.tar.bz2 ${NAME}*png
tar cvjf ${NAME}_tif.tar.bz2 ${NAME}*tif
 
echo “Cleanup”
rm ${NAME}_*.pdf *tif #*png

Using SSH as VPN

One of the not so commonly known features of SSH is: you may use it for a poor man’s VPN.

My Network 1   —   Client   <—SSH—>   Server   — My Network 2

Conventions

USER = username
HOST = ip of the server host
IDENTITY = ssh key file

Preparation of the Server Side
Edit the daemon configuration file and restart the sshd afterwards
#/etc/ssh/sshd_config
PermitTunnel yes

Open the SSH session from the client side:
sudo ssh -i $IDENTITY -F /home/$USER/.ssh/config -vvvw any:any root@$HOST true &

Prepare the local tunnel and configure the routes on the client:
sudo modprobe tun
sudo tunctl -t tun0 -u $USER
sudo ifconfig tun0 10.0.0.2 pointopoint 10.0.0.1
sudo route add -net 192.168.1.0 netmask 255.255.255.0 dev tun0
 
Prepare the remote tunnel and configure iptables on the server:
cat << eof | ssh root@$HOST
modprobe tun
tunctl -t tun0 -u $USER
ifconfig tun0 10.0.0.1 pointopoint 10.0.0.2
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -s 10.0.0.2 -j SNAT –to-source $SERVER_IP
iptables -A FORWARD -d 10.0.0.2 -j ACCEPT
eof

The whole script:

#!/bin/bash
 
HOST=
SERVER_IP=
USER=
IDENTITY=/home/$USER/.ssh/id_dsa
 
sudo ssh -i $IDENTITY -F /home/$USER/.ssh/config -vvvw any:any root@$HOST true &
sleep 5
tid=$(ps aux | awk ‘/ssh -i/{print$2;exit}’)
 
echo “Prepare local tunnel”
sudo modprobe tun
sudo tunctl -t tun0 -u $USER
#sudo ifconfig tun0 up 10.0.23.2 netmask 255.255.255.252
sudo ifconfig tun0 10.0.0.2 pointopoint 10.0.0.1
sudo route add -net 192.168.1.0 netmask 255.255.255.0 dev tun0
 
echo “Prepare remote tunnel”
cat << eof | ssh root@$HOST
modprobe tun
tunctl -t tun0 -u $USER
#ifconfig tun0 up 10.0.23.1 netmask 255.255.255.252
ifconfig tun0 10.0.0.1 pointopoint 10.0.0.2
echo 1 > /proc/sys/net/ipv4/ip_forward
#iptables -t nat -A POSTROUTING -s 10.0.23.2 -j SNAT –to-source $SERVER_IP
#iptables -A FORWARD -d 10.0.23.2 -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.0.0.2 -j SNAT –to-source $SERVER_IP
iptables -A FORWARD -d 10.0.0.2 -j ACCEPT
eof
 
echo “Press any key to kill (tid:$tid)”
read
sudo kill -KILL $tid
sudo rmmod tun
cat << eof | ssh root@$HOST 
rmmod tun
iptables -F
echo 0 > /proc/sys/net/ipv4/ip_forward
eof