MailMe: how to get notified automatically via email through BASH

In my job, I have to perform calculations on a remote machine to which I connect through ssh. It happens quite often that I have to run a few jobs overnight, if not for days. I could check manually for their termination, but I came up with a better way.

In a previous post I described how useful it can be to run script in background on the machine without having to be logged-in. Combining the power of screen with the utility of mail we can create a script which will automatically send an email whenever the job is finished.

The Script

#!/bin/bash
#
##
## This scripts sends an automated email as soon as a job is finished
##
## ***   Use with screen:
## ***  
## ***   $ screen -S [job-id]               (start screen)
## ***   $ mailme [job-id]                  (run mailme in screen)
## ***   Ctrl+a d                           (detach)
## ***   Ctrl+d                             (kill screen, from session)
## ***   $ screen -ls                       (check running screens)
## ***   $ screen -r [screen-id]            (resumes session)
## ***   screen -X -S [screen-id] kill      (kill screen, from outside)
## ***  

# Need argument: job id which needs to be checked for completion
if [ -z "${1}" ] ; then
    echo "Please insert job ID as argument"
    echo "Exiting status 1"
    exit 1
fi

# Need second argument: email
if [ -z "${2}" ] ; then
    echo "Please insert email as second argument"
    echo "Exiting status 1"
    exit 1
fi

# Put your email here
zzEMAIL="${2}"
zzJOBID="${1}"
zzFREQ=3600 #Frequency of checks, in ms
zzEND=true

# Loop that checks job status
while "${zzEND}"
do

    zzSTRING=$(bhist -d | grep "${zzJOBID}")
   
    if [ ! -z "${zzSTRING}" ] ; then
   
    zzEND=false
   
    echo -e "Dear Sir,\n\nThis is an automatic email from grep Linux indicating that the job ID\# ${zzJOBID} has been terminated (succesfully or not).\n\nThis is the output:\n\n $(bhist -d) \n\nBest, \ngrep Linux | mail -s "Job ${zzJOBID} Completed!" "${zzEMAIL}"
   
    else
   
    sleep "${zzFREQ}"
   
    fi
   
done

exit 0 

The comments should provide a satisfactory explanation, but it could be used straight away for jobs submitted through the routine bsub.

You would just need to run the script from a screen and the game is on. If you do not remember how to do this, I have put some reminders in the header section of the script, or you can check my guide to screen.

You can run multiple of these scripts in multiple screen sessions if more jobs need to be tracked.

Afterthoughts

This script has been designed to check job progress and send an email  whenever the job is complete. Through an easy modification of the condition to be checked in the variable zzSTRING in the loop, anything can trigger the delivery of an automated email, potentially making the scope of this script very broad.

Have a play with it and let me know how you use/modify it!

 

Motion: use your webcam as a security camera!

Last month, my landlord told me he had to send an engineer to fix the heating in my room. I could not be at home at the time they would come, but they would have access anyway.

This is one of the many occasions in which you wish you had a hidden security camera at hand (together with when you want to know if you roommate is really stealing your chocolate and when you suspect there are undesired animals in your room!).

But the solution is just in front of you, right now! Your computer's webcam is a hidden and usually unsuspectable object and it can be easily turned into a monitoring device.

And Linux helps us doing it very easily with a small program called motion.

Motion saves snapshots from your webcam whenever movement is detected by the webcam itself. Increasing the capture rate, it can produce decent videos as well.

Configure

Motion needs to be configured before it can be used. Configuration allows for a big list of customizations, as frame capture rate, image resolution of the frames, motion sensibility, webcam properties and many others. It is very easy as most of it is already configured.

First of all, copy the config file in your user account. From a terminal:

$ cd
$ mkdir .motion
$ sudo cp /etc/motion/motion.conf ./.motion

Then change the ownership of the file to your user (it was copied from root)

$ sudo chown [username] ./.motion/motion.conf

 And open it with your favourite text editor (which is emacs, of course):

$ emacs ./.motion.conf


In the ideal case, you would not need to modify anything. But it would be safe to skim through the file to see if there are things you want to change. Comments (line starting with #) are very exhaustive and it is difficult to get lost. The most important would be:

videodevice /dev/video0

which selects the webcam device used, or the directory where the snapshots are saved:

target_dir /home/user/your_folder

Usage

Funnily enough, using the program is easier than configuring it. Just open a terminal and run the command:

$ motion

You will see a lot of gibberish-looking lines and if you are in front of your webcam, you can test it straight away, as it will be saving pictures already, if you are moving.

To close the program when you are finished just press

ctrl-c

and you will not lose anything as the pictures will be saved in the folder you chose during the configuration stage.

Bonus

As an addition for extra security, you could connect the folder to the internet over cloud sharing services as dropbox or copy, and look for movements in your room real time! 

Screen: how to run processes in the background after log out

So, you want to run a very long process on your ssh server but you need to log out.

It sounds like an impossible problem, but as long the computer is on, why can't it run processes in the background? Well, it can, and there is a simple solution using a quite clever command-line window manager:

Screen

Screen can create multiple windows in a shell, from which you can "detach" and leave them running in the background, even if you log out of your user account.

Create a Screen

From a terminal, type:

$ screen -S [choose-a-window-name]

And you will immediately enter the window just created.
From this window you can type any command and leave it running in the background.

To detach from the window and go back to the terminal, just press the combination:

ctrl-a d

And you will go back to the terminal, where you can start more screens if needed, or even log out without stopping the job.

Access a previously created Screen

From a terminal, type:

$ screen -r [window-name]

to revert a previously created screen. If you do not remember the window name, you can list them through the command:

$ screen -ls

which gives information about the active screens and their names.

Kill a Screen

If your job is finished in one screen and you want to close it, you can go revert it and press:

ctrl-d

to close it, or you can close it from outside the screen through the command:

$ screen -X -S [window-name] kill

Conclusion

Screen is a very powerful tool and its uses span a much wider scope than the ones described here. It can be regarded as complete full-screen window manager.

For ssh users, though, it is a lifesaver as it allows to run for nasty and long processes at anytime of the day without needing your presence.

How to install software on Linux using repositories

This is a beginner guide on how to install software through terminal, using package managers. I will list the most common distributions (according to Distrowatch) and their command to search and install a certain software.

Search

Ubuntu, Debian, Mint (apt)

apt-cache search [software_name]

Fedora, Red Hat, CentOS (yum)

yum search [software_name]

Mandriva, Mageia (urpmi)

urpmi -y [software_name]

openSUSE (ZYpp)

zypper search [software_name]

Arch (pacman)

pacman -Ss [software_name]

Gentoo (portage)

emerge -s [software_name]

Install

Ubuntu, Debian, Mint (apt)

sudo apt-get install [software_name]

Fedora, Red Hat, CentOS (yum)

sudo yum install [software_name]

Mandriva, Mageia (urpmi)

urpmi [software_name]

openSUSE (ZYpp)

zypper install [software_name]

Arch (pacman)

pacman -S [software_name]

Gentoo (portage)

emerge [software_name]

Update 

Ubuntu, Debian, Mint (apt)

sudo apt-get upgrade

Fedora, Red Hat, CentOS (yum)

sudo yum update

Mandriva, Mageia (urpmi)

urpmi—auto-select

openSUSE (ZYpp)

zypper update

Arch (pacman)

pacman -Syu 

Gentoo (portage)

emerge -uD [software_name]

Uninstall

Ubuntu, Debian, Mint (apt)

sudo apt-get remove [software_name]

Fedora, Red Hat, CentOS (yum)

sudo yum remove [software_name]

Mandriva, Mageia (urpmi)

urpme [software_name]

openSUSE (ZYpp)

zypper remove [software_name]

Arch (pacman)

pacman -R [software_name]

Gentoo (portage)

emerge -C [software_name]

For Slackware, there is no package manager settling dependencies for you, and you can use pkgtool or slackpkg. Although, there is also slapt-get which somehow helps getting dependencies.


If you have a distribution that is based an older one, even if not listed here, it will most probably have the need the same commands to install software (e.g. Ubuntu is a Debian-based distribution, so they share the same installation commands).

Monitor your badwidth usage: bwbar

The repositories include a nice little program to monitor your bandwidth usage. Very useful if you have a restriction on your bandwidth and you want to track your daily bandwidth usage.

The program is called bwbar. It generates a graphical readout of the bandwidth usage locally in a .png and a .txt file.

Installation

Ubuntu/Debian

$ sudo apt-get install bwbar

Fedora

$ sudo yum install bwbar

Usage

When installed, you need to modify the configuration file:

$ sudo emacs -nw /etc/default/bwbar

And uncomment the lines:

#DIR=/var/www
#RUNAS=www-data
#OPTIONS="eth0 1.5"

removing the # from the beginning of the line (it should change color from red when you do), and change the values depending on the directory you want the logs saved in, the user you want the program to run as, and other options you want to include when you run bwbar.


Then, you need to change the permissions, as following (if you kept the same folder):

$ sudo chown -R www-data:www-data /var/www/bwbar/

And then you can start the daemon by typing:

$ sudo /etc/init.d/bwbar start

You can see the bandwidth bar on http://[your-server-ip]/bwba.


If you want to track traffic, check out the package darkstat.

Mogrify: scale and crop multiple images in one command

When dealing with a lot of pictures, sometimes we want to make a common modification such as a change of resolution. In the case of multiple screenshots it is also common to crop all of the images at a fixed position (such as selecting only the same window).
Doing this job by hand, even just for three or four times, using the same parameters, is stressful and tiring, so we can ask for help to the terminal!

Mogrify is a command of the ImageMagick package and allows us to perform many operations on multiple images. In addition to scaling, resizing and cropping, mogrify can compress, convert, apply filter to images and much more.

Installation

For Fedora:

$ sudo yum install ImageMagick

For Ubuntu/Debian:

$ sudo apt-get install ImageMagick

Usage

The program overwrites the existing images, so, in order to prevent any unwanted modification, make a backup of the images or at least try the command on a sample image to make sure that it is the wanted modification.
Firstly, enter in the directory where the images that we want to modify are

$ cd folder_name

Then, say you want to scale the pictures to a resolution of 640x480:

$ mogrify -scale 640x480 *.* 

(note: if using the wildcard *.* , every single file in the folder will be modified by the command, so make sure that the folder contains only the files you want to modify or use different wildcards as *.jpg or e.g. *-2012.jpg)

If you want to scale the pictures keeping the aspect ratio, specify only the length in pixels and the height will be adjusted to keep constant the aspect ratio:

$ mogrify -scale 640 *.* 

Or using the same argument for the height instead of the width:

 $ mogrify -scale x480 *.* 

Or can also use a percentage of the original size, so if you want to halve the resolution:

$ mogrify -scale 50% *.*  

Or you can even modify only pictures bigger (or smaller) than the specified resolution and keep the others unchanged, respectively:

$ mogrify -scale '640x480>' *.*  

$ mogrify -scale '640x480<' *.*  

If you want to crop pictures in a window of 300x200 starting at pixels x=0 and y=100 where x is the offset in length from left to right and y is the offset in height from top to bottom, you need to use another option.
Since it is difficult to know the offset precise values a priori using gimp (a graphical editor) to take these values from the cursor position or selection can help. Then, use:

$ mogrify -crop 300x200+0+100 *.* 

If you want to compress images, use:

 $ mogrify -compress JPEG -quality 85 *.jpg 

where JPEG is the compression type and 85 is the quality for JPEG compression, which varies from 0 (lowest) to 100 (highest).


Or convert images from gif to jpeg format:

$ mogrify -format jpg -quality 85 *.gif

And rotate by an angle, e.g. 90 degrees:

$ mogrify -rotate 90 *.* 

Or if you want to rotate the image only in specific occasions, as if height is bigger than length (e.g. 480x640) (and viceversa), use, respectively:

$ mogrify -rotate '90>' *.*

$ mogrify -rotate '90<' *.*

Useful Links

Check out the official page of ImageMagick on Mogrify here, where you can find with more detail all the options of mogrify.

Change language in Firefox and Thunderbird

Just after I installed Debian, I immediately felt the need to install Firefox and Thunderbird, as they are my favorite web and mail browsers.

As they are not present in the debian repositories, I had to use the ones from linuxmint (I added to my /etc/apt/sources.list):

deb http://debian.linuxmint.com/latest testing main contrib non-free

And after I installed Firefox and Thunderbird, I found myself with a Firefox in Afrikaans and Thunderbird in Arabic. Why this weird combination?

It is because of a bug in the Linux Mint repositories. When you go to install the browsers, they pick the first language pack they find, in alphabetical order!

In order to go back to english, you need to remove the language packs, using the package manager, and then install the english ones:

$ sudo apt-get remove firefox-l10n-af thunderbird-l10n-ar

This command alone actually automatically asked to install the English language packs. You can install also other languages by browsing in the repository:

$ apt-cache search firefox | grep l10n

For some reason, though, after this operations, the browsers did not work, so I tried:

$ sudo apt-get install firefox thunderbird

And everything worked like a charm, in a familiar language.

Pdftk: How to merge two pdf documents

Pdfs are an easy way to share documents through platforms without worrying about compatibility.

While this is clearly an advantage, pdfs lack the ease of modification that can be found in other documents types as Word documents. Essentially, pdfs are easy to read, but difficult to modify.

Fortunately, in Linux, there are many tricks to make the job easier. There is a piece of software called pdftk which allows great manipulation of pdfs.

Installation

Using Fedora:

$ sudo yum install pdftk

Using Ubuntu:

$ sudo apt-get install pdftk

 A few tricks to warm up

Pdftk allows to merge or split pdfs, rotate pdf pages, apply watermarks, attaching files to the pdfs, bursting the pdf in single pages, compressing it and repairing it in some cases, if corrupted.

In this post I will focus on how to merge pdf documents, but there is an extensive documentation on how to perform many other operations. That can be found on the PDFLabs website linked on the bottom of the page, or in the man pages.
The syntax to merge pdf files is relatively easy:

$ pdftk [input pdfs] cat output [output file name]

But its power comes from the possibility to use handles or wildcards. I'll provide a few examples.

To merge two pdf files into a file called combined.pdf :

$ pdftk file1.pdf file2.pdf cat output combined.pdf

If you want to merge all pdf files in the current directory can use the  wildcard *:

$ pdftk *.pdf cat output combined.pdf

 If you want to merge only specific pages of a pdf file with other pages of another pdf files (or the same) handles come handy:

$ pdftk A=file1.pdf B=file2.pdf cat A1 A2-10 B2 A3-7odd output combined.pdf

This may seem a bit complicated but I will explain: the input files are now called as two variables A and B, which are the handles. The codes in yellow are the rules to merge the two files, and it is here that lies the power of pdftk. It merges the pages from each file in the order we provide them. In this case:

   A1 = the first page of file1.pdf
   A2-10 = the pages from 2 to 10 of file1.pdf
   B2 = the second page of file2.pdf
   A3-7odd = the odd pages in the range from 3 to 7 of file1.pdf  (i.e. 3,5,7)

The even qualifier can be also used instead of odd, to take even pages only.

Main problem and solution

I came across pdftk when I needed to solve this problem:

I had scanned a set of papers through a fast document scanner, but since they were written on both sides, I found myself with two big pdfs, the first one containing odd pages only and the second one, containing even pages only, in reverse order!
I wanted a single file with the pages in the correct order but I could not find an easy way using a single command even using pdftk.

*** Update: This command would do the job for pdftk versions higher than 1.44:

 pdftk file2.pdf cat end-1 output - | pdftk file1.pdf - shuffle output combined.pdf

 Thanks to Andrew Peace ***

After some thinking I surrendered to the power of bash!

I decided to create a small bash script to form the command I needed to merge and order automatically the pages in a single file. For those who have done some basic bash, will immediately recognize the logic behind the script. For 50 pages pdfs:

#!/bin/bash
echo -e ' pdftk A=AQT.pdf B=AQTb.pdf cat \c'

max=50
for loop1 in {1..50}
do
    for loop2 in {A..B}
    do
    n=0
    if [[ ${loop2} == "A" ]] ; then
        let "n=$loop1"
    elif [[ ${loop2} == "B" ]] ; then
        let "n=$max-$loop1+1"
    fi
    echo -e  ${loop2}${n} '\c'
    n=0
    done
done

echo -e ' output combined.pdf \c'

Or you can download the script here and run it by simply typing:

$ ./pdftk_merge.sh

from the folder in which you downloaded the file.

The script prints automatically to the screen the command that you need to type (or better, copy) to merge the two set of pdfs! In this case the result is

pdftk A=file1.pdf B=file2.pdf cat A1 B50 A2 B49 A3 B48 A4 B47 A5 B46 A6 B45 A7 B44 A8 B43 A9 B42 A10 B41 A11 B40 A12 B39 A13 B38 A14 B37 A15 B36 A16 B35 A17 B34 A18 B33 A19 B32 A20 B31 A21 B30 A22 B29 A23 B28 A24 B27 A25 B26 A26 B25 A27 B24 A28 B23 A29 B22 A30 B21 A31 B20 A32 B19 A33 B18 A34 B17 A35 B16 A36 B15 A37 B14 A38 B13 A39 B12 A40 B11 A41 B10 A42 B9 A43 B8 A44 B7 A45 B6 A46 B5 A47 B4 A48 B3 A49 B2 A50 B1  output combined.pdf

Which, if executed from terminal, will correctly merge the two files in the right page order.

Useful links

Pdftk is a Pdflabs product: http://www.pdflabs.com. Many more examples on how to use it can be found here.

Problem with Grsync and (partial) remedy

I seize the opportunity of a recent problem with Grsync to talk about it.

Grsync is an useful GUI for the command rsync.
This command is very powerful, since it synchronizes two folders that can be on two different machines, or the same machine. Usually, if on the same machine, rsync is used for backup purposes.
Grsync provides a lot of options to make synchronizations and backups and is really user friendly: it only needs source and destination folders and some options to be set.

The synchronization consists in making the destination folder equal to the source folder.
This is a very common task, since it is common that a folder on a usb stick is used on other PCs and then we want to copy this folder on the PC at home. Copying the entire folder every time will be a slow method, especially if we are handling with gigabytes of data! Copying the files one by one requires a lot of memory and patience. Grsync requires just one click to identify (there are different methods that I will not list, for now) the differences between the two folders and copy only the files different, saving time, memory (human in this case) and cpu.


The usage is very simple: source and destination paths in the relative space and a few options; I'll explain the main ones, but a mouse roll-over the option gives a sufficient explanation of the option:
- Preserve time/owner/permissions/group: the destination files will have respectively the same modification time, the same owner (be careful, this is only a super-user option... in doubt un-tick it), the same permissions on the files (read-write-execute) and the same group.
- Delete on destination: makes a perfect synchronization, since file deleted in the source folder will be deleted also in the destination folder. Essential to have a faithful copy of the source, but to un-tick for incremental backups.
- Verbose: every deleted and copied file, error or message in general will be reported.

The next step is installing it! In Fedora you can use yum from a terminal:

sudo yum install grsync

or download and compile it from the official website.


---


Now let's focus on the problem: Grsync is not working now.
Why?
A conflict (maybe) with the gtk+ update (you don't need to know what gtk+ is, but if you really want... here).
Apparently, the new version of gtk: gtk+2.0-2.16.6 is not letting work grsync 0.9.1. The symptoms are different:
- Grsync window that won't open, but will be silently running in the processes.
- Message error, if grsync is run from terminal:

(grsync:#####): Gtk-CRITICAL **: gtk_combo_box_append_text: assertion `GTK_IS_LIST_STORE (combo_box->priv->model)' failed

repeated for how many grsync sessions are defined.
The problem has been discussed and there isn't still solution as reported here.
I absolutely don't have the solutions for this problem, since my beginner abilities could only help someone to explain or discover new cool programs as in this post.
Anyway, I've got two remedies, until someone find a radical solution to the problem.


The first remedy is pretty lame... but necessary if you want to stick to the program: a downgrade.

If you installed grsync with yum, that's very easy to do:

yum downgrade grsync

and it will downgrade grsync to the older version (0.6.3-2) that is working with the gtk+ update.


The second remedy is much more cool: using directly the rsync command in a terminal.
Rsync has a very easy syntax:

rsync [options] [source] [destination]

and it features a huge number of options. Check out the man pages to know more about this splendid command.
To automatize backups, you can also create a script. This is a BASH example that I use (I'm a BASH beginner too :D):

#!/bin/bash
#
# Syncronize using rsync "src" folder on the PC (source) with the "dest" folder
# on the stick (destination).
#

echo "Starting..."
rsync -vrlpEogth --delete --progress /home/user/src /media/usbstick/dest
exit

I'll introduce BASH and scripts in another post if you're not familiar with that.


That's all for now.

See you soon!


*** UPDATE ***
Thanks to the developer Piero Orsoni, the new version of the program, Grsync 0.9.2, is now full-working.
The version will be soon available also in the repositories.