Update a virtualenv on OSX

If you are using homebrew to manage your python toolchain on OSX, there will likely come a time where existing virtual env break because an older version of python has been deleted from the system. Below is a quick recipe to fix them.

NB: this is basically wiping out the virtualenv and recreating a new one. You need to ensure that your requirements.txt file is up to date so all dependencies can be restored.

For python 2.7

rm -rf bin/ lib/ include/ .Python
virtualenv --python=python2.7 ./
source bin/activate
pip install -r requirements.txt

For python 3.4

rm -rf bin/ lib/ include/ .Python
virtualenv --python=python3.4 ./
source bin/activate
pip install -r requirements.txt

Python script to generate a password file with salted SHA1

I have been looking around to generate a salted SHA1-encoded password file for nginx.

The existing tools I found did not do the trick:

  • htpasswd generates salted password using the crypt function (only 8 char counts) or an unsalted md5 password

I ended up putting together a small python script to get things done

import subprocess
import hashlib
import base64

username=raw_input("username:")
password =raw_input("password:")

# use openssl to generate securly a salt
pipe = subprocess.Popen("openssl rand -base64 3", shell=True, bufsize=100, stdout=subprocess.PIPE).stdout
salt = pipe.readline().rstrip()
pipe.close()

m = hashlib.sha1()
m.update(password+salt)

hash = m.digest()+salt

print "copy paste the following line to your password file"
print username+":{SSHA}"+base64.b64encode(hash)

Disable Alt key modifiers in Emacs MacOS X

I am the happy user of a Macbook Pro with a French keyboard on MacOS X.

One of the main issues I have had to date, is that Apple’s French keyboard has no direct key for \ ~ { }. You need to learn a ALT + keyboard strokes.

However when using Emacs, the ALT key is used to trigger commands and this can become a real issue.

The trick is (courtesy of Morten Rasmussen) is to add the following lines to your .emacs config file:

(when (eq system-type 'darwin)
  (setq mac-right-option-modifier 'none))	

After restarting Emacs, it will no longer catch the right ALT key to trigger commands and allow you to use it to input special characters.

Not loosing money with the Amazon Market place

I have been looking at selling some used books and DVDs on the Amazon.co.uk market place. Prices are pretty tight and if you miscalculate the shipping costs or miss one of the fees it is very easy to lose some money. So how does this work ?

First, the buyer has to pay a shipping fee on top of the purchase price.

Then Amazon retains 3 different fees (details are on this page):

  • Completion fee: £0.86 per transaction
  • Referral fee: A % of the item price (17.5% for a book)
  • Variable closing fee: Amazon keep of share of what is charged as shipping fee to the buyer.

Finally you have to pay packaging and shipping costs.

I build small spreadsheet to compute my profit on those transactions and avoid making a loss. I have included the fees for Amazon.co.uk and the Royal Mail fees. You can download it as an OpenOffice fill or as an MS Excel file.

One last piece of advise, always compare your forecast profit vs what you would yield from using the amazon “trade in” program

Enabling hardware encryption on QNAP TS-212 box

It took me a few hours (more if you count the compilation time) to enable the hardware encryption on a QNAP TS-212 box running Debian Wheezy. It is powered by a Marvell Kirkwood ARM chip which includes a hardware crypto co processor. There is quite a lot of doc out there but a there are a couple of tricks that are sometime omitted. I thought it would be nice to document it for later use.

The key services that need to use the crypto-coprocessor are openssh and openvnp. Both rely on openssl so if we can trick openssl to use co-processor we are done.

Step 1: pull the kernel development dependencies

We need to build an extra kernel module that is not provided by the default Debian install

$ sudo apt-get install linux-headers-kirkwood	

Step 2: Build and install the cryptodev module

Download the source of the cryptodev kernet module from http://cryptodev-linux.org/ then build it using

# make 
# sudo make install

Step 3: Load the module

You need to load the module before trying to build openssl as we need to ensure that the self tests that are part of the build process are running without error.

# sudo modprobe cryptodev
# echo cryptodev >> /etc/modules

Step 4: Build openssl

# apt-get source openssl 
# cd openssl-1.0.1e

Edit debian/rules and add “-DHAVE_CRYPTODEV -DUSE_CRYPTODEV_DIGESTS -DHASH_MAX_LEN=64” to the config line

Replace the crypto/engine/eng_cryptodev.c with the one that is included in the cryptodev module package extras/eng_cryptodev.c

You can now proceed with the build

# debian/rules binaries
# debian/rules package 

It might be worth using screen to run the compilation. It takes quite a lot of time on the TS-212 and if your SSH session dies, the process will be interrupted.

If there are errors during the self test it might be that you forgot to override the eng_cryptodev.c file

Step 5 (optional): install openssl

It this stage you might want to run some speed test.

# openssl speed -elapsed -evp aes-128-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 1566091 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 483296 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 127521 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 32463 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 4077 aes-128-cbc's in 3.00s
OpenSSL 1.0.1e 11 Feb 2013
built on: Mon Mar 18 21:53:12 UTC 2013
options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) blowfish(ptr) 
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DTERMIO -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -Wl,-z,relro -Wa,--noexecstack -Wall
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc       8352.49k    10310.31k    10881.79k    11080.70k    11132.93k

# openssl speed  -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 1557057 aes-128-cbc's in 2.91s
Doing aes-128-cbc for 3s on 64 size blocks: 482900 aes-128-cbc's in 2.90s
Doing aes-128-cbc for 3s on 256 size blocks: 127854 aes-128-cbc's in 2.90s
Doing aes-128-cbc for 3s on 1024 size blocks: 32283 aes-128-cbc's in 2.89s
Doing aes-128-cbc for 3s on 8192 size blocks: 4072 aes-128-cbc's in 2.91s
OpenSSL 1.0.1e 11 Feb 2013
built on: Mon Mar 18 21:53:12 UTC 2013
options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) blowfish(ptr) 
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DTERMIO -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -Wl,-z,relro -Wa,--noexecstack -Wall
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc       8561.14k    10657.10k    11286.42k    11438.68k    11463.17k

Step 6: install openssl

# sudo dpkg -i libssl1.0.0_1.0.1e-2_armel.deb 
(Reading database ... 58695 files and directories currently installed.)
Preparing to replace libssl1.0.0:armel 1.0.1e-2 (using libssl1.0.0_1.0.1e-2_armel.deb) ...
Unpacking replacement libssl1.0.0:armel ...
Setting up libssl1.0.0:armel (1.0.1e-2) ...
# dpkg -i openssl_1.0.1e-2_armel.deb 
(Reading database ... 58695 files and directories currently installed.)
Preparing to replace openssl 1.0.1e-2 (using openssl_1.0.1e-2_armel.deb) ...
Unpacking replacement openssl ...
Setting up openssl (1.0.1e-2) ...
Processing triggers for man-db ...

Step 7: test again

# openssl speed -elapsed -evp aes-128-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 87015 aes-128-cbc's in 2.99s
Doing aes-128-cbc for 3s on 64 size blocks: 84022 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 59190 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 39573 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 7978 aes-128-cbc's in 3.00s
OpenSSL 1.0.1e 11 Feb 2013
built on: Tue May  7 00:08:42 BST 2013
options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) blowfish(ptr) 
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DHAVE_CRYPTODEV -DUSE_CRYPTODEV_DIGESTS -DHASH_MAX_LEN=64 -DL_ENDIAN -DTERMIO -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -Wl,-z,relro -Wa,--noexecstack -Wall
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc        465.63k     1792.47k     5050.88k    13507.58k    21785.26k

# openssl speed  -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 86763 aes-128-cbc's in 0.08s
Doing aes-128-cbc for 3s on 64 size blocks: 84278 aes-128-cbc's in 0.09s
Doing aes-128-cbc for 3s on 256 size blocks: 58822 aes-128-cbc's in 0.07s
Doing aes-128-cbc for 3s on 1024 size blocks: 39493 aes-128-cbc's in 0.04s
Doing aes-128-cbc for 3s on 8192 size blocks: 7964 aes-128-cbc's in 0.00s
OpenSSL 1.0.1e 11 Feb 2013
built on: Tue May  7 00:08:42 BST 2013
options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) blowfish(ptr) 
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DHAVE_CRYPTODEV -DUSE_CRYPTODEV_DIGESTS -DHASH_MAX_LEN=64 -DL_ENDIAN -DTERMIO -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -Wl,-z,relro -Wa,--noexecstack -Wall
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      17352.60k    59931.02k   215120.46k  1011020.80k         infk

You will notice that the cryto hardware is at a disadvantage to the software solution on the smaller blocks. However it over-perform massively on larger blocks

Step 8: restart openssh and openvpn

# sudo /etc/init.dopenssh restart
# sudo /etc/init.dopenvpn restart

Blocking access to the .git folder

Why does it matter ?

git is not only a very convenient tool manage your source code and collaborate with fellow developers. A couple of hooks or a simple fabric script and git can become a very powerful toll to deploy your application to production server.

However there is a trick. When you deploy that way, the content of the .git folder might be publicly available to the whole Internet, courtesy of your webserver. That folder contains all your code, parameters, etc… You got me your passwords might be included in there. You are not alone. If that is the case for you then you are not alone.

Reading the content of the git objects file is pretty easy. The zpipe tool included in the zlib is all you need. You can download it, compile it and get exploring that .git folder of yours

$ wget https://github.com/madler/zlib/raw/master/examples/zpipe.c
$ gcc -o zpipe zpipe.c -lz
$ ./zpipe -d  < .git/objects/ff/66f....

I have gathered below a few solutions to block access to that folder using apache and nginx. Some might not fit your purpose so feel free to try, adapt and share any other way you find.

Blocking access to the .git folder with Apache

Change permissions to .git folder

A very simple way that does not require changing the configuration of the server. However it won't work if the .git folder is owned by the user running the web server

$ chmod o-rwx .git

Create a .htaccess in the .git folder

You just need to create a .htaccess file in the .git folder with the following content

Order deny,allow
deny from all

If this is not working it might be due to the fact that .htaccess are not allowed in the master configuration of Apache. Ensure that the AllowOverride directive is not set to None.

<Directory /var/www/>
	Options Indexes FollowSymLinks MultiViews
	AllowOverride All
	Order allow,deny
	allow from all
</Directory>

As this .htaccess file is located in the .git folder it cannot be checked-in. You will have to create it each time you deploy on a new server.

Block it in apache conf

If you have access to apache configuration you can add the following lines in file

<Directory /var/www/.git>
	Order deny,allow
	deny from all
</Directory>

Using apache rewriting rules

For this trick you will need to have the rewriting rule module loaded. On ubuntu / debian, you need to check that the file rewrite.load has been linked to the /etc/apache2/mods-enabled folder. If not you can enable it by running (as a superuser).

$ sudo a2enmod rewrite

Then add the following lines in .htaccess file that can be located at the root of the web folder

RewriteEngine On
RewriteRule ^(.*/)?\.git+ - [F,L]
ErrorDocument 403 "Access denied"

The trick here is that this file does not need to be located in the .git folder and can therefore be checked-in. No need to rereate it each time you deploy to a new machine.

Blocking access to the .git folder with nginx

Now nginx does not support the .htaccess file. We'll have to change the configuration of the server itself. Not that the trick of changing permissions to the .git folder will work in this case as well and will be subject to the same limitations.

In any of the server {} blocks add the following lines

location ~ /\.git {
  deny all;
}

That should do it as well