First, you need to install RPM devtools:

# yum install rpmdevtools

Then setup your local RPM build tree:

$ rpmdev-setuptree

This will create RPM build tree (~/rpmbuild/) in your home directory. Just a reminder – never, ever, under any circumstances, build RPMs as root user. Just a reminder. Obviously, you knew that already, right?

You also want to get and install additional header files and libraries, such as TCL/TK, expat, sqlite, etc, that are used to build some Python components:

# yum install tk-devel tcl-devel expat-devel db4-devel gdbm-devel sqlite-devel

Now we’re ready to start building an RPM. Get the RPM spec file from source build tree and copy it to your rpmbuild/SPECS directory, also copy the tarball to your build directory:

$ wget http://www.python.org/ftp/python/2.6.4/Python-2.6.4.tar.bz2
$ tar jxf Python-2.6.4.tar.bz2
$ cp Python-2.6.4/Misc/RPM/python-2.6.spec ~/rpmbuild/SPECS/
$ cp Python-2.6.4.tar.bz2 ~/rpmbuild/SOURCES/

This spec file is slightly broken and if you try building an RPM with it, it most likely is going to fail. So you need to patch it. The proposed patch has been submitted to python bug tracking system in the days of 2.6.1 release, but hasn’t yet been implemented. This patch doesn’t seem to work with 2.6.4 release and needs changing as well.

I have modified original 2.6.1 patch file slightly so that it can be used to build 2.6.4 Python RPM. You download it and apply to the original .spec:

$ cd ~/rpmbuild/SPECS/
$ wget http://www.grenadepod.com/wp-content/uploads/python-2.6.4.spec.diff
$ patch python-2.6.spec python-2.6.4.spec.diff

You should be OK to build the RPM now, but it’ll most likely fail with minor rpath-check errors, such as:

ERROR   0001: file '/usr/lib/python2.6/lib-dynload/_bsddb.so' contains a standard rpath '/usr/lib' in [/usr/lib]
ERROR   0001: file '/usr/lib/python2.6/lib-dynload/_sqlite3.so' contains a standard rpath '/usr/lib' in [/usr/lib]

Which indicates that some components are using hardcoded library paths. It’s not a big issue and can be ignored. Run build command as this:

$ QA_RPATHS=$[ 0x0001|0x0010 ] rpmbuild -ba python-2.6.spec

It should now build without any issues and the result is a set of python RPMs:

$ QA_RPATHS=$[ 0x0001|0x0010 ] rpmbuild -ba python-2.6.spec
...
Wrote: /home/pulegium/rpmbuild/SRPMS/python2.6-2.6.4-1gpod.src.rpm
Wrote: /home/pulegium/rpmbuild/RPMS/i386/python2.6-2.6.4-1gpod.i386.rpm
Wrote: /home/pulegium/rpmbuild/RPMS/i386/python2.6-devel-2.6.4-1gpod.i386.rpm
Wrote: /home/pulegium/rpmbuild/RPMS/i386/python2.6-tkinter-2.6.4-1gpod.i386.rpm
Wrote: /home/pulegium/rpmbuild/RPMS/i386/python2.6-tools-2.6.4-1gpod.i386.rpm
Wrote: /home/pulegium/rpmbuild/RPMS/i386/python2.6-debuginfo-2.6.4-1gpod.i386.rpm
...
$

This is it… You now have python RPMs that you can install on your CentOS 5.x server.

I’m planning to create my little own repository, where I’ll put this (and other) packages. For now, you can download python 2.6.4 source RPM if you want to change something in the spec file and rebuild it yourself.

Related posts:

1. Sorting out YUM repositories on CentOS 5.4
2. Use SSH to upgrade WordPress plugins automatically
3. Building and running Google Chrome OS on VirtualBox

By default, freshly installed CentOS 5.X has the following Yum repositories enabled:

addons                                             CentOS-5 - Addons
base                                               CentOS-5 - Base
extras                                             CentOS-5 - Extras
updates                                            CentOS-5 - Updates

These are all defined in /etc/yum.repos.d/CentOS-Base.repo configuration file. Repositories enabled by default provide you with the core CentOS packages and updates for them. So you must have them enabled if you want your updates to work correctly. It is also important that these repositories take precedence over other repositories that you are going to use.

I will show later how to use Yum priorities package, just note that these are going to be priority one repositories

Enable standard CentOS repositories

There are two useful repositories defined in the configuration, but not enabled:

• CentOS Plus. Packages in this repository contains upgraded versions of the software. If you enable this repository, after applying updates/upgrades your system will no longer be of the original version you have installed. Fear not though, all packages are tested by CentOS team and will no cause any issues. So unless you have really good reason to keep you installation at the same version level I’d recommend enabling this repository.
• Contrib. Packages supplied and maintained by CentOS users. These packages are not inspected by CentOS team, but they are not attempt to replace/modify core CentOS package set, so normally this repository should not cause any issues. Beware that some packages are not following mainstream CentOS very closely. I normally don’t have this enabled.

These two repositories are going to get priority two setting. If you wish to enable them, edit default /etc/yum.repos.d/CentOS-Base.repo Yum repository configuration file and remove (or rather comment out) “enabled=0” line.

Install and enable EPEL repository

I must mention, that in this article I’m mostly talking about CentOS installation that is used for server environment. Therefore I’m not really interested in repositories that provide packages such as DVD or other multimedia decoders and players.

One of the most useful repositories for your server environment is EPEL repository – Extra Packages for Enterprise Linux. This repository is maintained by Fedora project, and every effort is made to keep this repository as least  intrusive as possible, so in theory enabling and using this repository should not break or otherwise cause issues to your CentOS installation.

First of all, you need to install EPEL repository configuration files:

[root@centos54 ~]# rpm -ihv http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm
Retrieving http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm
warning: /var/tmp/rpm-xfer.RT5AzP: Header V3 DSA signature: NOKEY, key ID 217521f6
Preparing...                ########################################### [100%]
   1:epel-release           ########################################### [100%]
[root@centos54 ~]#

This provides you with two configuration files: base and testing. Don’t worry about testing one and leave it disabled.

Main configuration comes with three sections:

• Base packages. This is enabled by default.
• Debug packages. Disabled, and no need to enable unless you want to use debug packages.
• Source packages. Disabled. Enable only if you want to be able to install source RPMs from EPEL. Useful if you want to rebuild them to your specific needs. I’d recommend to leave it disabled and enable on yum command line only when you really need to install source RPMs.

These should be getting priority 3 setting.

Set priorities for Yum repositories

So if you enabled repositories from the previous section and installed EPEL repository configuration, here’s what you should see in you repositories list:

addons                             CentOS-5 - Addons
base                               CentOS-5 - Base
centosplus                         CentOS-5 - Plus
epel                               Extra Packages for Enterprise Linux 5 - i386
extras                             CentOS-5 - Extras
updates                            CentOS-5 - Updates

Now I need to set priorities for each repository. Install Yum priorities package:

# yum install yum-priorities

Make sure new plugin is enabled:

[root@centos54 ~]# cat /etc/yum/pluginconf.d/priorities.conf
[main]
enabled = 1
[root@centos54 ~]#

Now you can set priorities for each repository. It’s done by adding “priority=X” for each repository section in repository configuration files. Here’s what I ended up with:

[base]
...
priority=1
 
[updates]
...
priority=1
 
[addons]
...
priority=1
 
[extras]
...
priority=1
 
[centosplus]
...
priority=2
 
[contrib]
...
priority=2
 
[epel]
...
priority=3

Now you should be good to do upgrades and install packages as you see fit for your system.

Related posts:

1. Building python 2.6.4 RPM for CentOS 5.4
2. Building and running Google Chrome OS on VirtualBox
3. Use SSH to upgrade WordPress plugins automatically

Application logic is extremely simple:

• Retrieve specified page
• If cannot retrieve, try another from stored in the DB
• Store all found URLs (<a href>) in the DB
• Return one (and remove from DB)

I store all pages in memory, but for more serious crawling, consider storing data on a physical file, that way it’ll be more memory efficient. Considering the size of the internets, even if you store only URL strings you will not last for very long…

Anyway, happy crawling! (and let me know if there are any issues, so far it worked fine for me)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

#!/usr/bin/env python
 
from BeautifulSoup import BeautifulSoup
import urllib2, random, re, sqlite3, logging
 
DATABASE   = ':memory:'
LOG_LEVEL  = logging.INFO
ABS_URL_RE = re.compile('(?P<url>https?://.+?/)')
 
class WebCrawler:
    def __init__(self, logging_level=LOG_LEVEL, database=DATABASE):
        logging.basicConfig(level=logging_level)
        self.conn = sqlite3.connect(database)
        self.conn.execute('create table url_stack (url text)')
        self.conn.execute('create table url_visited (url text)')
 
    def _form_url(self, url, link):
        if link[0:4] == 'http':
            ret_url = link
        elif link[0] == '/':
            m = ABS_URL_RE.search(url)
            ret_url = "%s%s" % (m.group('url'), link[1:])
        else:
            ret_url = "%s%s" % (url, link)
        return ret_url
 
    def _pop_from_db(self):
        logging.debug("Retrieving one URL from the DB...")
        res = self.conn.execute('select url from url_stack limit 1').fetchone()
        logging.debug("Query result: %s", res)
        url = res[0]
        self.conn.execute("delete from url_stack where url = ?", (url,))
        return url
 
    def _push_to_db(self, url):
        logging.debug("Inserting record into DB...")
        logging.debug("Check if it hasn't been visited yet")
        if not self.conn.execute('select url from url_visited where url=?', (url,)).fetchone():
            logging.debug("URL not found in list of visited URLs, inserting")
            logging.debug("URL to insert: %s" % url)
            self.conn.execute('insert into url_stack values (?)', (url,))
            self.conn.execute('insert into url_visited values (?)', (url,))
        else:
            logging.debug("URL has already been visited or added for processing, skipping")
 
    def crawl(self, url):
        work_url = url
        logging.debug("Work URL: %s" % work_url)
        while True:
            try:
                logging.debug("Trying to open and parse the URL...")
                page = urllib2.urlopen(work_url)
                soup = BeautifulSoup(page)
                logging.debug("Parsed successfuly")
            except:
                logging.debug("Failed to parse, attempting to get next URL from DB")
                work_url = self._pop_from_db()
                continue
            links = soup('a')
            logging.debug("Found total of %d links (<a href=...>)" % len(links))
            for link in soup('a'):
                logging.debug("Processing link object: %s" % link)
                try:
                    if link['href'] != '':
                        self._push_to_db(self._form_url(work_url, link['href']))
                except:
                    logging.debug("An exception has occured, this may be ok (href attribute may be missing)")
                    logging.debug("  ... but can also indicate error in insert code")
            logging.debug("Finished adding URLs")
            logging.debug("Getting a new URL for processing from DB")
            work_url = self._pop_from_db()
            logging.info("Found URL: %s" % work_url)
            yield work_url
 
if __name__ == '__main__':
    wc = WebCrawler()
    for url in wc.crawl('http://www.google.com/'):
        pass

As you can see it’s very easy to use. Effectively it is an infinite generator (well, depending what you pass as initial URL). It’s then up to you what you’re going to do with the resulting URL…

One thing to bear in mind when you use it: this crawler pays absolutely no attention to the domain it’s searching. It just blindly collects links, selects one and follows it. So depending on the site link relative location it may stay for a while on a site, or may just wonder away quite quickly.

I wanted to have something that does not stick around for long on one site, so this suits me well, if you want to have something more pedantic, you may want to modify the code, so that it leaves current domain only when it has visited all pages and there are no new pages within the domain to analyse.

Related posts:

1. Changing menu order
2. Top level menu in Arras theme

• Insecure. Who knows who’s built the image and if they aren’t sending your Google login data to themselves when you login
• You’re stuck with that particular release. As I said, it’s in development, so new features and bug fixes get introduced on a daily basis

So I’ll show you how to (relatively) quickly build your very own Google Chrome OS. This instruction tells how to build Chromium OS with pre-built Chromium web browser. You will also need Sun VirtualBox.

In a nutshell:

• Install Ubuntu as a VirtualBox VM
• Download ChromeOS sources
• Build ChromeOS
• Create VMWare image
• Boot it in VirtualBox
• Enjoy

For the latest build release numbers check Chromium OS build page.

Preparation

Install Ubuntu

Do the standard installation, as you would normally do. I selected all defaults, and allocated 20GB single partition for the installation and assigned 512MB RAM.

Get OS source

Download/unpack http://build.chromium.org/buildbot/archives/chromiumos-0.4.22.8.tar.gz to /home/user/chromiumos/

NOTE! It seems that Google have removed OS tarballs and you now have to use these instructions to get the source code.

Install some additional packages required to build ChromiumOS

$ sudo apt-get install subversion pkg-config python perl g++ g++-multilib \
bison flex gperf libnss3-dev libgtk2.0-dev libnspr4-0d libasound2-dev \
libnspr4-dev msttcorefonts libgconf2-dev libcairo2-dev libdbus-1-dev
$ sudo apt-get install wdiff lighttpd php5-cgi sun-java6-fonts

OS build

Building local repository

cd ~/chromiumos/src/scripts
./make_local_repo.sh

Watch the output carefully and make sure it hasn’t failed with some errors!

Google says if the script fails, remove repo directory and call the script again. It hasn’t failed for me, so if you’ve done everything as per above you should be fine. I don’t really understand why calling the same script might make any difference…

This step is quite lengthy, so you might want to make yourself some coffee or tea. Or just take a short walk if the weather is good.

Create build environment

Another totally automated step. Just run

./make_chroot.sh

Which creates chroot’ed build environment for you. This uses all packages you downloaded in the previous step. There are adoption how to pull required packages from the remote repositories (Google and official Ubuntu), but I advise to take an easy way and download all packages first and build locally.

Get Chromium binary

make the following directory:

mkdir -p ~/chromiumos/src/build/x86/local_assets

And download chromium package from Google:

wget -O ~/chromiumos/src/build/x86/local_assets/chrome-chromeos.zip \
http://build.chromium.org/buildbot/archives/chromium-chromiumos-r32516.zip

Building OS

First you need to enter you chroot’ed build environment. Use the following command:

./enter_chroot.sh

I also recommend generating password for shared user, so that you can sudo from the terminal:

./set_shared_user_password.sh

And finally build all packages:

./build_all.sh

At this point, go and make some more tea or coffee. Which I wouldn’t recommend, though. Simply because you will have trouble getting asleep. Because sleeping is the most sane thing you might want to do at this moment. Building OS packages takes ages!…

But seriously, it’s not that bad, it took about an hour and a half on my Ubuntu VM to build it.

Make VM image and boot it in VirtualBox

Build bootable image

Once all packages have been built, you need to create OS image to boot from. Image build process creates two artefacts:
- Master boot record (mbr.image)
- Root FileSystem (rootfs.image)

./build_image.sh

Once the image files have been creates, the script will tell you where to find them. It is going to be in ~/chromiumos/src/build/images//

Here’s what you will have once the image build is done:

pulegium@ubuntu:~/chromiumos/src/build/images/999.999.33509.212332-a1$ ls -lh
total 729M
-rw-r--r--  1 pulegium 5000  512 2009-12-01 21:34 mbr.image
-rw-r--r--  1 root     root  40K 2009-12-01 21:31 package_list_installed.txt
-rw-r--r--  1 root     root  40K 2009-12-01 21:34 package_list_pruned.txt
drwxr-xr-x 22 root     root 4.0K 2009-12-01 21:29 rootfs
-rw-r--r--  1 pulegium 5000 950M 2009-12-01 21:23 rootfs.image

Tell VirtualBox to use existing image

Add new image

Image added

Selection of preinstalled application. All web based and ready to go. Make sure you have connection to the internet. Chrome OS bit dull and TBH useless without internet...

In general I think Chrome OS looks OK'ish, but this menu smells of M$ Windows...

Related posts:

1. Sorting out YUM repositories on CentOS 5.4
2. Building python 2.6.4 RPM for CentOS 5.4
3. The most unusual photocameras

Hide your identity

Well, first of all you need to hide details about who you are, or rather what your webserver is. It is a good practice to always run on the latest security patch, but not always feasible. So if you can’t upgrade in time, at least make attackers life harder by hiding details about your server:

#
# ServerTokens
# This directive configures what you return as the Server HTTP response
# Header. The default is 'Full' which sends information about the OS-Type
# and compiled in modules.
# Set to one of:  Full | OS | Minor | Minimal | Major | Prod
# where Full conveys the most information, and Prod the least.
#
ServerTokens Prod

#
# Optionally add a line containing the server version and virtual host
# name to server-generated pages (internal error documents, FTP directory
# listings, mod_status and mod_info output etc., but not CGI generated
# documents or custom error documents).
# Set to "EMail" to also include a mailto: link to the ServerAdmin.
# Set to one of:  On | Off | EMail
#
ServerSignature Off

Allow only basic HTTP methods

HTTP protocol defines GET, HEAD, POST, PUT, DELETE, OPTIONS, TRACE and CONNECT methods. Guess how many of those are actually used (intensively). Yup, only two. Most of the webservers would do just fine with only GET and POST methods. You might however find that you need more, so enable them as you see fit. In the example below I only allow two basic, commonly used methods:

<Location />
   <LimitExcept GET POST>
     Order allow,deny
     Deny from all
   </LimitExcept>
 </Location>

Disable old and insecure SSL

Use only new protocols and only strong ciphers.

SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM

Disable modules that you don’t need

Again, this depends on your installation and what you’re actually using, but in most of the cases most of the modules that Apache loads by default are not needed. Search for LoadModule instruction and remove anything you don’t need. Good list to start:

• mod_imap
• mod_include
• mod_info
• mod_userdir
• mod_status
• mod_cgi
• mod_autoindex
• mod_dav

Other settings

Reduce timeout, which is 300 seconds by default. Meaning that the server waits for 5 minutes before it decides that the client is no longer there. Reduce it to something sensible, like 20-30 seconds to avoid potential DDoS attacks.

Timeout 20

Disable directory browsing for any directory that has no index file:

Options -Indexes

Related posts:

1. Securing WordPress
2. Using OpenID for authentication in Django
3. Use SSH to upgrade WordPress plugins automatically

Build and install SSH2 libraries

Depending on your linux distribution you might need to use different method. On my Debian, I had to use the following to install:

#  wget http://downloads.sourceforge.net/project/libssh2/libssh2-1.2.1.tar.gz?use_mirror=kent
#  tar zxf libssh2-1.2.1.tar.gz
#  cd libssh2-1.2.1
#  ./configure
#  make
#  make install

What’s important here is that I had to build libssh2 from sources manually. However I hate doing this, it was apparently the only way. Aptitude was only offering me an older (0.12) version of the library, which failed to build PHP ssh2 extension.

Build and install PHP SSH2 extension

Now again, for some reason simple command failed to work for me… So I had to specify beta channel to install PHP SSH2 extension. Fear not though, just try this

#  pecl install ssh2

And if it doesn’t work, then do this

#  pecl install channel://pecl.php.net/ssh2-0.11.0

Simple, isn’t it?

Generate SSH public and private keys

You need to generate both public and private keys that are going to be used to connect to your server (even if it is the same server your connecting from!). Go to your home directory:

$ cd .ssh
$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/user/.ssh/id_rsa.
Your public key has been saved in /home/user/.ssh/id_rsa.pub.
The key fingerprint is:
fe:3b:5d:94:53:1e:d3:9f:87:45:73:ab:8d:9f:d7:cc user@server
$ cp id_rsa.pub authorized_keys

Private key is used to decrypt the data, whereas public key is used by the remote host to encrypt the data. You also need to create authorized_keys file, so that server knows your key is trusted and allows you to login without using actual user account password.

There is one annoying bit though. Apache user needs to be able to read both private and public keys. Normally they are kept secure in user’s .ssh/ directory, which is readable by user only, and allowing all to see it, is not a particularly good idea. So I had to copy both files to /etc/wordpress/ and make them readable to www-data group:

#  cd /etc
#  mkdir wordpress
#  cp /home/user/.ssh/id_rsa* wordpress/
#  chgrp www-data wordpress/*
#  chmod 640 wordpress/*

Configure WordPress to use public keys automatically

Add the following lines to your wp-config.php file, so you’re not asked any passwords or server names during the upgrade:

define('FTP_PUBKEY','/etc/wordpress/id_rsa.pub');
define('FTP_PRIKEY','/etc/wordpress/id_rsa');
define('FTP_USER','user');
define('FTP_PASS','');
define('FTP_HOST','localhost:22');

Related posts:

1. Securing WordPress
2. Building python 2.6.4 RPM for CentOS 5.4
3. Building and running Google Chrome OS on VirtualBox

We’ve all been in this situation. And for me personally, I would steer away from the site that forces me to register just to see some part of forum or similar. Mostly this is because I don’t want to provide my details and follow lengthy registration process.

I don’t mind to have an account (even though I forget about the website next day), but the whole process of typing in all the stuff, deciphering captcha images, logging in to my email account just to confirm registration, etc, etc. Why don’t they use OpenId? These days if you have Yahoo or Gmail account (and just who doesn’t?) you can use them to login to all websites that support OpenId.

In a nutshell, OpenId is a decentralised authentication standard. User (U) who has an OpenId account with a trusted company (C) wants to login to a website (W). W prompts with a login screen. U enters his/her OpenId URL (no personal information here provided at all). W then redirects to C login screen, where U is prompted to allow access to some parts of the information (usually it’s username only, sometimes email). If U is happy to provide this information, he then clicks login button and information is passed back to W. In this scenario, W will never get any personal information about U without explicit permission. And U benefits from having only one login that works for all websites.

Below is a simple diagram that shows how authentication is done when using OpenId to login to the websites:

Authentication with OpenID flow

Unfortunately Django does not have native support for OpenId, and their website recommends using django-authopenid library to perform authentication and login using OpenId.

Install required libraries

You need to install python-openid library first. Installation is a breeze, however there seems to be some issues with internal version number in the package, but let’s assume it’s all fine:

# easy_install python-openid
Searching for python-openid
Reading http://pypi.python.org/simple/python-openid/
Reading http://openidenabled.com/python-openid/
Reading http://www.openidenabled.com/openid/libraries/python/
Best match: python-openid 2.2.4
Downloading http://openidenabled.com/files/python-openid/packages/python-openid-2.2.4.zip
Processing python-openid-2.2.4.zip
Running python-openid-2.2.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Xyxx61/python-openid-2.2.4/egg-dist-tmp-oexXVf
zip_safe flag not set; analyzing archive contents...
Adding python-openid 2.2.4 to easy-install.pth file
 
Installed /usr/lib/python2.5/site-packages/python_openid-2.2.4-py2.5.egg
Processing dependencies for python-openid
Finished processing dependencies for python-openid
# python
Python 2.5.1 (r251:54863, Jul 31 2008, 23:17:43)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> openid.__version__
'2.2.1'
>>> openid.__path__
['/usr/lib/python2.5/site-packages/python_openid-2.2.4-py2.5.egg/openid']
>>> openid.version_info
(2, 2, 1)
>>>

Then install django-authopenid. Installation seems to spit out some errors, but I guess these can be ignored as they don’t seem to cause any issues:

# easy_install django-authopenid
Searching for django-authopenid
Reading http://pypi.python.org/simple/django-authopenid/
Reading http://hg.e-engura.org/django-authopenid/
Reading http://code.google.com/p/django-authopenid/
Best match: django-authopenid 1.0.1
Downloading http://pypi.python.org/packages/source/d/django-authopenid/django-authopenid-1.0.1.tar.gz#md5=93d44b4ce40de55bed36c9ed292adb49
Processing django-authopenid-1.0.1.tar.gz
Running django-authopenid-1.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-UtVC7d/django-authopenid-1.0.1/egg-dist-tmp-oWjnYf
warning: no files found matching 'CHANGES.MD'
Sorry: TypeError: ('compile() expected string without null bytes',)
...
Adding django-authopenid 1.0.1 to easy-install.pth file
 
Installed /usr/lib/python2.5/site-packages/django_authopenid-1.0.1-py2.5.egg
Processing dependencies for django-authopenid
Finished processing dependencies for django-authopenid

Prepare Django application environment

There are several things you’d need to do:

1. Create directory for application files

I will use /var/app/application/ in this example

2. Create directory for static content such as JS, CSS, HTML and images

I will use /var/www/application/ in this example

3. If you are using Apache web server add Virtual host definition and specify where web root is and when to pass control to mod_python module (note that all requests that do not start with /static/ will be passed to Django):

    ServerName test.example.com
    DocumentRoot /var/www/application/
    ErrorLog /var/log/apache2/application-error.log
    CustomLog /var/log/apache2/application-access.log combined
    SetHandler mod_python
    PythonHandler django.core.handlers.modpython
    PythonPath sys.path+['/var/app/']
    SetEnv DJANGO_SETTINGS_MODULE application.settings
    SetEnv PYTHON_EGG_CACHE /tmp
    <Location "/static/">
        SetHandler None
    </Location>

4. Also create database and configure appropriately in settings.py.
5. Enable administration application for Django

Enable OpenID support

These were the basic steps to get your Django environment set-up, now let’s enable OpenID module:

1. In settings.py add OpenID middleware, application and also tell Django where to find templates

MIDDLEWARE_CLASSES = (
    ...
    'django_authopenid.middleware.OpenIDMiddleware',
)
INSTALLED_APPS = (
    ...
    'django_authopenid',
    'registration',
)
LOGIN_URL = "/account/signin/"
TEMPLATE_DIRS = (
    ...
    os.path.join(os.path.dirname(__file__), 'templates')
)

2. In urls.py just include urls from django_authopenid package:

urlpatterns = patterns('',
    ...
    (r'^account/', include('django_authopenid.urls')),
)

3. Add the following context processors list (majority is what’s loaded by default anyway, you just need to add one line, but since it’s a tuple, you cannot modify it, so you must redefine it):

   TEMPLATE_CONTEXT_PROCESSORS = (
      'django.core.context_processors.auth',
      'django.core.context_processors.debug',
      'django.core.context_processors.i18n',
      'django.core.context_processors.media',
      'django.core.context_processors.request',
      'django_authopenid.context_processors.authopenid',
   )

4. Copy templates from the examples folder. Of course you’ll want them eventually to be fully integrated with your site, so that they match styles or even be incorporated in other pages, but for now, let’s just use the ones provided with the package. Location of the example templates obviously depends on your Linux distribution. On my Debian installation it was in /usr/lib/python2.5/site-packages/django_authopenid-1.0.1-py2.5.egg/example/templates. Copy examples/ directory to your Django application directory.

Test it

Once you’re done, test it. Navigate to http:///account/signin/ and if you followed this manual you should see something like this:

Django authopenid login screen

You may wonder what the login prompt on the left side is doing there. This is because Django authopenId logoc is to associate your OpenId account with local account and use standard Django “registration” middleware classes to handle all user management functions. If user does not exist – after you have selected your OpenID URL you will be asked to provide local username. You don’t need to memorise nor use it – it’s just to allow authopenId create association.

This may feel bit unintuitive, and I don’t particularly like this. I’m going to make some changes so that internal association is hidden from the end user and happens behind the scenes. This way my users are going to see only OpenId selector and login screen.

Related posts:

1. Python web crawler
2. How to use generic views in Django
3. Building and running Google Chrome OS on VirtualBox

Software development is really complex process and although  WordPress developers take security very seriously, you should also take extra measures to ensure safety and security of your blog/web site.

There are few simple steps to make your WordPress installation lot harder for attacker to compromise.

WordPress software

Always keep up to date. Flaws in security model are being identified and addressed immediately as soon as they are reported. So it’s important for you to always keep your WordPress installation up to date. It’s very easy to do now that WordPress has automatic update feature, where all you have to do is just to tell it to install the newer version of it.

File permissions

You need to make sure that webserver can modify only those files that it is allowed to. Do not rely on WordPress to enforce this, use file system permission model. All files in WordPress installation need to be owned and writeable to by the user that installed the system and not the user which is used to run webserver. Only exceprion to this is /wp-content/ directory, which contains uploaded contents.

Make sure you perform all actions in whatever your WordPress installation directory is, and not outside of it!

Let’s make all files owned by your user and set the group to web server group:

$ sudo chown -R myuser.www-data *

Then change all file permissions so that files can be written to by your user only, and read-only by other users:

$ find . -type d -exec chmod 755 {} \;
$ find . -type f -exec chmod 644 {} \;

Finally allow group write for wp-content/ directory, so that web server can do automatic updates for plugins and user content could be uploaded:

$ chmod -R g+w *

Secure wp-admin access

WordPress recommend using additional plugins and HTTP authentication to provide additional security to the administration pages, but I think this is not necessary if you implement the following two security measures: enforce SSL only traffic to /wp-admin/ and allow access only from certain IP addresses.

Make /wp-admin/ available on SSL connection only, so all traffic to and from (including passwords) is encrypted. This prevent attackers hijacking traffic and intercepting passwords and other sensitive data.

This may sound bit complicated, but bear with me, it’s not that scary as it may look like. So you will need two <VirtualServer> directives: one for normal web traffic and one for SSL.

In default HTTP definition, you then need to make a special case for /wp-admin/ URL, and enforce redirection to HTTPS, so whenever you try to access wp-admin/ using http:// you will be redirected to https:// instead. HTTPS VirtualHost on it turn has instructions to deny access from all, but only the IPs listed in the configuration:

<VirtualHost server_ip:80>
    ServerName example.com
    ServerAlias www.example.com
    DocumentRoot /var/www/virtual/www.example.com
    ErrorLog /var/log/apache2/www.example.com-error.log
    CustomLog /var/log/apache2/www.example.com-access.log combined
    <Location /wp-admin/>
        RewriteEngine on
        RewriteRule ^(.*)$ https://%{SERVER_NAME}/wp-admin/ [R=permanent,L]
    </Location>
</VirtualHost> 

<VirtualHost server_ip:443>
    ServerName example.com
    ServerAlias www.example.com
    SSLEngine on
    SSLCertificateFile /etc/ssl/certs/example.com.pem
    SSLCertificateKeyFile /etc/ssl/private/example.com.key
    DocumentRoot /var/www/virtual/www.example.com
    ErrorLog /var/log/apache2/www.example.com-error.log
    CustomLog /var/log/apache2/www.example.com-access.log combined
    <Location /wp-admin>
        Order deny,allow
        Deny from all
        Allow from trusted_ip_1
        Allow from trusted_ip_2
    </Location>
</VirtualHost>

Other security measures

Install WP Security scan plugin which will provide a good overview of how your installation looks like from the security point of view.

Also remove advertising of the WordPress version that you are using. Add the following line to functions.php file, which you are using:

remove_action('wp_head', 'wp_generator');

And did I mention that you need to make regular backups?…

Related posts:

1. Use SSH to upgrade WordPress plugins automatically
2. Basic Apache security
3. Essential WordPress plugins

Right, back to business. So I decided to write a WordPress plugin. There were few reasons I wanted to do that:

• See how difficult it is and what WordPress looks like under the hood
• Since I’m using WP, which is a PHP shop I needed to remind myself how PHP looks like
• Get some more publicity and bump up the Google rating (that’ll never hurt, will it?)

So, I remembered seeing this neat trick of reversing, or flipping over text by replacing regular ASCII latin characters with Unicode ones that resemble a mirror image of the original ones. There are numerous websites out there on the wild internets that would swallow your piece of texts and then spit out same text, but flipped over.

As you can see it is still text information, no magic with images or anything like that. You can select, copy-paste, etc.

So, I thought, right, let’s see if there was someone smart who has created a plugin for WordPress that would do this. Not that I’m a big fan of flipped over text, but just to make sure that I’m not reinventing the wheel. At the end of the day, what’s the point in creating something that’s already been build. If such plugin existed, I would have had to think of something else. Lucky for me, no one has yet created anything like it.

So, simple idea, just what I need to warm up.

First things first, go and read official manual from WordPress about how to develop plugins. It’s a really good manual, and everyone developing a plugin must read it. Or read it if you feel you need more details in following my manual…

All plugins go into /wp-content/plugins/ directory in your WordPress installation. Each plugin has it’s own directory, but this isn’t a strict requirement, but if you’re not creating a directory for your plugin, it must be just a single PHP file. I’d advise to create a directory for your plugin.

Right, this is how I saw me implementing this: user types in any text in the post (or page, or comment, or even excerpt) and encloses any text he/she wants to flip over in my new (made up) tags, just like that:

<upsidedown>your text</upsidedown>

Because WordPress does some clever validation of the text and only certain tags are allowed, I had to include this code in my plugin to declare my new tags as valid ones:

if (!CUSTOM_TAGS) {
    // allow new tag in the posts
    $allowedposttags['upsidedown'] = array();
    // ... and also in the comments
    $allowedtags['upsidedown'] = array();
}

Next thing is to register new filter function with WordPress. Hold on, what is this filter function? Filter function is a function that WordPress calls and passes contents of the post to it. Filter function can do anything it likes to the text and then spits is back to WordPress engine. WordPress will then pass the result to another filter, and so on and on until there are no more filter registered. Each filter get a priority flag assigned, and those with priority 0 will get executed before those that have priority 10. By default each filter gets priority 6 assigned to them.

Right, going back to my filters, let’s register them:

add_filter('the_content', 'flip_text_upside_down_filter');
add_filter('the_excerpt', 'flip_text_upside_down_filter');
add_filter('comment_text', 'flip_text_upside_down_filter');

As you can see, I register my filter with three types of content: post (or page), excerpt and comment.

Next thing is to define the filter function:

function flip_text_upside_down_filter($content)
{
    return preg_replace_callback(
        '/\s*<upsidedown>(.*)<\/upsidedown>/siU',
        'convert_to_upsidedown_unicode',
        $content
    );
}

WordPress is going to call it with a parameter that contains content text in it. I pass the contents to PHP function preg_replace_callback, which matches the regular expression (any text in between my tags) and calls another function with the contents of the regular expression match (hope that makes sense to you, if not, read the sentence again, repeat until you got it… ).
Finally, a function that actually does the conversion. Which is pretty simple – go through all characters and replace them with matching Unicode chars from the table. I’m not including the whole table here, you can download source code from WordPress plugins directory (in fact, if you install the plugin – you’ll have the sourcecode in your plugins/ directory):

function convert_to_upsidedown_unicode($match)
{
    // conversion table
    $flipTable = array(
...
    )
    $origStr = strrev($match[1]);
    $newStr = ' ';
    for ($i = 0; $i < strlen($origStr); $i++) {
        $ch = $origStr[$i];
        if (array_key_exists($ch, $flipTable)) {
            $newStr .= $flipTable[$ch];
        } else {
            $newStr .= $ch;
        }
    }
    return $newStr;
}

This is it. Well, ok nearly it. You still need to do the boring bit, ie put a formal header so WP recognises your plugin code (put this, replacing what’s necessary accordingly):

/*
Plugin Name: Upside Down Text
Plugin URI: http://www.grenadepod.com/projects/upside-down-text-plugin-for-wordpress/
Description: This plugin allows to "flip" any section of text upside down
Version: 1.1.0
Author: Grenadepod
Author URI: http://www.grenadepod.com
*/

Now you can activate the plugin from your WP admin console, and start typing text that everyone struggles to read.

Sooner or later you’ll notice that the implementation is not ideal. When you switch over to visual editor (which I use most of the time) and then back to HTML editor, all “new” tags are gone. This is because the editor does not know about them and so it removes them. Solution was to type in text in Visual (if you prefer to use it) then switch to HTML and place tags where needed. Save without switching back, and publish.

This wasn’t really nice, and I wasn’t going to get lots of users with this approach. Some nice soul on WordPress forum suggested to have a look at Shortcode API. This API allows plugin to register shortcode that looks like: [somethinghere]content[/somethinghere], and passes all content back to your function. And this would be immune to switching between editors issue.

So I decided to make a second release of the plugin, where shortcodes are supported alongside with tags. And in fact using shortcodes not only allows me to type them in directly in the Visual editor, but also coding is simpler, I don’t need to do any regexp magic.

Register my new shortcode:

add_shortcode('upsidedown', 'flip_text_upside_down_shortcode');

And define what function to call:

function flip_text_upside_down_shortcode($atts, $content=null)
{
    return convert_to_upsidedown_unicode(array('', $content));
}

As you can see, I’m reusing same character conversion function, so the changes were minimal.

All seems to be working quite fine, at the end of the day, what can go wrong with only few lines of code?…

Related posts:

1. Securing WordPress
2. Essential WordPress plugins
3. Use SSH to upgrade WordPress plugins automatically

Listing is a Django application, so theoretically I could just go through the list of IPs, ping and add status to the list that gets passed to Django template. But nice it is not. It might take ages to query the whole list, and even if I spend time implementing multithreaded ping and ping them all concurrently it still going to take at leas couple of seconds to complete.

So obvious choice is to delay the ping action. I will serve the page with all IPs and other bits and pieces and then let my JavaScript to do the AJAX magic.

So, how do I do that?

Well, first I need a view that checks if the given IP responds to a ping and displays the result. To keep things simple, I’m going to return a string, which will be displayed on the list page:

def ping(request, address=None):
    if responding_to_ping(address):
        msg = "Ping OK"
    else:
        msg = "No response"
    return HttpResponse(msg)

I then need to add URL pattern, so I can call the view:

    url(r'^ping/(?P<address>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/$', views.ping),

Good, so now I can navigate to http:///ping// and it’ll come back with either “Ping OK” or “No response” message.
Right, now let’s add some JavaScript magic that’ll query the server and update my web page with details.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

<script type="text/javascript" src="/static/js/jquery-1.3.2.min.js"></script>
<script type="text/javascript">
    $(document).ready(function(){
        $(".address").each(function () {
            var curId = $(this).attr('id');
            updateStatus(curId);
        });
    });
 
    function updateStatus(attrId) {
        address = attrId.replace('ip_', '');
        address = address.replace(/_/g, '.');
        $.ajax({
            url: '/ping/' + address,
            success: function(response) {
                $('#' + attrId).html(response);
            }
        });
    }
</script>

So, what exactly is happening here?

• Line 3: This waits for a document too load and call the callback function when ready to proceed, calback function defined in the lines below
• Lines 4-6:Goes through each element of the HTML document (my web page) that are of class “address”, extracts the element’s ‘id’ attribute, and calls ‘updateStatus’ function passing the ‘id’ to it. So all elements that are going to be updated should look like: <span class=”address” id=”ip_address” />
• Lines 11-12: All id’s are going to be in the following format: ip_1_2_3_4, so before I pass this to the URL, I need to strip the ip_ bit, and replace underscores with dots.
• Lines 13-17: This is where I perform AJAX call to the URL. The ‘success’ bit is a call to the function that gets executed when the result is successfully retrieved. This function is pretty simple, it just replaces the contents of the tag with whatever it receives from the server

And this is it, all that is left to do is list all IPs that need checking. Below is the raw HTML, but I trust you can be creative in writing a decent Django template to generate the page.

<ul>
    <li>192.168.0.1 [Status: <span class="address" id="ip_192_168_0_1">Unknown</span> ]</li>
    <li>192.168.0.2 [Status: <span class="address" id="ip_192_168_0_2">Unknown</span> ]</li>
    <li>192.168.0.3 [Status: <span class="address" id="ip_192_168_0_3">Unknown</span> ]</li>
    ...
</ul>

Related posts:

1. How to use generic views in Django
2. Developing my first WordPress plugin
3. Using OpenID for authentication in Django