First, you need to install RPM devtools:

# yum install rpmdevtools

Then setup your local RPM build tree:

$ rpmdev-setuptree

This will create RPM build tree (~/rpmbuild/) in your home directory. Just a reminder – never, ever, under any circumstances, build RPMs as root user. Just a reminder. Obviously, you knew that already, right?

You also want to get and install additional header files and libraries, such as TCL/TK, expat, sqlite, etc, that are used to build some Python components:

# yum install tk-devel tcl-devel expat-devel db4-devel gdbm-devel sqlite-devel

Now we’re ready to start building an RPM. Get the RPM spec file from source build tree and copy it to your rpmbuild/SPECS directory, also copy the tarball to your build directory:

$ wget http://www.python.org/ftp/python/2.6.4/Python-2.6.4.tar.bz2
$ tar jxf Python-2.6.4.tar.bz2
$ cp Python-2.6.4/Misc/RPM/python-2.6.spec ~/rpmbuild/SPECS/
$ cp Python-2.6.4.tar.bz2 ~/rpmbuild/SOURCES/

This spec file is slightly broken and if you try building an RPM with it, it most likely is going to fail. So you need to patch it. The proposed patch has been submitted to python bug tracking system in the days of 2.6.1 release, but hasn’t yet been implemented. This patch doesn’t seem to work with 2.6.4 release and needs changing as well.

I have modified original 2.6.1 patch file slightly so that it can be used to build 2.6.4 Python RPM. You download it and apply to the original .spec:

$ cd ~/rpmbuild/SPECS/
$ wget http://www.grenadepod.com/wp-content/uploads/python-2.6.4.spec.diff
$ patch python-2.6.spec python-2.6.4.spec.diff

You should be OK to build the RPM now, but it’ll most likely fail with minor rpath-check errors, such as:

ERROR   0001: file '/usr/lib/python2.6/lib-dynload/_bsddb.so' contains a standard rpath '/usr/lib' in [/usr/lib]
ERROR   0001: file '/usr/lib/python2.6/lib-dynload/_sqlite3.so' contains a standard rpath '/usr/lib' in [/usr/lib]

Which indicates that some components are using hardcoded library paths. It’s not a big issue and can be ignored. Run build command as this:

$ QA_RPATHS=$[ 0x0001|0x0010 ] rpmbuild -ba python-2.6.spec

It should now build without any issues and the result is a set of python RPMs:

$ QA_RPATHS=$[ 0x0001|0x0010 ] rpmbuild -ba python-2.6.spec
...
Wrote: /home/pulegium/rpmbuild/SRPMS/python2.6-2.6.4-1gpod.src.rpm
Wrote: /home/pulegium/rpmbuild/RPMS/i386/python2.6-2.6.4-1gpod.i386.rpm
Wrote: /home/pulegium/rpmbuild/RPMS/i386/python2.6-devel-2.6.4-1gpod.i386.rpm
Wrote: /home/pulegium/rpmbuild/RPMS/i386/python2.6-tkinter-2.6.4-1gpod.i386.rpm
Wrote: /home/pulegium/rpmbuild/RPMS/i386/python2.6-tools-2.6.4-1gpod.i386.rpm
Wrote: /home/pulegium/rpmbuild/RPMS/i386/python2.6-debuginfo-2.6.4-1gpod.i386.rpm
...
$

This is it… You now have python RPMs that you can install on your CentOS 5.x server.

I’m planning to create my little own repository, where I’ll put this (and other) packages. For now, you can download python 2.6.4 source RPM if you want to change something in the spec file and rebuild it yourself.

Related posts:

1. Sorting out YUM repositories on CentOS 5.4
2. Use SSH to upgrade WordPress plugins automatically
3. Building and running Google Chrome OS on VirtualBox

Application logic is extremely simple:

• Retrieve specified page
• If cannot retrieve, try another from stored in the DB
• Store all found URLs (<a href>) in the DB
• Return one (and remove from DB)

I store all pages in memory, but for more serious crawling, consider storing data on a physical file, that way it’ll be more memory efficient. Considering the size of the internets, even if you store only URL strings you will not last for very long…

Anyway, happy crawling! (and let me know if there are any issues, so far it worked fine for me)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

#!/usr/bin/env python
 
from BeautifulSoup import BeautifulSoup
import urllib2, random, re, sqlite3, logging
 
DATABASE   = ':memory:'
LOG_LEVEL  = logging.INFO
ABS_URL_RE = re.compile('(?P<url>https?://.+?/)')
 
class WebCrawler:
    def __init__(self, logging_level=LOG_LEVEL, database=DATABASE):
        logging.basicConfig(level=logging_level)
        self.conn = sqlite3.connect(database)
        self.conn.execute('create table url_stack (url text)')
        self.conn.execute('create table url_visited (url text)')
 
    def _form_url(self, url, link):
        if link[0:4] == 'http':
            ret_url = link
        elif link[0] == '/':
            m = ABS_URL_RE.search(url)
            ret_url = "%s%s" % (m.group('url'), link[1:])
        else:
            ret_url = "%s%s" % (url, link)
        return ret_url
 
    def _pop_from_db(self):
        logging.debug("Retrieving one URL from the DB...")
        res = self.conn.execute('select url from url_stack limit 1').fetchone()
        logging.debug("Query result: %s", res)
        url = res[0]
        self.conn.execute("delete from url_stack where url = ?", (url,))
        return url
 
    def _push_to_db(self, url):
        logging.debug("Inserting record into DB...")
        logging.debug("Check if it hasn't been visited yet")
        if not self.conn.execute('select url from url_visited where url=?', (url,)).fetchone():
            logging.debug("URL not found in list of visited URLs, inserting")
            logging.debug("URL to insert: %s" % url)
            self.conn.execute('insert into url_stack values (?)', (url,))
            self.conn.execute('insert into url_visited values (?)', (url,))
        else:
            logging.debug("URL has already been visited or added for processing, skipping")
 
    def crawl(self, url):
        work_url = url
        logging.debug("Work URL: %s" % work_url)
        while True:
            try:
                logging.debug("Trying to open and parse the URL...")
                page = urllib2.urlopen(work_url)
                soup = BeautifulSoup(page)
                logging.debug("Parsed successfuly")
            except:
                logging.debug("Failed to parse, attempting to get next URL from DB")
                work_url = self._pop_from_db()
                continue
            links = soup('a')
            logging.debug("Found total of %d links (<a href=...>)" % len(links))
            for link in soup('a'):
                logging.debug("Processing link object: %s" % link)
                try:
                    if link['href'] != '':
                        self._push_to_db(self._form_url(work_url, link['href']))
                except:
                    logging.debug("An exception has occured, this may be ok (href attribute may be missing)")
                    logging.debug("  ... but can also indicate error in insert code")
            logging.debug("Finished adding URLs")
            logging.debug("Getting a new URL for processing from DB")
            work_url = self._pop_from_db()
            logging.info("Found URL: %s" % work_url)
            yield work_url
 
if __name__ == '__main__':
    wc = WebCrawler()
    for url in wc.crawl('http://www.google.com/'):
        pass

As you can see it’s very easy to use. Effectively it is an infinite generator (well, depending what you pass as initial URL). It’s then up to you what you’re going to do with the resulting URL…

One thing to bear in mind when you use it: this crawler pays absolutely no attention to the domain it’s searching. It just blindly collects links, selects one and follows it. So depending on the site link relative location it may stay for a while on a site, or may just wonder away quite quickly.

I wanted to have something that does not stick around for long on one site, so this suits me well, if you want to have something more pedantic, you may want to modify the code, so that it leaves current domain only when it has visited all pages and there are no new pages within the domain to analyse.

Related posts:

1. Changing menu order
2. Top level menu in Arras theme

We’ve all been in this situation. And for me personally, I would steer away from the site that forces me to register just to see some part of forum or similar. Mostly this is because I don’t want to provide my details and follow lengthy registration process.

I don’t mind to have an account (even though I forget about the website next day), but the whole process of typing in all the stuff, deciphering captcha images, logging in to my email account just to confirm registration, etc, etc. Why don’t they use OpenId? These days if you have Yahoo or Gmail account (and just who doesn’t?) you can use them to login to all websites that support OpenId.

In a nutshell, OpenId is a decentralised authentication standard. User (U) who has an OpenId account with a trusted company (C) wants to login to a website (W). W prompts with a login screen. U enters his/her OpenId URL (no personal information here provided at all). W then redirects to C login screen, where U is prompted to allow access to some parts of the information (usually it’s username only, sometimes email). If U is happy to provide this information, he then clicks login button and information is passed back to W. In this scenario, W will never get any personal information about U without explicit permission. And U benefits from having only one login that works for all websites.

Below is a simple diagram that shows how authentication is done when using OpenId to login to the websites:

Authentication with OpenID flow

Unfortunately Django does not have native support for OpenId, and their website recommends using django-authopenid library to perform authentication and login using OpenId.

Install required libraries

You need to install python-openid library first. Installation is a breeze, however there seems to be some issues with internal version number in the package, but let’s assume it’s all fine:

# easy_install python-openid
Searching for python-openid
Reading http://pypi.python.org/simple/python-openid/
Reading http://openidenabled.com/python-openid/
Reading http://www.openidenabled.com/openid/libraries/python/
Best match: python-openid 2.2.4
Downloading http://openidenabled.com/files/python-openid/packages/python-openid-2.2.4.zip
Processing python-openid-2.2.4.zip
Running python-openid-2.2.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Xyxx61/python-openid-2.2.4/egg-dist-tmp-oexXVf
zip_safe flag not set; analyzing archive contents...
Adding python-openid 2.2.4 to easy-install.pth file
 
Installed /usr/lib/python2.5/site-packages/python_openid-2.2.4-py2.5.egg
Processing dependencies for python-openid
Finished processing dependencies for python-openid
# python
Python 2.5.1 (r251:54863, Jul 31 2008, 23:17:43)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> openid.__version__
'2.2.1'
>>> openid.__path__
['/usr/lib/python2.5/site-packages/python_openid-2.2.4-py2.5.egg/openid']
>>> openid.version_info
(2, 2, 1)
>>>

Then install django-authopenid. Installation seems to spit out some errors, but I guess these can be ignored as they don’t seem to cause any issues:

# easy_install django-authopenid
Searching for django-authopenid
Reading http://pypi.python.org/simple/django-authopenid/
Reading http://hg.e-engura.org/django-authopenid/
Reading http://code.google.com/p/django-authopenid/
Best match: django-authopenid 1.0.1
Downloading http://pypi.python.org/packages/source/d/django-authopenid/django-authopenid-1.0.1.tar.gz#md5=93d44b4ce40de55bed36c9ed292adb49
Processing django-authopenid-1.0.1.tar.gz
Running django-authopenid-1.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-UtVC7d/django-authopenid-1.0.1/egg-dist-tmp-oWjnYf
warning: no files found matching 'CHANGES.MD'
Sorry: TypeError: ('compile() expected string without null bytes',)
...
Adding django-authopenid 1.0.1 to easy-install.pth file
 
Installed /usr/lib/python2.5/site-packages/django_authopenid-1.0.1-py2.5.egg
Processing dependencies for django-authopenid
Finished processing dependencies for django-authopenid

Prepare Django application environment

There are several things you’d need to do:

1. Create directory for application files

I will use /var/app/application/ in this example

2. Create directory for static content such as JS, CSS, HTML and images

I will use /var/www/application/ in this example

3. If you are using Apache web server add Virtual host definition and specify where web root is and when to pass control to mod_python module (note that all requests that do not start with /static/ will be passed to Django):

    ServerName test.example.com
    DocumentRoot /var/www/application/
    ErrorLog /var/log/apache2/application-error.log
    CustomLog /var/log/apache2/application-access.log combined
    SetHandler mod_python
    PythonHandler django.core.handlers.modpython
    PythonPath sys.path+['/var/app/']
    SetEnv DJANGO_SETTINGS_MODULE application.settings
    SetEnv PYTHON_EGG_CACHE /tmp
    <Location "/static/">
        SetHandler None
    </Location>

4. Also create database and configure appropriately in settings.py.
5. Enable administration application for Django

Enable OpenID support

These were the basic steps to get your Django environment set-up, now let’s enable OpenID module:

1. In settings.py add OpenID middleware, application and also tell Django where to find templates

MIDDLEWARE_CLASSES = (
    ...
    'django_authopenid.middleware.OpenIDMiddleware',
)
INSTALLED_APPS = (
    ...
    'django_authopenid',
    'registration',
)
LOGIN_URL = "/account/signin/"
TEMPLATE_DIRS = (
    ...
    os.path.join(os.path.dirname(__file__), 'templates')
)

2. In urls.py just include urls from django_authopenid package:

urlpatterns = patterns('',
    ...
    (r'^account/', include('django_authopenid.urls')),
)

3. Add the following context processors list (majority is what’s loaded by default anyway, you just need to add one line, but since it’s a tuple, you cannot modify it, so you must redefine it):

   TEMPLATE_CONTEXT_PROCESSORS = (
      'django.core.context_processors.auth',
      'django.core.context_processors.debug',
      'django.core.context_processors.i18n',
      'django.core.context_processors.media',
      'django.core.context_processors.request',
      'django_authopenid.context_processors.authopenid',
   )

4. Copy templates from the examples folder. Of course you’ll want them eventually to be fully integrated with your site, so that they match styles or even be incorporated in other pages, but for now, let’s just use the ones provided with the package. Location of the example templates obviously depends on your Linux distribution. On my Debian installation it was in /usr/lib/python2.5/site-packages/django_authopenid-1.0.1-py2.5.egg/example/templates. Copy examples/ directory to your Django application directory.

Test it

Once you’re done, test it. Navigate to http:///account/signin/ and if you followed this manual you should see something like this:

Django authopenid login screen

You may wonder what the login prompt on the left side is doing there. This is because Django authopenId logoc is to associate your OpenId account with local account and use standard Django “registration” middleware classes to handle all user management functions. If user does not exist – after you have selected your OpenID URL you will be asked to provide local username. You don’t need to memorise nor use it – it’s just to allow authopenId create association.

This may feel bit unintuitive, and I don’t particularly like this. I’m going to make some changes so that internal association is hidden from the end user and happens behind the scenes. This way my users are going to see only OpenId selector and login screen.

Related posts:

1. Python web crawler
2. How to use generic views in Django
3. Building and running Google Chrome OS on VirtualBox

Listing is a Django application, so theoretically I could just go through the list of IPs, ping and add status to the list that gets passed to Django template. But nice it is not. It might take ages to query the whole list, and even if I spend time implementing multithreaded ping and ping them all concurrently it still going to take at leas couple of seconds to complete.

So obvious choice is to delay the ping action. I will serve the page with all IPs and other bits and pieces and then let my JavaScript to do the AJAX magic.

So, how do I do that?

Well, first I need a view that checks if the given IP responds to a ping and displays the result. To keep things simple, I’m going to return a string, which will be displayed on the list page:

def ping(request, address=None):
    if responding_to_ping(address):
        msg = "Ping OK"
    else:
        msg = "No response"
    return HttpResponse(msg)

I then need to add URL pattern, so I can call the view:

    url(r'^ping/(?P<address>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/$', views.ping),

Good, so now I can navigate to http:///ping// and it’ll come back with either “Ping OK” or “No response” message.
Right, now let’s add some JavaScript magic that’ll query the server and update my web page with details.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

<script type="text/javascript" src="/static/js/jquery-1.3.2.min.js"></script>
<script type="text/javascript">
    $(document).ready(function(){
        $(".address").each(function () {
            var curId = $(this).attr('id');
            updateStatus(curId);
        });
    });
 
    function updateStatus(attrId) {
        address = attrId.replace('ip_', '');
        address = address.replace(/_/g, '.');
        $.ajax({
            url: '/ping/' + address,
            success: function(response) {
                $('#' + attrId).html(response);
            }
        });
    }
</script>

So, what exactly is happening here?

• Line 3: This waits for a document too load and call the callback function when ready to proceed, calback function defined in the lines below
• Lines 4-6:Goes through each element of the HTML document (my web page) that are of class “address”, extracts the element’s ‘id’ attribute, and calls ‘updateStatus’ function passing the ‘id’ to it. So all elements that are going to be updated should look like: <span class=”address” id=”ip_address” />
• Lines 11-12: All id’s are going to be in the following format: ip_1_2_3_4, so before I pass this to the URL, I need to strip the ip_ bit, and replace underscores with dots.
• Lines 13-17: This is where I perform AJAX call to the URL. The ‘success’ bit is a call to the function that gets executed when the result is successfully retrieved. This function is pretty simple, it just replaces the contents of the tag with whatever it receives from the server

And this is it, all that is left to do is list all IPs that need checking. Below is the raw HTML, but I trust you can be creative in writing a decent Django template to generate the page.

<ul>
    <li>192.168.0.1 [Status: <span class="address" id="ip_192_168_0_1">Unknown</span> ]</li>
    <li>192.168.0.2 [Status: <span class="address" id="ip_192_168_0_2">Unknown</span> ]</li>
    <li>192.168.0.3 [Status: <span class="address" id="ip_192_168_0_3">Unknown</span> ]</li>
    ...
</ul>

Related posts:

1. How to use generic views in Django
2. Developing my first WordPress plugin
3. Using OpenID for authentication in Django

Sooner or later in your Django development career you’ll notice that you keep on repeating and implementing same views over and over again:

• list objects
• display details for any particular object
• add/delete/modify the object

When I was reading about Django, I’ve noticed something mentioned that was called “generic views”, but at that time it somehow escaped me, and I wrote numerous views, just to implement those functions I mentioned above. In some extreme cases I’ve had 3-4 different views, effectivelly implementing the same for different models.

So the time came to educate myself and learn about “generic views” that are designed to do just that: implement basic and common tasks, so you don’t have to write a single line of code.

Now let’s start with the model definition. Suppose I have the following model class:

class Rule(models.Model):
    rule = models.TextField()
    description = models.CharField(max_length=400)
 
    def __unicode__(self):
        return self.description[:20]

First of all I need to import generic view functions from the Django libraries, here’s what you need to add to your urls.py file:

from django.views.generic import list_detail, create_update

list_detail provides two methods that are used to represent data:

• object_list – to display a list of objects
• object_detail – to display details about any given object

create_update provides three methods to manipulate data:

• create_object – creates new object
• update_object – modifies existing object
• delete_object – deletes any given object

I then need to define a queryset dictionary that contains a list of objects which generic views will operate upon. Add this to your urls.py:

rule_info = {
    'queryset': Rule.objects.all(),
    'template_name': 'display_rule.html',
}

The only required field here is the queryset, others are optional. You don’t even need to define a template name as Django will automatically generate one. If you omit template_name, it will attempt to load a template <app_label>/<model_name>_form.html.

Right, now I’ll do the display bit first. Add the following URL patterns (as you might already guessed it, to ulrs.py file):

url(r'^rule/$', list_detail.object_list, rule_info, name='rule-displaytop'),
url(r'^rule/(?P<object_id>\d+)/, list_detail.object_detail, rule_info, name='rule-display'),

As you can see, I instruct urls.py to call generic views list_detail.object_list and list_detail.object_list and also pass my rule_info dictionary as an argument. The dictionary contains a list of all objects that interest me. List view would simply pass the list on to the template, and the detailed view would select an individual object and pass it on to the template. Therefore the detailed view also expects object_id (which must be a primary key), so that it can find the required object.

When views call templates to display the information, they pass object list as object_list template variable and an instance of any object as object. You can override this by adding template_object_name to the dictionary and setting it to any name you like. If you set it to foo, the names would become: foo_list and foo. For now I leave it as it is. Below is my display_rule.html template:

{% extends "base.html" %}
{% block contents %}
  {% if object %}
    <h1>Rules details</h1>
    <ul>
      <li>ID: {{ object.id }}</li>
      <li>Description: {{ object.description }}</li>
      <li>Rule text:
          {{ object.rule }}
      </li>
    </ul>
    ( <a href="{% url rule-modify object.id %}">modify</a> |
      <a href="{% url rule-delete object.id %}">delete</a> )
  {% else %}
    <h1>List of all Rules</h1>
    {% if object_list %}
      <ul>
        {% for rule in object_list %}
          <li>{{ rule.description }} ( <a href="{% url rule-display rule.id %}">details</a> |
                                       <a href="{% url rule-modify rule.id %}">modify</a> |
                                       <a href="{% url rule-delete rule.id %}">delete</a> )</li>
        {% endfor %}
      </ul>
    {% else %}
      No rules defined yet.
    {% endif %}
  <h3><a href="{% url rule-add %}">Add new rule</a></h3>
{% endif %}
{% endblock %}

As you can see the logic is somewhat simple:

• Check if the template object object is defined
• If it is, that means we’ve been called from the detailed object view method and so need to display object information
• If not, then we’ve been called from the object list view, so we need to iterate through the list
• Check if the list is not empty (very important! as the lists tend to come empty in some cases…)
• If the list is not empty – display all objects and provide links to basic functions

Let’s start with adding new objects. You will need to define get_absolute_url method for your class (this will be used to get a URL where to redirect the user once the form has been submitted) and also define Model Form class (not required, strictly speaking, but I like to define it, so that I don’t need to if I decide to change form behaviour). This is how these two classes are going to look now:

class Rule(models.Model):
    rule = models.TextField()
    description = models.CharField(max_length=400)
 
    def __unicode__(self):
        return self.description[:20]
 
    @models.permalink
    def get_absolute_url(self):
        return ('rule-display', (), {'object_id': self.id})
 
class RuleForm(ModelForm):
    class Meta:
        model = Rule

For now ignore the “magic” with URL names, these are used to help reverse URL resolver, and I’m going to talk about these in other posts. Just note that get_absolute_url returns generated URL with object_id embedded in it.

add.html template is very simplistic and contains only few lines:

<form action="." method="POST">
{{ form.as_p }}
<input type="submit" value="Add" />
</form>

This is enough to display all fields and a submit button. If you need anything fancier than that, you’d need to alter the form to suit your needs.

Next, I’ll need to define a dictionary with settings, which will be passed to generic create and modify views, just like a queryset dictionary was used with display views:

rule_form = {
  'form_class': RuleForm,
  'template_name': 'add.html',
}

And finally some URL patterns to handle the requests:

url(r'^rule/(?P<object_id>\d+)/modify/$', create_update.update_object, rule_form, name='rule-modify'),
url(r'^rule/add/$', create_update.create_object, rule_form, name='rule-add'),

As you can see, these two are very similar with one exception, update_object requires object_id, which is embedded in the URL, so you don’t need to worry about it.
Finally let’s sort out the delete object function. This function requires three fields: model name, post delete redirect and also a confirmation template name. All defined in a dictionary form:

rule_delete = {
  'model': Rule,
  'post_delete_redirect': '../..',
  'template_name': 'delete_confirm_rule.html',
}

You also need additional URL pattern:

url(r'^rule/(?P<object_id>\d+)/delete/$, create_update.delete_object, rule_delete, name='rule-delete'),

Confirmation template just asks for a user confirmation and redirects to the same URL, however the method is POST, so the view knows it needs to delete the object rather than display the confirmation page (which happens if the request is GET):

<form method="post" action=".">
<p>Are you sure?</p>
<input type="submit" />
</form>

This is it, now you know how to perform some of the common tasks without writing any views.

Related posts:

1. Query data from Django site with jQuery
2. Python web crawler