Application logic is extremely simple:

• Retrieve specified page
• If cannot retrieve, try another from stored in the DB
• Store all found URLs (<a href>) in the DB
• Return one (and remove from DB)

I store all pages in memory, but for more serious crawling, consider storing data on a physical file, that way it’ll be more memory efficient. Considering the size of the internets, even if you store only URL strings you will not last for very long…

Anyway, happy crawling! (and let me know if there are any issues, so far it worked fine for me)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

#!/usr/bin/env python
 
from BeautifulSoup import BeautifulSoup
import urllib2, random, re, sqlite3, logging
 
DATABASE   = ':memory:'
LOG_LEVEL  = logging.INFO
ABS_URL_RE = re.compile('(?P<url>https?://.+?/)')
 
class WebCrawler:
    def __init__(self, logging_level=LOG_LEVEL, database=DATABASE):
        logging.basicConfig(level=logging_level)
        self.conn = sqlite3.connect(database)
        self.conn.execute('create table url_stack (url text)')
        self.conn.execute('create table url_visited (url text)')
 
    def _form_url(self, url, link):
        if link[0:4] == 'http':
            ret_url = link
        elif link[0] == '/':
            m = ABS_URL_RE.search(url)
            ret_url = "%s%s" % (m.group('url'), link[1:])
        else:
            ret_url = "%s%s" % (url, link)
        return ret_url
 
    def _pop_from_db(self):
        logging.debug("Retrieving one URL from the DB...")
        res = self.conn.execute('select url from url_stack limit 1').fetchone()
        logging.debug("Query result: %s", res)
        url = res[0]
        self.conn.execute("delete from url_stack where url = ?", (url,))
        return url
 
    def _push_to_db(self, url):
        logging.debug("Inserting record into DB...")
        logging.debug("Check if it hasn't been visited yet")
        if not self.conn.execute('select url from url_visited where url=?', (url,)).fetchone():
            logging.debug("URL not found in list of visited URLs, inserting")
            logging.debug("URL to insert: %s" % url)
            self.conn.execute('insert into url_stack values (?)', (url,))
            self.conn.execute('insert into url_visited values (?)', (url,))
        else:
            logging.debug("URL has already been visited or added for processing, skipping")
 
    def crawl(self, url):
        work_url = url
        logging.debug("Work URL: %s" % work_url)
        while True:
            try:
                logging.debug("Trying to open and parse the URL...")
                page = urllib2.urlopen(work_url)
                soup = BeautifulSoup(page)
                logging.debug("Parsed successfuly")
            except:
                logging.debug("Failed to parse, attempting to get next URL from DB")
                work_url = self._pop_from_db()
                continue
            links = soup('a')
            logging.debug("Found total of %d links (<a href=...>)" % len(links))
            for link in soup('a'):
                logging.debug("Processing link object: %s" % link)
                try:
                    if link['href'] != '':
                        self._push_to_db(self._form_url(work_url, link['href']))
                except:
                    logging.debug("An exception has occured, this may be ok (href attribute may be missing)")
                    logging.debug("  ... but can also indicate error in insert code")
            logging.debug("Finished adding URLs")
            logging.debug("Getting a new URL for processing from DB")
            work_url = self._pop_from_db()
            logging.info("Found URL: %s" % work_url)
            yield work_url
 
if __name__ == '__main__':
    wc = WebCrawler()
    for url in wc.crawl('http://www.google.com/'):
        pass

As you can see it’s very easy to use. Effectively it is an infinite generator (well, depending what you pass as initial URL). It’s then up to you what you’re going to do with the resulting URL…

One thing to bear in mind when you use it: this crawler pays absolutely no attention to the domain it’s searching. It just blindly collects links, selects one and follows it. So depending on the site link relative location it may stay for a while on a site, or may just wonder away quite quickly.

I wanted to have something that does not stick around for long on one site, so this suits me well, if you want to have something more pedantic, you may want to modify the code, so that it leaves current domain only when it has visited all pages and there are no new pages within the domain to analyse.

Related posts:

1. Changing menu order
2. Top level menu in Arras theme

Right, back to business. So I decided to write a WordPress plugin. There were few reasons I wanted to do that:

• See how difficult it is and what WordPress looks like under the hood
• Since I’m using WP, which is a PHP shop I needed to remind myself how PHP looks like
• Get some more publicity and bump up the Google rating (that’ll never hurt, will it?)

So, I remembered seeing this neat trick of reversing, or flipping over text by replacing regular ASCII latin characters with Unicode ones that resemble a mirror image of the original ones. There are numerous websites out there on the wild internets that would swallow your piece of texts and then spit out same text, but flipped over.

As you can see it is still text information, no magic with images or anything like that. You can select, copy-paste, etc.

So, I thought, right, let’s see if there was someone smart who has created a plugin for WordPress that would do this. Not that I’m a big fan of flipped over text, but just to make sure that I’m not reinventing the wheel. At the end of the day, what’s the point in creating something that’s already been build. If such plugin existed, I would have had to think of something else. Lucky for me, no one has yet created anything like it.

So, simple idea, just what I need to warm up.

First things first, go and read official manual from WordPress about how to develop plugins. It’s a really good manual, and everyone developing a plugin must read it. Or read it if you feel you need more details in following my manual…

All plugins go into /wp-content/plugins/ directory in your WordPress installation. Each plugin has it’s own directory, but this isn’t a strict requirement, but if you’re not creating a directory for your plugin, it must be just a single PHP file. I’d advise to create a directory for your plugin.

Right, this is how I saw me implementing this: user types in any text in the post (or page, or comment, or even excerpt) and encloses any text he/she wants to flip over in my new (made up) tags, just like that:

<upsidedown>your text</upsidedown>

Because WordPress does some clever validation of the text and only certain tags are allowed, I had to include this code in my plugin to declare my new tags as valid ones:

if (!CUSTOM_TAGS) {
    // allow new tag in the posts
    $allowedposttags['upsidedown'] = array();
    // ... and also in the comments
    $allowedtags['upsidedown'] = array();
}

Next thing is to register new filter function with WordPress. Hold on, what is this filter function? Filter function is a function that WordPress calls and passes contents of the post to it. Filter function can do anything it likes to the text and then spits is back to WordPress engine. WordPress will then pass the result to another filter, and so on and on until there are no more filter registered. Each filter get a priority flag assigned, and those with priority 0 will get executed before those that have priority 10. By default each filter gets priority 6 assigned to them.

Right, going back to my filters, let’s register them:

add_filter('the_content', 'flip_text_upside_down_filter');
add_filter('the_excerpt', 'flip_text_upside_down_filter');
add_filter('comment_text', 'flip_text_upside_down_filter');

As you can see, I register my filter with three types of content: post (or page), excerpt and comment.

Next thing is to define the filter function:

function flip_text_upside_down_filter($content)
{
    return preg_replace_callback(
        '/\s*<upsidedown>(.*)<\/upsidedown>/siU',
        'convert_to_upsidedown_unicode',
        $content
    );
}

WordPress is going to call it with a parameter that contains content text in it. I pass the contents to PHP function preg_replace_callback, which matches the regular expression (any text in between my tags) and calls another function with the contents of the regular expression match (hope that makes sense to you, if not, read the sentence again, repeat until you got it… ).
Finally, a function that actually does the conversion. Which is pretty simple – go through all characters and replace them with matching Unicode chars from the table. I’m not including the whole table here, you can download source code from WordPress plugins directory (in fact, if you install the plugin – you’ll have the sourcecode in your plugins/ directory):

function convert_to_upsidedown_unicode($match)
{
    // conversion table
    $flipTable = array(
...
    )
    $origStr = strrev($match[1]);
    $newStr = ' ';
    for ($i = 0; $i < strlen($origStr); $i++) {
        $ch = $origStr[$i];
        if (array_key_exists($ch, $flipTable)) {
            $newStr .= $flipTable[$ch];
        } else {
            $newStr .= $ch;
        }
    }
    return $newStr;
}

This is it. Well, ok nearly it. You still need to do the boring bit, ie put a formal header so WP recognises your plugin code (put this, replacing what’s necessary accordingly):

/*
Plugin Name: Upside Down Text
Plugin URI: http://www.grenadepod.com/projects/upside-down-text-plugin-for-wordpress/
Description: This plugin allows to "flip" any section of text upside down
Version: 1.1.0
Author: Grenadepod
Author URI: http://www.grenadepod.com
*/

Now you can activate the plugin from your WP admin console, and start typing text that everyone struggles to read.

Sooner or later you’ll notice that the implementation is not ideal. When you switch over to visual editor (which I use most of the time) and then back to HTML editor, all “new” tags are gone. This is because the editor does not know about them and so it removes them. Solution was to type in text in Visual (if you prefer to use it) then switch to HTML and place tags where needed. Save without switching back, and publish.

This wasn’t really nice, and I wasn’t going to get lots of users with this approach. Some nice soul on WordPress forum suggested to have a look at Shortcode API. This API allows plugin to register shortcode that looks like: [somethinghere]content[/somethinghere], and passes all content back to your function. And this would be immune to switching between editors issue.

So I decided to make a second release of the plugin, where shortcodes are supported alongside with tags. And in fact using shortcodes not only allows me to type them in directly in the Visual editor, but also coding is simpler, I don’t need to do any regexp magic.

Register my new shortcode:

add_shortcode('upsidedown', 'flip_text_upside_down_shortcode');

And define what function to call:

function flip_text_upside_down_shortcode($atts, $content=null)
{
    return convert_to_upsidedown_unicode(array('', $content));
}

As you can see, I’m reusing same character conversion function, so the changes were minimal.

All seems to be working quite fine, at the end of the day, what can go wrong with only few lines of code?…

Related posts:

1. Securing WordPress
2. Essential WordPress plugins
3. Use SSH to upgrade WordPress plugins automatically

Listing is a Django application, so theoretically I could just go through the list of IPs, ping and add status to the list that gets passed to Django template. But nice it is not. It might take ages to query the whole list, and even if I spend time implementing multithreaded ping and ping them all concurrently it still going to take at leas couple of seconds to complete.

So obvious choice is to delay the ping action. I will serve the page with all IPs and other bits and pieces and then let my JavaScript to do the AJAX magic.

So, how do I do that?

Well, first I need a view that checks if the given IP responds to a ping and displays the result. To keep things simple, I’m going to return a string, which will be displayed on the list page:

def ping(request, address=None):
    if responding_to_ping(address):
        msg = "Ping OK"
    else:
        msg = "No response"
    return HttpResponse(msg)

I then need to add URL pattern, so I can call the view:

    url(r'^ping/(?P<address>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/$', views.ping),

Good, so now I can navigate to http:///ping// and it’ll come back with either “Ping OK” or “No response” message.
Right, now let’s add some JavaScript magic that’ll query the server and update my web page with details.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

<script type="text/javascript" src="/static/js/jquery-1.3.2.min.js"></script>
<script type="text/javascript">
    $(document).ready(function(){
        $(".address").each(function () {
            var curId = $(this).attr('id');
            updateStatus(curId);
        });
    });
 
    function updateStatus(attrId) {
        address = attrId.replace('ip_', '');
        address = address.replace(/_/g, '.');
        $.ajax({
            url: '/ping/' + address,
            success: function(response) {
                $('#' + attrId).html(response);
            }
        });
    }
</script>

So, what exactly is happening here?

• Line 3: This waits for a document too load and call the callback function when ready to proceed, calback function defined in the lines below
• Lines 4-6:Goes through each element of the HTML document (my web page) that are of class “address”, extracts the element’s ‘id’ attribute, and calls ‘updateStatus’ function passing the ‘id’ to it. So all elements that are going to be updated should look like: <span class=”address” id=”ip_address” />
• Lines 11-12: All id’s are going to be in the following format: ip_1_2_3_4, so before I pass this to the URL, I need to strip the ip_ bit, and replace underscores with dots.
• Lines 13-17: This is where I perform AJAX call to the URL. The ‘success’ bit is a call to the function that gets executed when the result is successfully retrieved. This function is pretty simple, it just replaces the contents of the tag with whatever it receives from the server

And this is it, all that is left to do is list all IPs that need checking. Below is the raw HTML, but I trust you can be creative in writing a decent Django template to generate the page.

<ul>
    <li>192.168.0.1 [Status: <span class="address" id="ip_192_168_0_1">Unknown</span> ]</li>
    <li>192.168.0.2 [Status: <span class="address" id="ip_192_168_0_2">Unknown</span> ]</li>
    <li>192.168.0.3 [Status: <span class="address" id="ip_192_168_0_3">Unknown</span> ]</li>
    ...
</ul>

Related posts:

1. How to use generic views in Django
2. Developing my first WordPress plugin
3. Using OpenID for authentication in Django

Sooner or later in your Django development career you’ll notice that you keep on repeating and implementing same views over and over again:

• list objects
• display details for any particular object
• add/delete/modify the object

When I was reading about Django, I’ve noticed something mentioned that was called “generic views”, but at that time it somehow escaped me, and I wrote numerous views, just to implement those functions I mentioned above. In some extreme cases I’ve had 3-4 different views, effectivelly implementing the same for different models.

So the time came to educate myself and learn about “generic views” that are designed to do just that: implement basic and common tasks, so you don’t have to write a single line of code.

Now let’s start with the model definition. Suppose I have the following model class:

class Rule(models.Model):
    rule = models.TextField()
    description = models.CharField(max_length=400)
 
    def __unicode__(self):
        return self.description[:20]

First of all I need to import generic view functions from the Django libraries, here’s what you need to add to your urls.py file:

from django.views.generic import list_detail, create_update

list_detail provides two methods that are used to represent data:

• object_list – to display a list of objects
• object_detail – to display details about any given object

create_update provides three methods to manipulate data:

• create_object – creates new object
• update_object – modifies existing object
• delete_object – deletes any given object

I then need to define a queryset dictionary that contains a list of objects which generic views will operate upon. Add this to your urls.py:

rule_info = {
    'queryset': Rule.objects.all(),
    'template_name': 'display_rule.html',
}

The only required field here is the queryset, others are optional. You don’t even need to define a template name as Django will automatically generate one. If you omit template_name, it will attempt to load a template <app_label>/<model_name>_form.html.

Right, now I’ll do the display bit first. Add the following URL patterns (as you might already guessed it, to ulrs.py file):

url(r'^rule/$', list_detail.object_list, rule_info, name='rule-displaytop'),
url(r'^rule/(?P<object_id>\d+)/, list_detail.object_detail, rule_info, name='rule-display'),

As you can see, I instruct urls.py to call generic views list_detail.object_list and list_detail.object_list and also pass my rule_info dictionary as an argument. The dictionary contains a list of all objects that interest me. List view would simply pass the list on to the template, and the detailed view would select an individual object and pass it on to the template. Therefore the detailed view also expects object_id (which must be a primary key), so that it can find the required object.

When views call templates to display the information, they pass object list as object_list template variable and an instance of any object as object. You can override this by adding template_object_name to the dictionary and setting it to any name you like. If you set it to foo, the names would become: foo_list and foo. For now I leave it as it is. Below is my display_rule.html template:

{% extends "base.html" %}
{% block contents %}
  {% if object %}
    <h1>Rules details</h1>
    <ul>
      <li>ID: {{ object.id }}</li>
      <li>Description: {{ object.description }}</li>
      <li>Rule text:
          {{ object.rule }}
      </li>
    </ul>
    ( <a href="{% url rule-modify object.id %}">modify</a> |
      <a href="{% url rule-delete object.id %}">delete</a> )
  {% else %}
    <h1>List of all Rules</h1>
    {% if object_list %}
      <ul>
        {% for rule in object_list %}
          <li>{{ rule.description }} ( <a href="{% url rule-display rule.id %}">details</a> |
                                       <a href="{% url rule-modify rule.id %}">modify</a> |
                                       <a href="{% url rule-delete rule.id %}">delete</a> )</li>
        {% endfor %}
      </ul>
    {% else %}
      No rules defined yet.
    {% endif %}
  <h3><a href="{% url rule-add %}">Add new rule</a></h3>
{% endif %}
{% endblock %}

As you can see the logic is somewhat simple:

• Check if the template object object is defined
• If it is, that means we’ve been called from the detailed object view method and so need to display object information
• If not, then we’ve been called from the object list view, so we need to iterate through the list
• Check if the list is not empty (very important! as the lists tend to come empty in some cases…)
• If the list is not empty – display all objects and provide links to basic functions

Let’s start with adding new objects. You will need to define get_absolute_url method for your class (this will be used to get a URL where to redirect the user once the form has been submitted) and also define Model Form class (not required, strictly speaking, but I like to define it, so that I don’t need to if I decide to change form behaviour). This is how these two classes are going to look now:

class Rule(models.Model):
    rule = models.TextField()
    description = models.CharField(max_length=400)
 
    def __unicode__(self):
        return self.description[:20]
 
    @models.permalink
    def get_absolute_url(self):
        return ('rule-display', (), {'object_id': self.id})
 
class RuleForm(ModelForm):
    class Meta:
        model = Rule

For now ignore the “magic” with URL names, these are used to help reverse URL resolver, and I’m going to talk about these in other posts. Just note that get_absolute_url returns generated URL with object_id embedded in it.

add.html template is very simplistic and contains only few lines:

<form action="." method="POST">
{{ form.as_p }}
<input type="submit" value="Add" />
</form>

This is enough to display all fields and a submit button. If you need anything fancier than that, you’d need to alter the form to suit your needs.

Next, I’ll need to define a dictionary with settings, which will be passed to generic create and modify views, just like a queryset dictionary was used with display views:

rule_form = {
  'form_class': RuleForm,
  'template_name': 'add.html',
}

And finally some URL patterns to handle the requests:

url(r'^rule/(?P<object_id>\d+)/modify/$', create_update.update_object, rule_form, name='rule-modify'),
url(r'^rule/add/$', create_update.create_object, rule_form, name='rule-add'),

As you can see, these two are very similar with one exception, update_object requires object_id, which is embedded in the URL, so you don’t need to worry about it.
Finally let’s sort out the delete object function. This function requires three fields: model name, post delete redirect and also a confirmation template name. All defined in a dictionary form:

rule_delete = {
  'model': Rule,
  'post_delete_redirect': '../..',
  'template_name': 'delete_confirm_rule.html',
}

You also need additional URL pattern:

url(r'^rule/(?P<object_id>\d+)/delete/$, create_update.delete_object, rule_delete, name='rule-delete'),

Confirmation template just asks for a user confirmation and redirects to the same URL, however the method is POST, so the view knows it needs to delete the object rather than display the confirmation page (which happens if the request is GET):

<form method="post" action=".">
<p>Are you sure?</p>
<input type="submit" />
</form>

This is it, now you know how to perform some of the common tasks without writing any views.

Related posts:

1. Query data from Django site with jQuery
2. Python web crawler