Old web browsers do not support Allow-acces-origin HTTP header needed to do cross-domain AJAX requests (IE6, IE7).

Below is an example how to work around this for jQuery getJSON() calls by

• Detecting browsers which do not support this using jQuery.support API
• Doing an alternative code path through a local website proxy view which uses Python urllib to make server-to-server call and return it as it would be a local call, thus working around cross-domain restriction

This example is for Plone/Grok, but the code is easily port to other Python web frameworks.

Javascript example

/**
 * Call a RESTful service vie AJAX
 *
 * The final URL is constructed by REST function name, based
 * on a base URL from the global settings.
 *
 * If the browser does not support cross domain AJAX calls
 * we'll use a proxy function on the local server. For
 * performance reasons we do this only when absolutely needed.
 *
 * @param {String} functionName REST function name to a call
 *
 * @param {Object} Arguments as a dictionary like object, passed to remote call
 */
function callRESTful(functionName, args, callback) {

    var src = myoptions.restService + "/" +functionName;

    // set to true to do proxied request on every browser
    // useful if you want to use Firebug to debug your server-side proxy view
    var debug = false;

        console.log("Doing remote call to:" + src)

        // We use jQuery API to detect whether a browser supports cross domain AJAX calls
        // http://api.jquery.com/jQuery.support/
        if(!jQuery.support.cors || debug) {
                // http://alexn.org/blog/2011/03/24/cross-domain-requests.html
                // Opera 10 doesn't have this feature, neither do IExplorer < 8, Firefox < 3.5

                console.log("Mangling getJSON to go through a local proxy")

                // Change getJSON to go to our proxy view on a local server
                // and pass the orignal URL as a parameter
                // The proxy view location is given as a global JS variable
                args.url = src;
                src = myoptions.portalUrl + "/@@proxy";
        }

        // Load data from the server
        $.getJSON(src, args, function(data) {
                // Parse incoming data and construct Table rows according to it
                console.log("Data succesfully loaded");
                callback(data, args);

     });

}

The server-side view:

import socket
import urllib
import urllib2
from urllib2 import HTTPError

from five import grok
from Products.CMFCore.interfaces import ISiteRoot
from mysite.app import options

class Proxy(grok.CodeView):
    """
    Pass a AJAX call to a remote server. This view is mainly indended to be used
    with jQuery.getJSON() requests.

    This will work around problems when a browser does not support Allow-Access-Origin HTTP header (IE).

    Asssuming only HTTP GET requests are made.s
    """

    # This view is available only at the root of Plone site
    grok.context(ISiteRoot)

    def isAllowed(self, url):
        """
        Check whether we are allowed to call the target URL.

        This prevents using your service as an malicious proxy
        (to call any internet service).
        """

        allowed_prefix = options.REST_SERVICE_URL

        if url.startswith(allowed_prefix):
            return True

        return False

    def render(self):
        """
        Use HTTP GET ``url`` query parameter for the target of the real request.
        """

        # Make sure any theming layer won't think this is HTML
        # http://stackoverflow.com/questions/477816/the-right-json-content-type
        self.request.response.setHeader("Content-type", "application/json")

        url = self.request.get("url", None)
        if not url:
            self.request.response.setStatus(500, "url parameter missing")

        if not self.isAllowed(url):
            # The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeate
            self.request.response.setStatus(403, "proxying to the target URL not allowed")
            return

        # Pass other HTTP GET query parameters direclty to the target server
        params = {}
        for key, value in self.request.form.items():
            if key != "url":
                params[key] = value

        # http://www.voidspace.org.uk/python/articles/urllib2.shtml
        data = urllib.urlencode(params)

        full_url = url + "?" + data
        req = urllib2.Request(full_url)

        try:

            # Important or if the remote server is slow
            # all our web server threads get stuck here
            # But this is UGLY as Python does not provide per-thread
            # or per-socket timeouts thru urllib
            orignal_timeout = socket.getdefaulttimeout()
            try:
                socket.setdefaulttimeout(10)

                response = urllib2.urlopen(req)
            finally:
                # restore orignal timeoout
                socket.setdefaulttimeout(orignal_timeout)

            # XXX: How to stream respone through Zope
            # AFAIK - we cannot do it currently

            return response.read()

        except HTTPError, e:
            # Have something more useful to log output as plain urllib exception
            # using Python logging interface
            # http://docs.python.org/library/logging.html
            logger.error("Server did not return HTTP 200 when calling remote proxy URL:" + url)
            for key, value in params.items():
                logger.error(key + ": "  + value)

            # Print the server-side stack trace / error page
            logger.error(e.read())

            raise e

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Some examples:

• Django local flavour defines phone number validations for different countries
• Misc. regular expression validators in Plone’s Archetypes content type subsystem
• Every framework define their own email validator, with various regular expressions

Isn’t writing one’s own validation code a bit redundant and exactly “reinventing the wheel” what open source principles so hard try to avoid? Could validation be a low hanging fruit to share among fellow Python projects? As I see it, for the simple data validation, like email and URL, the core code could be easily shared and different Python projects. You basically want just method is_valid_phonenumber(str) and then framework specific way to raise the error to the user.

Do such frameworks already exist? At least I haven’t seen one being used in any big Python project yet

… or is validation so complex thing, so that validation functions must be tightly integrated with the parent framework and I am missing some big things (like locales, etc.) here?

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Preface

Please do not take this post too seriously if I am ranting tongue-in-cheek. This thinking does not reflect the official opinion of Sauna Sprint team any way, it is just my personal rambling.

Background

Plone is having popularity issues. This can be deducted by:

• PHP CMS are more popular (Google trends and so on)
• Plone lacks love
• By discussing with Python programmers outside Plone community: they see Plone as monster

Plone is a superior CMS product (feature wise, polish wise, tech wise, and so on). So how did this happen?

My theory, you are free to disagree, is that the community has been driving “wrong” ideologies

• Plone’s “buyers” are not users, but developers
• There has been no driving community factor to make Plone accessible for more developers. Plone community has been working to make Plone work for the existing community members, not new members. The consulting companies have been too busy to extract money from their existing big clients. It might even be that it’s the best interest of these companies to keep the barrier of entry high, as this way they protect the Plone business for themselves. Note that this strategy will fail in the point there is no longer Plone business.

The big boys are telling that customers are abandoning Plone, because there are no developers. I know a lot of Plone developers, looking for more Plone work.  I’ll translate this that there are not enough affordable Plone developers entering the market.

I am also not buying arguments like

• “don’t compare apples to oranges (Django vs. Plone)”
• “its history and evolved that way”
• “big tool must be complex”

The result

Plone is easy to use, but hard to develop. If you need to use code generators and boiler-plate in Python code, you are doing it wrong.

Plone is a soup of different, non-mainstream, technologies. To be a Plonista you need master: Python (ok docs), Plone user mode (good docs), Zope component architecture (some docs), ZCML (no useful docs), GenericSetup (no docs), TAL (some docs), Archetypes (basically no docs), buildout (aaaaaarghs) … You have too much to learn from material which does not exist. Today the material effectively does not exist if it’s not your first Google hit – I am against book.

New developers are not getting in. It is easier to duplicate basic Plone functionality in a hacky in-house Django CMS than customize Plone for your use case. Plone does not give any more added value to the new developers. There is no business motivation to adapt Plone as a technology platform in new ventures.

The quest

Let’s fix it, as I see the developer acceptance the most crucial challenge Plone community is facing if the community wants to survive.

What we can do here is

• Make Plone easier for developers
• Make more, better, learning material available
• Eliminate and protect against factors in Plone which hinder the goals of two above bullets (do not include new technologies or modules to core if they make developers sad)

To accomplish this we can

1. Promote “easy” technologies inside and outside Plone community (Dexterity over Archetypes, Grok over ZCML)
2. Making a standard workflow available for them (Django style tutorial that do 1, 2, 3 and you are a certified Plone developer)
3. Kill technologies which make the stack unnecessary complex and are not necessarily needed in newbie use cases (buildout, I am looking at you)
4. Replace old, badly documented and difficult solutions with easier ones (could we actually replace TAL with something more popular?)

Operational plan

Here are some concrete tasks what we could tackle in Sauna Sprint.

Make it easy department

• Plone auto-restart team: provide working and fast auto-restart á la Django or Tornado (hint: plone.reload doesn’t cut it – you need real restarts process-wise)
• Default custom product team: ship Plone with a file-system based custom product skeleton, so that when you need to include your first custom Python module you don’t need to learn about buildout and paster first . You actually can copy-paste in Python code and it just works.
• Through-the-web is happy again team: Make it possible to code and execute Python through-the-web so that it actually works.: a working replacement for RestrictedPython

Fix user-wise broken things department

• Buildout team: make buildout more user friendly: spit out error messages which are actually useful, have newbie buildout tutorial from the developer perspective. Nothing bad should happen when buildout is run. Make buildout to confirm version updates, etc.
• Anti-buildout team: how to install eggs without buildout. Why did we need the buildout in the first place?

This is how you do it department

• Tutorial team: Make “gapless” hand-holding tutorial for creating your first view, form and content-type. Make it so that you do not need to look outside the tutorial or learn anything unnecessary (don’t make me think)
• Picture team: draw big pictures of Plone architecture, how different parts are connected, so that new developers get hang on things easily
• Document generator team: Integrate collective.developermanual with Sphinx tool that generate the reference  documentation from the source code: views, viewlets, templates, CSS classes, portlets, etc.

Hasta la vista babe department

• Kill it with fire team: Hunt down all bad learning material and make it redirect to new learning material (no ExternalMethods, no Install.py, no plone_skins, etc.)
• Nuke it from orbit team: I am not sure what this team could do, but I like the feeling of the name

We hope that novice  developers attend the sprint. This way we can use you as guinea pigs and document the pain points on a road to become Plone developer. Don’t worry – we’ll inject you with proper medicine to numb the pain and forget bad memories.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

• The rules for the association are accepted
• The rules include the purpose of the association: the main purpose is to arrange yearly Pycon Finland event
• The association will have its own bank account to deal with Pycon expenses and tickets
• Naturally, the association will also further promote Python in Finland

The meeting is held in 7.5.2011 at the same time with DjangoCom Finland.

If you want to participate our Python group activity please read the note below. Unfortunately it’s in Finnish, but I can provide translation if there are English speaking people who want to partipate:

Klara vappen!

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

• Javascripts are processed through portal_javascripts utility automatically, making them suitable for production deployment (minimized, cached). Plone does all this for you. Also the same rules apply for CSS and portal_css tool.
• A special condition is created in Python code to determine whether to include the script or not on a page: it checks whether we are on a page needing the script and then signals portal_javascript which in turn puts the script to HTML <head> section
• Javascripts are served from a static media folder in a Plone add-on utilizing Grok framework. Static media folder is mapped to unique resource names spaces (++resource++yourpythonmodule.name), so that there are no file conflicts between different add-ons.
• These instructions mainly are designed for Dexterity content type subsystem, but can be used with any content types by replacing the behavior check with a marker interface check

Dexterity content types can use Dexterity behaviors control panel to add new behaviors on your content types. One such a behavior is to include certain Javascript files when these content types are viewed or edited.

The example here shows how to include a Javascript if the following conditions are met

• You are developing a Plone add-on as a Python package (as oppose, to say, edit files directly through-the-web)
• Content type has a certain Dexterity behavior applied on it
• Different files are served for view (visitors) and edit modes (content editing interface)

We import Javascript files to portal_javascripts using GenericSetup file jsregistry.xml:

<?xml version="1.0"?>
<object name="portal_javascripts">

        <!-- View mode javascript -->
        <javascript
                id="++resource++yourcompany.app/integration.js"
                authenticated="False"
                cacheable="True" compression="safe" cookable="True"
                enabled="True" expression="context/@@integration_javascript"
                inline="False"
                />

        <!-- Edit mode javascript -->
        <javascript
                id="++resource++yourcompany.app/integration.edit.js"
                authenticated="False"
                cacheable="True" compression="safe" cookable="True"
                enabled="True" expression="context/@@edit_integration_javascript"
                inline="False"
                />

</object>

Then we create the special conditions using Grok views and Python code. This code would go to yourcompany/app/browser/views.py Python module:

# Zope imports
from Acquisition import aq_inner
from zope.interface import Interface
from five import grok
from zope.component import getMultiAdapter

from yourcompany.app.behavior.lsmintegration import IYourWidgetIntegration

class IntegrationJavascriptHelper(grok.CodeView):
    """ Used by portal_javascripts to determine when to include our
        custom Javascript integration code.

    This view is referred from the expression in jsregistry.xml.
    """

    # The view is available on every content item type
    grok.context(Interface)
    grok.name("integration_javascript")

    def render(self):
        """ Check if we are in a specific content type.

        Check that the Dexerity content type has a certain
        behavior set on it through Dexterity settings panel.

        Alternative, just check for a marker interface here.
        """

        # render() methot is a the only traversable
        # Grok CodeView method. It can be used for rendering
        # HTML code, but also for utility views
        # to return raw Python data

        try:
            # Check if a Dexterity behavior is available on the current context object
            # - if it is not, behavior adapter will raise TypeError
            avail = IYourWidgetIntegration(self.context)
        except TypeError:
            return False

        # If called directly from the browser like
        # http://localhost:8080/Plone/integration_javascript
        # will return HTTP 204 No Content

        return True

class EditModeIntegrationJavascriptHelper(IntegrationJavascriptHelper):
    """ Used by portal_javascripts to determine when to include our custom Javascript
        integration code *on edit pages* only.

    Subclass the existing checked and add more limiting conditions.
    """
    grok.name("edit_integration_javascript")

    def render(self):
        """
        @return True: If this template is rendered "Edit view" of the item
        """

        if not IntegrationJavascriptHelper.render(self):
            # We are not even on the correct content type
            return False

        # This is a hacked together as Plone does not provide a real
        # mechanism to separate edit views to other views.
        # We simply check if the current view URI ends with "edit"

        path = self.request.get("PATH_INFO", "")

        if path.endswith("/edit") or path.endswith("/@@edit"):
            return True

        return False

Question: how much Grok is included with Plone 4.1: at least parts of its are, but would this example work with vanilla Plone 4.01 without including five.grok manually as a dependency?

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

• Low-end VPS come with 512 MB memory or less
• Front front-end server Apache / Nginx / Varnish takes > 100 MB +  min. 20 MB for each child process
• Memecached takes its toll
• MySQL takes 200 – 400 MB
• Each Python / PHP process takes at least 15 MB and you need parallel processes for paraller HTTP requests (FCGI, pre-fork, others… )
• Operating system processes need some memory (SSH, cron, sendmail)

As you can see it gets very crowded in 512 MB.

It’s especially troublesome since the memory is allocated lazily and the memory usage builds up slowly. In some point caches are no longer caches, but swapped to a disk – virtual memory usage grows beyond available RAM. To keep the server response, everything time critical should fit to RAM once and if the processes themselves don’t know how to release memory in this situation you need to tune a memory cap for them.

MySQL memory consumption

MySQL can be a greedy bastard what comes to memory consumption. Here on this server MySQL seems to take 417M virtual memory which seems to be little excessive for just running two WordPress instances and one Django / Python application:

1310 mysql     20   0  417M 21100  2776 S  0.0  1.2  0:00.00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/v

After some tuning I was able to bring it down a bit

3354 mysql     20   0  276m  19m 2848 S    0  1.2   3:41.19 mysqld

A reduction of 130 MB, or 1/4 of the server total memory. Not bad.

Use mtop to monitor running MySQL, its querieries, etc. so you know what’s going on. As you can see this MySQL has very good cache rate meaning that basically it is keeping everything in memory. If the content of the sites is less than 10 MBytes total, 400 MB contains plenty of space to cache the content:

load average: 0.05, 0.08, 0.16 mysqld 5.0.51a-3ubuntu5.8-log up 1 day(s), 19:47 hrs                                                             
2 threads: 1 running, 6 cached. Queries/slow: 187.1K/0 Cache Hit: 99.39%

What eats memory

I am not an expert on MySQL, so I hope someone with more insight could post comments regarding how to tune MySQL for low memory situations and how it is expected to behave.

Some ideas I run through my head

• MySQL default cache settings are not too tight on Ubuntu/Debian, making it suitable for moderate loads, not low loads. If you don’t have much content, everything is just kept in memory (even if not needed)
• MySQL uses round robin for connections and if there is 100 max connections it will allocate a thread stack for each connection (someone please confirm this – I found contracting infos).

Configuring MySQL

Here are listed some methods how to reduce the memory usage. This is what I done on this little box

MySQL is mostly configured in /etc/mysql/my.cnf on Ubuntu / Debian.

• Let’s halve key_buffer from 16M to 8M. It is used by MyISAM table cache.
• Halve query_cache_size. query_cache_size  = 8M. Also, decrease query_cache_limit to 512 K.
• Each connection, even if idle, will have 256 KB buffer. Decrease the number of max. connections. (XXX: not sure about this). Drop max connections from 100 -> 30, as we do not have that many concurrent visitors on the site. Also, set less aggressive thread_stack size.

The final adjustments

key_buffer              = 8M
max_connections         = 30
query_cache_size        = 8M
query_cache_limit       = 512K
thread_stack            = 128K

More info

• Fine-tuning MySQL
• Showing what database engine MySQL tables are using
• MySQL server memory usage
• Reducing MySQL memory usage for low end boxes

Send in more tips please! Is 32-bit better than 64-bit for low end VPS, how much this affects MySQL?

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

• Because of duck-typing, you should rigorously document how methods should be called (try epytext and its fields).
• Most open source Python projects do the exact opposite
• Even Python standard library is poorly documented and sets a very bad example (missing manual ???)
• Thus, programming in Python becomes nightmare of grepping through source code (the implementation) or stepping into it in pdb just to figure out how APIs should work (Plone/Zope, anyone?)

Should Python community stop in some point to focus on delivering better documentation instead of focusing on new features and goodies (like the syntax moratorium which was recently lifted)?

From my personal experience

• The best, and the only, person to document the code adequately is the person how originally wrote it
• Because the author already knows how to use the code he doesn’t need to care about the fact how to enable the code for other users.  Many libraries and projects are driven by “scratching your own need” mentality, not by “let’s make this a happy community” mentality. The exception is something like Facebook or Google whose sole purpose is to attract new users their platform bringing in new €€€.
• If you are developing a framework or community project make the documentation a requirement for deliverable and stick with it. If you let one person to skip one hour of writing documentation you are making 10 persons spending one hour figuring out how to use the damn thing.
• Doctests are not documentation. They are tests. They are extremely unreadable way to say “how I should use this thing”, because doctests are often executed in the context of test stubs which do not reflect connections to the other parts of the framework or real contexts.
• “Buy a book – it tells you everything” business model is not feasible in long run. Books get old. Books are not searchable. People don’t buy books.

The good documentation is a way to differentiate, and win, in the situation where there are competing frameworks. I believe the success of Django was mostly driven by its good documentation.

This points could be applied to other duck-typed, open source driven programming languages (PHP anyone?). With good documentation we can reduce the need of Valium recipes for everyone of us.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Quassel is a distributed IRC client. Distributed in the sense that it keeps your chat sessions on the server side: even if you get disconnected you won’t lose any chat messages. This is a nice feature for people who need to be constantly in touch with the communities.  For IRC veterans, Quassel is like the good ol’ screen + irssi combo, but with an user interface which does not require a degree in Perl scripting. This is a short tutorial for getting or building  the mobile version of Quassel, Quasseldroid for Android.

Everyone how has tried it knows that irssi or any terminal software suits very badly for mobile screens. QuasselDroid fixes this: it provides an user interface which is suitable even for touch screens. Also, with Quassel core on the server side, you do not need to worry about disconnections which often happen with mobile networks, With desktop Quassel and mobile QuasselDroid you should be able to seamlessly leave your desktop, go traveling whilst continuing the on-going dispute of the next most important internet thing with your on-line friends and foes.

This tutorial show how to get binary or source from Github and compile the project for your phone. Note that QuasselDroid is still in very early developing phase and the main purpose of this is to drive more development into QuasselDroid. Do not expect it to be ready for your specific use cases yet! Also, these instructions are tested only on Samsung Galaxy S with Android 2.2.

Quassel is originally based on Qt C++ libraries and uses Qt based serialization protocol to communicate between the server (your IRC session) and client (your phone). QuasselDroid implements Qt protocol from scratch in Java and does not use any Qt libraries on Android. There also existed a prior version which required a proxy server, but this has been fixed for now.

Getting and building QuasselDroid

Before proceeding you need

• A server which allows you to run a background process with more than hundred megabytes of memory usage
• Quassel core installed on the server side (this maintains your IRC session). Core itself comes with decent command-line help. If you get stuck ask help on #quassel channel @ freenode IRC network.
• Preferable Quassal desktop client (Windows, OSX, Linux) for ircing from the desktop

Downloading pre-built binary

Note: there exists a pre-built Android binaries for non-Android hackers. Download QuasselDroid Android binary from here. You still need to set-up the server side core yourself.

Prerequisites for building and running QuasselDroid

This is the recommend method of installing QuasselDroid if you are looking for helping in QuasselDroid development. It is not that difficult, as Android tools are quite friendly for newcomers.

• OSX or Linux (your mileage on Windows may vary)
• Android Phone (Android 2.1 or newer)
• Android SDK
• Eclipse IDE for Java Developers
• Android Eclipse plug-in (you need to install it through Find and Install updates inside Eclipse)
• Git (for OSX use Macports to install it)
• Minimal understanding of terminal and UNIX style command line

Compiling and running

Install Android SDK, Eclipse  Android plug-ins for Eclipse. Note this is several hundreds of megabytes worth of software, so be patient and entertain yourself with some cool music whilst downloading.

Install Quassel client, Quassel core and get yourself familiar with the noveau IRC experience.

Get code from Github:

git clone https://github.com/sandsmark/QuasselDroid

Open project in Eclipse. Start Eclipse, choose or create any empty directory as workspace of just choose the default one. Then choose File -> Import -> General -> Existing projects into Workspace. In Select Root Directory you need to choose a directory above Github checkout directory. Select QuasselDroid in Projects list (which is now refreshed after selecting root directory).

In this point I had to select the project, choose Project -> Clean from Eclipse menu (there were some errors popping up in Eclipse internal console).

Right click Project -> choose Run as -> Android application.

• If your phone is connected via USB Eclipse will automatically build and install APK directly to your phone
• If you do not have phone connected QuasselDroid will start in the emulator

Installing QuasselDroid

Creating a core entry

When QuasselDroid is run for the first time. press Android Menu key on the start-up screen(this was little hard to discover) and choose Manage cores. Now you need to configure quassel core for the server – for this process you should already have experience with Quassel desktop client.

SSL

Also, you might need to disable SSL support if you are running a core without SSL compiled in. Again, press Android Menu key, choose Preferences and uncheck SSL.

Debugging issues

When you run an issue and you can repeat it reliably with necessary log traces available, file a bug report on Github project page.

Android has a logging tool called LogCat, showing log output from emulator or device. You will see Android log output in Eclipse in LogCat view when

• Emulator is running
• A device is connected via USB, debug mode enabled (from system Settings menu)

Use Eclipse Window -> Show -> Debug perspective to see LogCat window if it’s hidden.

If you launch the application in Eclipse using Debug run (the bug icon) Eclipse will stop on the line where an uncaught Java exception is thrown. This is very useful for debugging crashing bugs (Android pops up force close dialog).

On the server side, you can see quasselcore output in stdout and see how the core reacts to client connection attempts.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

• Your program will run faster…
• … faster…
• and ..faster!

Here are quick instructions for using PyPy on OSX. You should be able to apply these instructions to other UNIX systems as well.

Install PyPy by getting binary tarball.

wget http://pypy.org/download/pypy-1.4.1-osx64.tar.bz2
tar -xjf pypy-1.4.1-osx64.tar.bz

Install Distribute / easy_install command (setuptools fork) in order to install third party libraries.

cd pypy-1.4.1-osx64
wget http://python-distribute.org/distribute_setup.py
bin/pypy distribute_setup.py

Install libraries – easy_install seems to run just fine on PyPy as long as you don’t try to install Python native extensions

bin/easy_install plac # command line parser
bin/easy_install iso8601 # date format parser

Then run your application

~/code/pypy-1.4.1-osx64/bin/pypy exceptionanalysis.py     -s=2010-01-01T00:00:00 /Users/moo/xxx/log/client1.log

In my case the application was a simple log analyzer based on Python generators (code will be posted later). It reads a text file and tries to interpret it. The script utilizes only one CPU core and thus is very CPU bound. The speed comparison results are

• With Python 2.4: Completed in 111.08 seconds
• With PyPy: Completed in 30.39 seconds

That’s almost 4x speed-up!

Things did not work with PyPy

• import multiprocessing – was not hard dependency, needed to comment out in plac_ext

Here is the traceback if anyone can help:

Traceback (most recent call last):
 File "app_main.py", line 53, in run_toplevel
 File "exceptionanalysis.py", line 20, in <module>
 import plac
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/plac-0.8.0-py2.5.egg/plac.py", line 35, in <module>
 from plac_ext import Interpreter, import_main, ReadlineInput, stdout, runp, Monitor
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/plac-0.8.0-py2.5.egg/plac_ext.py", line 7, in <module>
 import itertools, traceback, multiprocessing, signal, threading
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/multiprocessing-2.6.2.1-py2.5-macosx-10.6-i386.egg/multiprocessing/__init__.py", line 87, in <module>
 import _multiprocessing
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/multiprocessing-2.6.2.1-py2.5-macosx-10.6-i386.egg/multiprocessing/_multiprocessing.py", line 7, in <module>
 __bootstrap__()
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/multiprocessing-2.6.2.1-py2.5-macosx-10.6-i386.egg/multiprocessing/_multiprocessing.py", line 6, in __bootstrap__
 imp.load_dynamic(__name__,__file__)
AttributeError: 'module' object has no attribute 'load_dynamic'

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Your terminal on Linux or OSX may end up to a bad state.

• You cat a binary file to terminal
• You interrupt pdb or some other application reading lines in a bad way

Symptoms

• New lines don’t work
• You cannot see your own typing to terminal
• Backspace stops working

Fix

Type command

stty sane

This may fix some problems without needing to restart the terminal. More info.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter