Some examples:

• Django local flavour defines phone number validations for different countries
• Misc. regular expression validators in Plone’s Archetypes content type subsystem
• Every framework define their own email validator, with various regular expressions

Isn’t writing one’s own validation code a bit redundant and exactly “reinventing the wheel” what open source principles so hard try to avoid? Could validation be a low hanging fruit to share among fellow Python projects? As I see it, for the simple data validation, like email and URL, the core code could be easily shared and different Python projects. You basically want just method is_valid_phonenumber(str) and then framework specific way to raise the error to the user.

Do such frameworks already exist? At least I haven’t seen one being used in any big Python project yet

… or is validation so complex thing, so that validation functions must be tightly integrated with the parent framework and I am missing some big things (like locales, etc.) here?

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

• Low-end VPS come with 512 MB memory or less
• Front front-end server Apache / Nginx / Varnish takes > 100 MB +  min. 20 MB for each child process
• Memecached takes its toll
• MySQL takes 200 – 400 MB
• Each Python / PHP process takes at least 15 MB and you need parallel processes for paraller HTTP requests (FCGI, pre-fork, others… )
• Operating system processes need some memory (SSH, cron, sendmail)

As you can see it gets very crowded in 512 MB.

It’s especially troublesome since the memory is allocated lazily and the memory usage builds up slowly. In some point caches are no longer caches, but swapped to a disk – virtual memory usage grows beyond available RAM. To keep the server response, everything time critical should fit to RAM once and if the processes themselves don’t know how to release memory in this situation you need to tune a memory cap for them.

MySQL memory consumption

MySQL can be a greedy bastard what comes to memory consumption. Here on this server MySQL seems to take 417M virtual memory which seems to be little excessive for just running two WordPress instances and one Django / Python application:

1310 mysql     20   0  417M 21100  2776 S  0.0  1.2  0:00.00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/v

After some tuning I was able to bring it down a bit

3354 mysql     20   0  276m  19m 2848 S    0  1.2   3:41.19 mysqld

A reduction of 130 MB, or 1/4 of the server total memory. Not bad.

Use mtop to monitor running MySQL, its querieries, etc. so you know what’s going on. As you can see this MySQL has very good cache rate meaning that basically it is keeping everything in memory. If the content of the sites is less than 10 MBytes total, 400 MB contains plenty of space to cache the content:

load average: 0.05, 0.08, 0.16 mysqld 5.0.51a-3ubuntu5.8-log up 1 day(s), 19:47 hrs                                                             
2 threads: 1 running, 6 cached. Queries/slow: 187.1K/0 Cache Hit: 99.39%

What eats memory

I am not an expert on MySQL, so I hope someone with more insight could post comments regarding how to tune MySQL for low memory situations and how it is expected to behave.

Some ideas I run through my head

• MySQL default cache settings are not too tight on Ubuntu/Debian, making it suitable for moderate loads, not low loads. If you don’t have much content, everything is just kept in memory (even if not needed)
• MySQL uses round robin for connections and if there is 100 max connections it will allocate a thread stack for each connection (someone please confirm this – I found contracting infos).

Configuring MySQL

Here are listed some methods how to reduce the memory usage. This is what I done on this little box

MySQL is mostly configured in /etc/mysql/my.cnf on Ubuntu / Debian.

• Let’s halve key_buffer from 16M to 8M. It is used by MyISAM table cache.
• Halve query_cache_size. query_cache_size  = 8M. Also, decrease query_cache_limit to 512 K.
• Each connection, even if idle, will have 256 KB buffer. Decrease the number of max. connections. (XXX: not sure about this). Drop max connections from 100 -> 30, as we do not have that many concurrent visitors on the site. Also, set less aggressive thread_stack size.

The final adjustments

key_buffer              = 8M
max_connections         = 30
query_cache_size        = 8M
query_cache_limit       = 512K
thread_stack            = 128K

More info

• Fine-tuning MySQL
• Showing what database engine MySQL tables are using
• MySQL server memory usage
• Reducing MySQL memory usage for low end boxes

Send in more tips please! Is 32-bit better than 64-bit for low end VPS, how much this affects MySQL?

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

• Because of duck-typing, you should rigorously document how methods should be called (try epytext and its fields).
• Most open source Python projects do the exact opposite
• Even Python standard library is poorly documented and sets a very bad example (missing manual ???)
• Thus, programming in Python becomes nightmare of grepping through source code (the implementation) or stepping into it in pdb just to figure out how APIs should work (Plone/Zope, anyone?)

Should Python community stop in some point to focus on delivering better documentation instead of focusing on new features and goodies (like the syntax moratorium which was recently lifted)?

From my personal experience

• The best, and the only, person to document the code adequately is the person how originally wrote it
• Because the author already knows how to use the code he doesn’t need to care about the fact how to enable the code for other users.  Many libraries and projects are driven by “scratching your own need” mentality, not by “let’s make this a happy community” mentality. The exception is something like Facebook or Google whose sole purpose is to attract new users their platform bringing in new €€€.
• If you are developing a framework or community project make the documentation a requirement for deliverable and stick with it. If you let one person to skip one hour of writing documentation you are making 10 persons spending one hour figuring out how to use the damn thing.
• Doctests are not documentation. They are tests. They are extremely unreadable way to say “how I should use this thing”, because doctests are often executed in the context of test stubs which do not reflect connections to the other parts of the framework or real contexts.
• “Buy a book – it tells you everything” business model is not feasible in long run. Books get old. Books are not searchable. People don’t buy books.

The good documentation is a way to differentiate, and win, in the situation where there are competing frameworks. I believe the success of Django was mostly driven by its good documentation.

This points could be applied to other duck-typed, open source driven programming languages (PHP anyone?). With good documentation we can reduce the need of Valium recipes for everyone of us.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

To have the über-experience of customer engagement with mobile push notifications you need

• A mobile application (iOS, Android 2.2)
• RSS feed to notifications middleware server (our solution)
• RSS feeds themselves
• Windows/UNIX server running the middleware

How it is put together

Tornado web server is used to handle incoming HTTP requests in scalable manner.

feedparser library fetches RSS feeds and processes them to client notifications.

BitReader (post, source) library is used to create messages to interact with Apple push notification service (APNs). The protocol is bit based protocol running directly over TCP/IP. Apple service has been designed to handle high volumes of traffic – it does not use anything like stateless HTTP to waste bandwidth.

Django models are used to store the state of each individual subscriber. Django’s ORM abstraction allows us to use the same middleware for small distributions (< 1000 clients, SQLite database) or big ones (millions of clients, MySQL database). The stored state information includes the subscriber id and the current badge number – the red circle on the app icon showing the count unread posts. When the application is launched, it can decrease its badge number by doing a HTTP call to the server.

Django settings are used to put together required certificates and whether the application is run in sandbox mode.

Walkthrough

There is a core IO loop, running in a separate process, called stream observer. This loops updates fetches RSS feeds’ status and passes updates to Tornado server over HTTP.  With this arrangement, any HTTP capable client can send push notifications.

Tornado handles incoming updates, updates the related subscribe status – how many unread notifications, etc. through exposed Django views. The notification is formatted according to the variables available on the subscriber mobile platform. In Apple’s case, the notification message gets title, badge, sound and a launch image. Payload is checked against hard 256 byte limit.

Then the payload is pushed to Apple servers over TCP/IP protocol. SSL certificaties needed.

A subscriber is registered  when the mobile application is launched. The application asks a subscriber id from Apple servers. Then, this subscriber id is delivered to our middleware over normal HTTP call.

The middleware also handles feedback service which gives you list of devices which have unsubscribed from your service. This way you can cut off notifications from unsubscribed clients. This is also done using BitReader and TCP/IP.

Future

The architecture is built so that different push backends can be included in the service. Android support is on the roadmap and we probably will have Blackberry and Meego support (when/if Nokia announces such a service).

We have currently tested this solution with RSS streams from WordPress and Plone.

We may release source code when it’s ready.

More info

• http://blog.boxedice.com/2009/07/10/how-to-build-an-apple-push-notification-provider-server-tutorial/

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

http://groups.google.com/group/pigfi

As far as I know such thing didn’t exist before. We discussed about the establishment of Python Finnish user group with several Python activists in Sauna Sprint few days ago (the event arranged by EESTEC and Plonistans, mid term report, final report). There used to exist Plone-related mailing list, but that’s pretty much dead now. If there are other parallel efforts I am not aware of them, but I really like to merge with the existing work whether it exists. I know at least Nokia, Finnish universities and several small enterprises are very active Python users in Finland.

Our goal is to establish a vibrant, self-sustaining, Python community of Finland. We want to promote the sharing of Python knowledge across organizational borders, the adoption of Python in corporate and hobbyist cultures and have jolly good time with other Python enthusiasts. This probably means discussion (in Finnish), events and  lots of, lots of, sauna. After all, the community will take the shape what suits it for the best, it is not forced externally.

The short term goal is to gather enough participants to have a meaningful community. So, Join now!. You can use web interface, email or RSS to follow the discussion. Spread the word to make sure that the message reaches everyone. Tweet it. IRC it. Email it to your friends. SMS it to your parents. Go to streets and cry aloud “Rakastan Pythonia.”

For most of us being Python programmer is a choice you have made  or would like to take. Let’s together create an environment where this choice is fun and will generate great personal wealth for all of us.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Because monkey patches do not need a compilation stage, the patch will work with the future versions of the application, assuming the patched function or method is not changed. So you can “safely” update the patched software and the patch will apply to the new version without need to go to command-line to perform some cumbersome commands. However, it is the best practice of open source community to report the bugs and submit the fixing patches, as source code patch, in the corresponding issue trackers.

In Django context, you can use monkey patching to

• Fix bugs or modify features of Django core without touching the source code
• Fix bugs or modify features of Django plug-ins (TinyMCE, filebrowser, Django CMS) without touching the source code

Patches are usually applied when Python does module imports. You have a special module called “monkeypatches.py” and when that is imported, it applies the patches when the module body level code runs. However, it is difficult to find stable import point in Django to run monkey patching. Django does some really evil magic to initialize INSTALLED_APPS, database models and stuff and doing any kind of work during import causes headache.

So I figured out that you can apply monkey patches using middleware. Middleware applies the monkey patch when the first HTTP request hits the process (note that if you run preforked web server like FCGI every process has its own run-time code in memory). This technique, of course, cannot be used to monkey-patch things that happen before middleware processing, but it is not often needed.

Below is an example how to monkey-patch Django CMS to normalize its unicode output. There was an issue with unicode characters and this is a stop-gap measure to fix it. (I think the proper fix would be fix related Cufon font renderin Javascript library).

We add our monkey patcher to loaded middleware in settings.py.

MIDDLEWARE_CLASSES = (
    ...
    'foobar.middleware.FixCufonUnicodeNormalization',
)

The actual monkey patching happens by fiddling with the class code in process_request(). Note that in this particular case we only need to transform the output of the original function, we can simply hold a reference into it, call it and perform our transformation on the result. This way our monkey patch do not hinder the orignal function and is update safe (it does not matter if the code of PlaceholderNode.render method changes).

import unicodedata

# Orignal function we monkey-patched away
_orignal_render = None

def _patched_render(self, context):
    """ Normalize all unicode output of Placeholder nodes in Django templates """

    content = _orignal_render(self, context)

    # http://docs.python.org/library/unicodedata.html
    return unicodedata.normalize("NFC", content)

class FixCufonUnicodeNormalization(object):
    """ Fix issue with Cufon, user-generated HTML and unicode decomposed characters.

http://github.com/sorccu/cufon/issues/issue/133

    """ 

    def process_request(self, request):
        """ Install monkey-patch on demand.

        If monkey-patch has not been run in for this process (assuming multiple preforked processes),
        then do it now.

        """
        from cms.templatetags.cms_tags import PlaceholderNode

        global _orignal_render, _patched_render

        if not _orignal_render:
            # replace one of the class's method with own fixed version
            _orignal_render = PlaceholderNode.render
            PlaceholderNode.render = _patched_render

As far as I know, monkey patching is something PHP cannot do

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

The route, which you can bicycle through is drawn on Google Maps. There are nine  action points where the user can listen to streaming audio clips, with still images, in his/her mobile phone. This is sort of  augmented reality experience: The user sees the real world (where he/she is now bicycling) combined with the historic events (audio playback narrative). For example, at Linnansaari (a location on the route) you’ll see the actual 17th century castle ruins and the narrator tells how the castle exploded when fire, caused by a lighting, reached gunpowder warehouse… boom. The explosion caused stones fly over 400 meters.

Alternatively, the clips are available as podcasts from Oulu Tourism pages. You can download them into your iPod for offline listening and use in conjuction with a paper map. This demostrates interesting mix of multichannel publishing: paper, web, mobile and podcasts.

The tour is bilingual in Finnish and English.

There exists unreleased iPhone application, based on PhoneGap, which allows the user to track his/her location real-time on the web page. We didn’t see it worth of trouble to go through Apple iPhone application review process. When location based service support comes for the browser this feature is indended to be included as the standard HTML5 feature of the service.

There also exists Nokia Series 60 mobile application, based on PyS60 and Series 60 BrowserControl API, which allows the user to track his/her location in real-time. The application provides wrapper around Series 60 WebKit control and allows Javascript to access phone native functions (GPS) over localhost socket communication. Like with Apple, we didn’t see real-time tracking feature interesting enough to go through Symbian Signed process to get our application released. Also, BrowserControl had seriousquality problems and we didn’t consider it stable enough for the end users. Some work is available in PyS60 Community Edition repository.

The service is hosted on Python specific virtual server on Twinapex services server farm.

Features

• Premium content tailored for audio listening
• Dubbed in English and Finnish by a professional voice actor
• Bilingual: English/Finnish
• Adapts for smartphones (WebKit based browsers) and low end phones (XHTML mobile profile browsers)
• Streaming video and audio (RTSP / progressive HTTP download forv iPhone). Different audio quality is provided on depending on the handset features.
• Screen resolution detection based on user agent sniffing. Three different version of images are used.
• Custom Google Maps component for mobile is used. The component adapts for different mobile phones based on sniffing. Features include zoom, show action point, show the current location, search street address name. This component can be published on a request.
• Management interface features include video upload, video transcoding different mobile versions and editing bilingual content
• Apex Vertex handset database is used to detect the user’s mobile phone capabilities
• Apex Vertex logging and traffic analytics capabilities are used for the site statistics

Software stack

• Ubuntu 8.04 Hardy Heron virtual server
• Apache 2.2 / mod_python
• Python 2.5
• Django 1.0
• Django-CMS 1.0
• mobile.sniffer Python package to provide abstraction over different handset databases
• Apex Vertex streaming solution (RTSP based on Darwin streaming server by Apple)
• TinyMCE WYSIWYG editor
• Darwin streaming server
• ExtJS is extensively used in Apex Vertext management interface

Development effort

Development time: Around 100 hours. Three different developers where involved. Used development tools: Eclipse, PyDev, Subclipse, Subversion. There were around five meetings between the content provider and the technology provider. Few beta testing rounds using iPhone application were performed by bicycling in -10 celcius degrees weather (north and so on…). No polar bears were harmed during the creation of this mobile service.

The service is linked in from Oulu Tourism pages and thousands of paper brochures printed for Oulu summer season 2009.

About the author Mikko Ohtamaa

• LinkedIn
• Twitter

• Content pages which have a path and will appear in the navigation and in the sitemap. These are stored in the database.
• View based pages and template based pages which usually present an action  (accessibility, sitemap, contact info form). They do not appear in the navigation. They are stored as source code on the file system. You cannot navigate to view based pages and just click edit. To change them you need to use various customization methods (add-on product, Zope management interface) to modify the code.

Sometimes it is desirable, for the sake of uniformness, to put view based pages (accessibility, sitemap) into the content tree. For example, one could want to have the sitemap link appearing only in the navigation tree under the site section “About this site”.

Plone add-on product Easy Template provides an easy method to show any Plone view(s) on a normal page. Easy Template uses Django like template syntax (Jinja 2 engine). It gives you great power to drop dynamic content easily on pages. Easy Template also has some security awarness ensuring the members using it cannot escape from their sandbox.

Easy Template works in WYSIWYG and non-WYSIWYG modes

• You can directly mix templates into text in Visual Editor (Kupu). This is mostly useful for non-HTML aware content editors, who use WYSIWYG editor and can use snippets from a reference card prepared by a developer. Note: Visual Editor has some limitations or undesired behavior. Sometimes it tries to put arbitary HTML tags into text (&nbsp; which breaks the template code).
• You can write templatized HTML source code in “raw” mode. You can write source code on “Template” schemata in Edit view.

Example how to show a sitemap on an arbitary Plone page

1. Install Easy Template (if you are a developer I suggest you to try trunk version)
2. Create a Templated document content
3. Write some arbitary text in Kupu
4. Put in the code snippet {{ view(“sitemap”, “createSiteMap”) }} which triggers the sitemap view rendering
5. Save and view the document in View mode

It turns out to be:

There is no such thing as a “views reference” for Plone. View names and functions can be figured out by searching and reading through ZCML and Python files in Plone source tree. Some developer insight is needed. For example. for sitemap we can do the grep search:

grep -Ri --include="*.zcml" sitemap *

Then read Products/CMFPlone/browser/configure.zcml and Products/CMFPlone/browser/sitemap.py.

The same thing works in portlets. Use Templated Portlet portlet type. See Easy Template PyPi homepage for the full reference of the product’s potential.

About the author Mikko Ohtamaa

• LinkedIn
• Twitter

• Kupu
• Outgoing email actions (Content rules ones)

The users can place ${title}, ${object_url} and other template in the edit mode. These variables which are directly mapped from Archetypes fields when the content is viewed/sent. Also, one can register custom snippet generators like $list_folder_content.

I hope Easy Template to cover some more actions in the future. I have noted PloneFormGen and Singing & Dancing product authors that we could add some mixed in functionality together.

Currently Easy Template uses Cheetah template backend. Cheetah is not Zope security friendly and exposing templated actions should be allowed only to trusted members. I am not huge fan of Plone’s TAL template language which is based on XML attributes and thus suitable only be used in XML context – this language is aimed only for hardcore hackers and software designers and ordinary folk really cannot wrap their minds around it.

Because I am not sure which will be the chosen template backend in the future I chose to abstract the template engine layer away. I created collective.templateengines product. It is a bunch of Zope interfaces and utility functions to abstract away common template actions like

• Applying a template
• Adding a template context variables
• Registering custom template tags

Currently collective.templateengines supports Cheetah and Django templates.

So, dear audience, what do you think of all this? What template engine would you suggest which would be Kupu friendly – you can edit the template language in WYSIWYG editor? Do you see any other usages for collective.templateengines? Which other projects could adopt template engine abstraction layer?

Buildout is from the creators of Zope (I suppose) so you can expect a powerful project code management tool. The question is, however, whether or not it suits your needs. In my case I found out it too heavy. I mean, to add even a simple task you have to create a new “recipe” (a package) that does the tricks. Of course some recipes are generic (found from PyPi) and you can just run them with your own INI options, but in my case I had to do some custom implementation. Creating a new python package isn’t that hard for sure but there’s of course some learning curve, so the real question is should you spend time to learn it or not?

I found out that zc.buildout has some nice features like:

• Automatic requirements processing through setuptools
• Automatic (yet simple) removal of directories during recipe uninstall
• Clear structure (install(), update() & uninstall() methods)
• INI-syntax, python does have a clear syntax but INI is always clearer for a newbie
• Easy script creation (adjust python paths somewhat automatically)
• Easily repeatable
• Passing of arguments from one recipe to another
• etc.

The problems?

• It takes a while to learn zc.buildout
• It takes ‘another while’ to learn to write recipes
• Too much hassle for little things
• INI-syntax is very limited in features
• Buildout easily updates all your packages (that means also the ones you didn’t want to!)
• Lack of documentation (it has good docs to get you going.. but after a while it leaves you with open questions)
• Unnecessary overhead (for each script you launch, you’ll need a launcher script created via buildout)

There’s no denying zc.buildout is powerful, but I wouldn’t use it for projects which need reasonable amount of customization. It’s just plain easier and quicker to write shell scripts and while those won’t provide you with any sort of ready tools you won’t propably need them. For bringing up somewhat static environment, where you don’t need to hack things (like that for Plone) it’s quite a decent option, however.

I also explored alternatives to zc.buildout. I’ve been reading about earlier virtualenv but haven’t really tried it out until now. It looks very promising and creates a more flexible environment compared to zc.buildout. Of course their goals are not exactly the same. Also, there are a few other alternatives out there, among them a new Python code management tool called Paver (just look at that cool logo.. it does remind you of Indiana Jones, does it not?). I glanced through the Paver docs and it looks like it might be the way to go (Paver also supports virtualenv), but didn’t quite get the grasp of the benefits just yet. Anyway, if you are still interested in code management and deployment, I’d recommend you to read the Paver release announcement and also Paver forewords. They should clear things up.