Below is an example how you use five.grok to install an event handler which checks in the site root for a through-the-web editable Python script. If such script exist it asks it to check for a HTTP redirect.

This behavior allows you to write site-wide redirects easily

• In Python (thank god no Apache regular expressions)
• Redirects can access Plone content items
• You can easily have some redirects migrated from the old (non-Plone) sites

redirect.py:

"""

    Call a custom TTW script and allow it to handle redirects.

    Use Zope Management Interface to add a ``Script (Python)`` item named ``redirect_handler``
    to your site root - you can edit this script in fly to change the redirects.

    * Redirect script must contain ``url`` in its parameter list

"""

import logging

# Now we import things through the last decade...

# Really old old stuff
from zExceptions import Redirect

# Really old stuff
from Products.CMFCore.interfaces import ISiteRoot

# Old stuff
from zope.traversing.interfaces import IBeforeTraverseEvent

# Modern stuff
from five import grok

logger = logging.getLogger("redirect")

@grok.subscribe(ISiteRoot, IBeforeTraverseEvent)
def check_redirect(site, event):
    """
    Check if we have a custom redirect script in Zope application server root.

    If we do then call it and see if we get a redirect.

    The script itself is TTW Python script which may return
    string in the case of redirect or None if no redirect is needed.

    For more examples, check

http://svn.zope.org/Zope/trunk/src/Zope2/App/tests/testExceptionHook.py?rev=115555&view=markup

    """
    request = event.request

    url = request["ACTUAL_URL"]

    if "no_redirect" in request.form:
        # Use no_redirect query parameter to disable this behavior in the case
        # you mess up with the redirect script
        return

    # Check if we have a redirect handler script in the site root
    if "redirect_handler" in site:

        try:
            # Call the script and get its output
            value = site.redirect_handler(url)
        except Exception, e:
            # No silent exceptions plz
            logger.error("Redirect exception for URL:" + url)
            logger.exception(e)
            return

        if value is not None and value.startswith("http"):
            # Trigger redirect, but only if the output value looks sane
            raise Redirect, value

Then an example redirect_handler script added through ZMI. Remember to add url to the Parameter List field of TTW interface. If the script returns a string starting http it is considered as the need for a redirect and normal Plone traversing is interrupted.

if "blaablaa" in url:
    return "http://webandmobile.mfabrik.com"

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Old web browsers do not support Allow-acces-origin HTTP header needed to do cross-domain AJAX requests (IE6, IE7).

Below is an example how to work around this for jQuery getJSON() calls by

• Detecting browsers which do not support this using jQuery.support API
• Doing an alternative code path through a local website proxy view which uses Python urllib to make server-to-server call and return it as it would be a local call, thus working around cross-domain restriction

This example is for Plone/Grok, but the code is easily port to other Python web frameworks.

Javascript example

/**
 * Call a RESTful service vie AJAX
 *
 * The final URL is constructed by REST function name, based
 * on a base URL from the global settings.
 *
 * If the browser does not support cross domain AJAX calls
 * we'll use a proxy function on the local server. For
 * performance reasons we do this only when absolutely needed.
 *
 * @param {String} functionName REST function name to a call
 *
 * @param {Object} Arguments as a dictionary like object, passed to remote call
 */
function callRESTful(functionName, args, callback) {

    var src = myoptions.restService + "/" +functionName;

    // set to true to do proxied request on every browser
    // useful if you want to use Firebug to debug your server-side proxy view
    var debug = false;

        console.log("Doing remote call to:" + src)

        // We use jQuery API to detect whether a browser supports cross domain AJAX calls
        // http://api.jquery.com/jQuery.support/
        if(!jQuery.support.cors || debug) {
                // http://alexn.org/blog/2011/03/24/cross-domain-requests.html
                // Opera 10 doesn't have this feature, neither do IExplorer < 8, Firefox < 3.5

                console.log("Mangling getJSON to go through a local proxy")

                // Change getJSON to go to our proxy view on a local server
                // and pass the orignal URL as a parameter
                // The proxy view location is given as a global JS variable
                args.url = src;
                src = myoptions.portalUrl + "/@@proxy";
        }

        // Load data from the server
        $.getJSON(src, args, function(data) {
                // Parse incoming data and construct Table rows according to it
                console.log("Data succesfully loaded");
                callback(data, args);

     });

}

The server-side view:

import socket
import urllib
import urllib2
from urllib2 import HTTPError

from five import grok
from Products.CMFCore.interfaces import ISiteRoot
from mysite.app import options

class Proxy(grok.CodeView):
    """
    Pass a AJAX call to a remote server. This view is mainly indended to be used
    with jQuery.getJSON() requests.

    This will work around problems when a browser does not support Allow-Access-Origin HTTP header (IE).

    Asssuming only HTTP GET requests are made.s
    """

    # This view is available only at the root of Plone site
    grok.context(ISiteRoot)

    def isAllowed(self, url):
        """
        Check whether we are allowed to call the target URL.

        This prevents using your service as an malicious proxy
        (to call any internet service).
        """

        allowed_prefix = options.REST_SERVICE_URL

        if url.startswith(allowed_prefix):
            return True

        return False

    def render(self):
        """
        Use HTTP GET ``url`` query parameter for the target of the real request.
        """

        # Make sure any theming layer won't think this is HTML
        # http://stackoverflow.com/questions/477816/the-right-json-content-type
        self.request.response.setHeader("Content-type", "application/json")

        url = self.request.get("url", None)
        if not url:
            self.request.response.setStatus(500, "url parameter missing")

        if not self.isAllowed(url):
            # The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeate
            self.request.response.setStatus(403, "proxying to the target URL not allowed")
            return

        # Pass other HTTP GET query parameters direclty to the target server
        params = {}
        for key, value in self.request.form.items():
            if key != "url":
                params[key] = value

        # http://www.voidspace.org.uk/python/articles/urllib2.shtml
        data = urllib.urlencode(params)

        full_url = url + "?" + data
        req = urllib2.Request(full_url)

        try:

            # Important or if the remote server is slow
            # all our web server threads get stuck here
            # But this is UGLY as Python does not provide per-thread
            # or per-socket timeouts thru urllib
            orignal_timeout = socket.getdefaulttimeout()
            try:
                socket.setdefaulttimeout(10)

                response = urllib2.urlopen(req)
            finally:
                # restore orignal timeoout
                socket.setdefaulttimeout(orignal_timeout)

            # XXX: How to stream respone through Zope
            # AFAIK - we cannot do it currently

            return response.read()

        except HTTPError, e:
            # Have something more useful to log output as plain urllib exception
            # using Python logging interface
            # http://docs.python.org/library/logging.html
            logger.error("Server did not return HTTP 200 when calling remote proxy URL:" + url)
            for key, value in params.items():
                logger.error(key + ": "  + value)

            # Print the server-side stack trace / error page
            logger.error(e.read())

            raise e

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Plone is a byproduct of community members' co-operation

These are notes from the Sauna Sprint daily debriefing #2. It was intention we’d done this already yesterday, but unfortunately we had problems with boat and swamp.

The current head count is around 27 living souls, plus or minus few depending on how you define living. We split to different teams, each working on its own topic.

Team tutorial video

This team works to create new Plone video installation and usage tutorials. It will be in the form of screencast with voice over narration.

The team has created the first voice over, screencast and now editing it together.

Team add-ons fail

Plone add-ons come with various quality of instructions. This team is working to make Plone add-on experience more user-friendly: installing, uninstalling, plone.org add-on area, individual add-on instructions.

Around 200 lines of issues collected.

Team blogging

This team collects and publishes a blog posts, participant information, etc. motivational information about the sprint and the sprinters.

Both EESTECers and Plonistans are being covered.

Currently the team has collected the bio of everyone and prepares to put the information out to a wiki page.

Team auto restart

This team is working to make Plone auto-restart perform correctly and timely manner. You just edit .py file and hit the browser refresh button and the changed code is loaded correctly. The team is utilizing the fork trick introduced in this blog post. The resulting work goes under a project name sauna.relaod.

Currently the team has managed to make the file-system monitor using Python WatchDog backend. Already 13 seconds have been shoveled off from Zope restart time of 30 seconds.

Team Logistics

This team feeds the coders with firewood, food and beer.

Yesterday the team prepared went to a shop, prepared a lunch and brought a guitar (for live music).

The day plan for Wednesday is to collect blueberries for pies and mushrooms and later cook them in the big oven of the cabin.

The new beer is on its way, though has temporary taken some steps backwards.

Team Greek

This ad-hoc team prepared the dinner.

It takes four Greeks to create one omelette.

Team collective.table

This is Google Summer of Code team is working on new collective.table add-on for Plone. Yesterday was API talk, but the progress was slow due to lack of caffeine.

Team TinyMCE

This is a Google Summer of Code team and is working on making Plone’s TinyMCE experience better.

Yesterday was spent testing on a JS/CSS compressor to make TinyMCE load faster.

Uncompressed: 19 request / 900 kb. Compressed 50% less requests, 700 kb.

Team hardcore

This team of EESTEC member was formed to introduce hardcode Plone development for newbies.

The team members actually wanted it themselves. They will regret this decision later.

Sometimes you need solo heroes

Plone butterfly preparing to hatch

A lonely coder in the forest

Fueling the hungty developers with potatoes

A statue honoring Plone developers

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

The first day of Sauna Sprint was spent introducing new EESTEC people to Plone. Almost everyone got Ubuntu installed, there was a short crash course for IRC and then we started installing Plone.

We had a bunch of EESTEC students as a test subject and our goal was to map obstacles in the process of installing Plone for local computer for development. We got some interesting results.

The data was collected on whiteboard where everyone was free to express any problem he or she had during the installation. The advisors also monitored the user group for possible problems and made notes.

Aperture Science test subjects installing Plone in hope they’ll get a cake if they success

Good job, but the cake was a lie.

Plone is #%!!”€&! difficult, but we can fix it

The comments below are collected from the whiteboard.

Don’t make assumptions in the documentation – people are tabula rasa (i.e. how to use cd command and so on)

Documentation needed to install gcc, PIL dependencies etc. on OS level

No good instructions how to set-up Python with virtualenv (was being fixed in the sprint). Buildout actually does not work with the system Python out of the box: you need to break your OS http://stackoverflow.com/questions/5818100/buildout-tries-to-update-system-wide-distribute-installation-and-refuses-to-run

People try to run buildout by “cd bin ; ./buildout” – it does not work and the error message is confusing

Unified Installer – ZEO installation is broken (need clarification what was broken)

Detect Python version in buildout and don’t try to run on incompatible version

When creating first Plone site input field has label “Path identifier” – people try to enter Plone FS path there

README.txt in Unified Installer needs updates

ZopeSkel template plone4_buildout should autodetect the latest Plone version (now was 4.0.1)

Update the out of the box Plone installation front page to contain more developer friendly instructions too

plone.org/downloads: Show add-on version compatibility as the first item on the page – if now compatibility set warn that the compatibility is unknown (same kind of warning if no updates for one year)

Have an easy way to report broken packages

Force a process to improve add-on documentation in p.org, especially regarding installation. Only 1/5 themes was successfully installed by the test personel without guidance, mostly due to lack of instructions and version incompatibilities.

Some work in progress on PiratePad regarding the installation of Plone.

Evening program

We had Sauna time in Hervanta, student style. EESTEC lived up to their reputation as party animals.

Delicious pizza was baked, though the audience loudly disagree when the pizza was burnt enough to be eaten

Magic Toni Sause

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Delete the project if you have already the same project (collective.developermanual) imported Eclipse some other way.

Create a Github account (if you don’t have one) and put your SSH public key in the account settings.

Configure Eclipse to use your private Github SSH key in Settings -> General -> Network connections -> SSH2. Eclipse also offers a SSH key generation wizard.

Import collective.developermanual from GitHub using File -> Import -> Git and then use Clone button (not Add). collective.developermanual URL is git@github.com:collective/collective.developermanual.git Change protocol to SSH.

Click next next next until you see the project in your Eclipse workspace. Note that EGit puts projects to your home folder, not workspace (slight argh in this point).

Edit some files.

Right click the project to commit.

Then you can right-click to push. This is the tricky part. The dialogue looks like a USS Aircraft Carrier deck control panel. What you need to do is to

• In Source ref drop down choose master [branch]
• In Destination ref drop down choose master [branch]
• Press Add spec button
• Press Next
• Press Finish

Then it pops up some dialogs which mean that the push was succesful.  Just keep clicking Ok to every window as you won’t understand them in any case.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Here are instructions how to run Firefox Mobile on Ubuntu Linux (tested on 32-bit Ubuntu 10.10)

wget http://releases.mozilla.org/pub/mozilla.org/mobile/releases/latest/linux-i686/fennec-5.0.en-US.linux-i686.tar.bz2
tar -xjf fennec-5.0.en-US.linux-i686.tar.bz2
cd fennec
./fennec

.. and thats all you need. It works out of the box! 400x times faster than using Android emulator browser.

There are also OSX and Windows builds available.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Some examples:

• Django local flavour defines phone number validations for different countries
• Misc. regular expression validators in Plone’s Archetypes content type subsystem
• Every framework define their own email validator, with various regular expressions

Isn’t writing one’s own validation code a bit redundant and exactly “reinventing the wheel” what open source principles so hard try to avoid? Could validation be a low hanging fruit to share among fellow Python projects? As I see it, for the simple data validation, like email and URL, the core code could be easily shared and different Python projects. You basically want just method is_valid_phonenumber(str) and then framework specific way to raise the error to the user.

Do such frameworks already exist? At least I haven’t seen one being used in any big Python project yet

… or is validation so complex thing, so that validation functions must be tightly integrated with the parent framework and I am missing some big things (like locales, etc.) here?

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Preface

Please do not take this post too seriously if I am ranting tongue-in-cheek. This thinking does not reflect the official opinion of Sauna Sprint team any way, it is just my personal rambling.

Background

Plone is having popularity issues. This can be deducted by:

• PHP CMS are more popular (Google trends and so on)
• Plone lacks love
• By discussing with Python programmers outside Plone community: they see Plone as monster

Plone is a superior CMS product (feature wise, polish wise, tech wise, and so on). So how did this happen?

My theory, you are free to disagree, is that the community has been driving “wrong” ideologies

• Plone’s “buyers” are not users, but developers
• There has been no driving community factor to make Plone accessible for more developers. Plone community has been working to make Plone work for the existing community members, not new members. The consulting companies have been too busy to extract money from their existing big clients. It might even be that it’s the best interest of these companies to keep the barrier of entry high, as this way they protect the Plone business for themselves. Note that this strategy will fail in the point there is no longer Plone business.

The big boys are telling that customers are abandoning Plone, because there are no developers. I know a lot of Plone developers, looking for more Plone work.  I’ll translate this that there are not enough affordable Plone developers entering the market.

I am also not buying arguments like

• “don’t compare apples to oranges (Django vs. Plone)”
• “its history and evolved that way”
• “big tool must be complex”

The result

Plone is easy to use, but hard to develop. If you need to use code generators and boiler-plate in Python code, you are doing it wrong.

Plone is a soup of different, non-mainstream, technologies. To be a Plonista you need master: Python (ok docs), Plone user mode (good docs), Zope component architecture (some docs), ZCML (no useful docs), GenericSetup (no docs), TAL (some docs), Archetypes (basically no docs), buildout (aaaaaarghs) … You have too much to learn from material which does not exist. Today the material effectively does not exist if it’s not your first Google hit – I am against book.

New developers are not getting in. It is easier to duplicate basic Plone functionality in a hacky in-house Django CMS than customize Plone for your use case. Plone does not give any more added value to the new developers. There is no business motivation to adapt Plone as a technology platform in new ventures.

The quest

Let’s fix it, as I see the developer acceptance the most crucial challenge Plone community is facing if the community wants to survive.

What we can do here is

• Make Plone easier for developers
• Make more, better, learning material available
• Eliminate and protect against factors in Plone which hinder the goals of two above bullets (do not include new technologies or modules to core if they make developers sad)

To accomplish this we can

1. Promote “easy” technologies inside and outside Plone community (Dexterity over Archetypes, Grok over ZCML)
2. Making a standard workflow available for them (Django style tutorial that do 1, 2, 3 and you are a certified Plone developer)
3. Kill technologies which make the stack unnecessary complex and are not necessarily needed in newbie use cases (buildout, I am looking at you)
4. Replace old, badly documented and difficult solutions with easier ones (could we actually replace TAL with something more popular?)

Operational plan

Here are some concrete tasks what we could tackle in Sauna Sprint.

Make it easy department

• Plone auto-restart team: provide working and fast auto-restart á la Django or Tornado (hint: plone.reload doesn’t cut it – you need real restarts process-wise)
• Default custom product team: ship Plone with a file-system based custom product skeleton, so that when you need to include your first custom Python module you don’t need to learn about buildout and paster first . You actually can copy-paste in Python code and it just works.
• Through-the-web is happy again team: Make it possible to code and execute Python through-the-web so that it actually works.: a working replacement for RestrictedPython

Fix user-wise broken things department

• Buildout team: make buildout more user friendly: spit out error messages which are actually useful, have newbie buildout tutorial from the developer perspective. Nothing bad should happen when buildout is run. Make buildout to confirm version updates, etc.
• Anti-buildout team: how to install eggs without buildout. Why did we need the buildout in the first place?

This is how you do it department

• Tutorial team: Make “gapless” hand-holding tutorial for creating your first view, form and content-type. Make it so that you do not need to look outside the tutorial or learn anything unnecessary (don’t make me think)
• Picture team: draw big pictures of Plone architecture, how different parts are connected, so that new developers get hang on things easily
• Document generator team: Integrate collective.developermanual with Sphinx tool that generate the reference  documentation from the source code: views, viewlets, templates, CSS classes, portlets, etc.

Hasta la vista babe department

• Kill it with fire team: Hunt down all bad learning material and make it redirect to new learning material (no ExternalMethods, no Install.py, no plone_skins, etc.)
• Nuke it from orbit team: I am not sure what this team could do, but I like the feeling of the name

We hope that novice  developers attend the sprint. This way we can use you as guinea pigs and document the pain points on a road to become Plone developer. Don’t worry – we’ll inject you with proper medicine to numb the pain and forget bad memories.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

The event is July 23 – July 30, 2011. Come to #saunasprint IRC channel on Freenode and enlist yourself now by the joining CoActive project and adding your name to the guest list on the wiki style project page!

In the social program we have beer tasting, sauna, wood chopping, installing internet connection and who kills most mosquitos competition.

This year we have secured a sprint location near Tampere city (editorial note: is near a correct adjective here?). The image is not corrupted – the green color is forest.

Here is our mansion:

We have reserved meeting rooms for the sprinters to work:

As Finland is the land of thousand lake and wilderness on all sort you may have a chance to see some majestic wild life.

Reindeers:

Mooses:

Bears:

Mosquitos:

A Plone sprinter:

See you in Sauna Sprint 2011!

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter