Old web browsers do not support Allow-acces-origin HTTP header needed to do cross-domain AJAX requests (IE6, IE7).

Below is an example how to work around this for jQuery getJSON() calls by

• Detecting browsers which do not support this using jQuery.support API
• Doing an alternative code path through a local website proxy view which uses Python urllib to make server-to-server call and return it as it would be a local call, thus working around cross-domain restriction

This example is for Plone/Grok, but the code is easily port to other Python web frameworks.

Javascript example

/**
 * Call a RESTful service vie AJAX
 *
 * The final URL is constructed by REST function name, based
 * on a base URL from the global settings.
 *
 * If the browser does not support cross domain AJAX calls
 * we'll use a proxy function on the local server. For
 * performance reasons we do this only when absolutely needed.
 *
 * @param {String} functionName REST function name to a call
 *
 * @param {Object} Arguments as a dictionary like object, passed to remote call
 */
function callRESTful(functionName, args, callback) {

    var src = myoptions.restService + "/" +functionName;

    // set to true to do proxied request on every browser
    // useful if you want to use Firebug to debug your server-side proxy view
    var debug = false;

        console.log("Doing remote call to:" + src)

        // We use jQuery API to detect whether a browser supports cross domain AJAX calls
        // http://api.jquery.com/jQuery.support/
        if(!jQuery.support.cors || debug) {
                // http://alexn.org/blog/2011/03/24/cross-domain-requests.html
                // Opera 10 doesn't have this feature, neither do IExplorer < 8, Firefox < 3.5

                console.log("Mangling getJSON to go through a local proxy")

                // Change getJSON to go to our proxy view on a local server
                // and pass the orignal URL as a parameter
                // The proxy view location is given as a global JS variable
                args.url = src;
                src = myoptions.portalUrl + "/@@proxy";
        }

        // Load data from the server
        $.getJSON(src, args, function(data) {
                // Parse incoming data and construct Table rows according to it
                console.log("Data succesfully loaded");
                callback(data, args);

     });

}

The server-side view:

import socket
import urllib
import urllib2
from urllib2 import HTTPError

from five import grok
from Products.CMFCore.interfaces import ISiteRoot
from mysite.app import options

class Proxy(grok.CodeView):
    """
    Pass a AJAX call to a remote server. This view is mainly indended to be used
    with jQuery.getJSON() requests.

    This will work around problems when a browser does not support Allow-Access-Origin HTTP header (IE).

    Asssuming only HTTP GET requests are made.s
    """

    # This view is available only at the root of Plone site
    grok.context(ISiteRoot)

    def isAllowed(self, url):
        """
        Check whether we are allowed to call the target URL.

        This prevents using your service as an malicious proxy
        (to call any internet service).
        """

        allowed_prefix = options.REST_SERVICE_URL

        if url.startswith(allowed_prefix):
            return True

        return False

    def render(self):
        """
        Use HTTP GET ``url`` query parameter for the target of the real request.
        """

        # Make sure any theming layer won't think this is HTML
        # http://stackoverflow.com/questions/477816/the-right-json-content-type
        self.request.response.setHeader("Content-type", "application/json")

        url = self.request.get("url", None)
        if not url:
            self.request.response.setStatus(500, "url parameter missing")

        if not self.isAllowed(url):
            # The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeate
            self.request.response.setStatus(403, "proxying to the target URL not allowed")
            return

        # Pass other HTTP GET query parameters direclty to the target server
        params = {}
        for key, value in self.request.form.items():
            if key != "url":
                params[key] = value

        # http://www.voidspace.org.uk/python/articles/urllib2.shtml
        data = urllib.urlencode(params)

        full_url = url + "?" + data
        req = urllib2.Request(full_url)

        try:

            # Important or if the remote server is slow
            # all our web server threads get stuck here
            # But this is UGLY as Python does not provide per-thread
            # or per-socket timeouts thru urllib
            orignal_timeout = socket.getdefaulttimeout()
            try:
                socket.setdefaulttimeout(10)

                response = urllib2.urlopen(req)
            finally:
                # restore orignal timeoout
                socket.setdefaulttimeout(orignal_timeout)

            # XXX: How to stream respone through Zope
            # AFAIK - we cannot do it currently

            return response.read()

        except HTTPError, e:
            # Have something more useful to log output as plain urllib exception
            # using Python logging interface
            # http://docs.python.org/library/logging.html
            logger.error("Server did not return HTTP 200 when calling remote proxy URL:" + url)
            for key, value in params.items():
                logger.error(key + ": "  + value)

            # Print the server-side stack trace / error page
            logger.error(e.read())

            raise e

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

• All users are not interested in the information and do not necessary read the article long enough to see it
• By lazily loading such elements one can speed up the initial page load time
• You save bandwidth
• If you use AJAX for the dynamic elements of the page you can more easily cache your pages in static page cache (Varnish) even if the pages contain personalized bits

For example, Disqus is doing this (see comments in jQuery API documentation).

You can achieve this with in-view plug-in for jQuery.

Below is an example for Plone triggering productappreciation_view loading when our placeholder div tag becomes visible.

...
<head>
  <script type="text/javascript" tal:attributes="src string:${portal_url}/++resource++your.app/in-view.js"></script>
</head>
...
<div id="comment-placefolder">

 <!-- Display spinning AJAX indicator gif until our AJAX call completes -->

 <p>
 <!-- Image is in Products.CMFPlone/skins/plone_images -->
 <img tal:attributes="src string:${context/@@plone_portal_state/portal_url}/spinner.gif" /> Loading comments
 </p>

 <!-- Hidden link to a view URL which will render the view containing the snippet for comments -->                       
 <a rel="nofollow" style="display:none" tal:attributes="href string:${context/absolute_url}/productappreciation_view" />

 <script>

 jq(document).ready(function() {

   // http://remysharp.com/2009/01/26/element-in-view-event-plugin/                                        
   jq("#comment-placeholder").bind("inview", function() {

     // This function is executed when the placeholder becomes visible

     // Extract URL from HTML page
     var commentURL = jq("#comment-placeholder a").attr("href");

     if (commentURL) {
     // Trigger AJAX call
       jq("#comment-placeholder").load(commentURL);
     }

   });                                     

 });     
 </script>
</div>

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Start Google Chrome with no security from command lin, OSX:

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --disable-web-security

Start Google Chrome from command line, Ubuntu/Linux:

chromium-browser --disable-web-security

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Well… then there was an error. KSS validation was not working for the text fields on a certain pages…. or it did sometimes. We were not sure. This is so called Heisenbug. I armed myself with sleepy eyes, Firebug and a lot of energy drinks.

I saw a KSS error in the Firebug log window and failed HTTP POST in the server logs.

Invalid request.

The parameter, value, was omitted from the request.

Looks like the field value was not properly posted for the field validation.

The first thing was locate the error and get function traceback for the messy situation. Unfortunately Firefox Javascript engine or Firebug cannot show tracebacks properly… the grass is so much greener on the other side of the fence. So I had to manually search through the codebase by manually plotting console.log() calls here and there.

Finally I thought I pinpointed the cause of the failure. By shaking finger (excitement, tireness and all that extra caffeinen from energy drinks), I opened the Javascript file just to realize why Javascript is utterly utterly shitty and why no sane person wants to do low level Javascript development. If ECMA standard committee had been clever and had been able to enforce anything long time ago, the following piece could be replaced with one function call.

fo.getValueOfFormElement = function(element) {
    // Returns the value of the form element / or null
    // First: update the field in case an editor is lurking
    // in the background
    this.fieldUpdateRegistry.doUpdate(element);
    if (element.disabled) {
        return null;
    }
    // Collect the data
    if (element.selectedIndex != undefined) {
        // handle single selects first
        if(!element.multiple) {
                if (element.selectedIndex < 0) {
                    value="";
                } else {
                    var option = element.options[element.selectedIndex];
                    // on FF and safari, option.value has the value
                    // on IE, option.text needs to be used
                    value = option.value || option.text;
                }
        // Now process selects with the multiple option set
        } else {
            var value = [];
            for(i=0; i<element.options.length; i++) {
                var option = element.options[i];
                if(option.selected) {
                    // on FF and safari, option.value has the value
                    // on IE, option.text needs to be used
                    value.push(option.value || option.text);
                }
            }
        }
    // Safari 3.0.3 no longer has "item", instead it works
    // with direct array access []. Although other browsers
    // seem to support this as well, we provide checking
    // in both ways. (No idea if item is still needed.)
    } else if (typeof element.length != 'undefined' &&
        ((typeof element[0] != 'undefined' &&
        element[0].type == "radio") ||
        (typeof element.item(0) != 'undefined' &&
        element.item(0).type == "radio"))) {
        // element really contains a list of input nodes,
        // in this case.
        var radioList = element;
        value = null;
        for (var i=0; i < radioList.length; i++) {
            var radio = radioList[i] || radioList.item(i);
            if (radio.checked) {
                value = radio.value;
            }
        }
    } else if (element.type == "radio" || element.type == "checkbox") {
        if (element.checked) {
           value = element.value;
        } else {
            value = null;
        }
    } else if ((element.tagName.toLowerCase() == 'textarea')
               || (element.tagName.toLowerCase() == 'input' &&
                    element.type != 'submit' && element.type != 'reset')
              ) {
        value = element.value;
    } else {
        value = null;
    }
    return value;
};

It turned out that the element in this case was an empty list of radio buttons. When you are tab keying through a radio button group without any value selected, like in the case a content object is just created, KSS validation is triggered even though there is no value in any of the radio buttons. This makes KSS think the value is null and it does not properly handle the situation. This does not cause any user visible effects unless you have Javascript debugging on (Firebug + debugging mode in Plone’s Javascript registry).

But this was not the bug I was looking for. It was just masking the original bug, because I had an empty radio button group next to the text field whose validation was not correctly done. More server side debugging…

I inserted some funky debug prints to Archetypes.Field.validate_validators():

validate_validators()

Calling validators:(('isEmptyNoError', V_SUFFICIENT), ('validDecRange03', V_REQUIRED))

We can see that not triggered validator, validDecRange03, is still with us. Then I add more debug prints to see where things go wrong, this time to to Products.validation.chain.__call__.

Calling validators:(('isEmptyNoError', V_SUFFICIENT), ('validDecRange03', V_REQUIRED))
Name:isEmptyNoError value:234234234234234234234234234232342342342344534232342344234534554 result:True

Ok – we have a case here. isEmptyNoError validator is executed before our custom validator. Since this validator is flagged as “sufficient” other validators are not evaluated. I think this has not been the case before and our validator have worked properly… maybe there was API change in Plone 3.1 which broke the things?

After digging and digging and digging I found this 4 years old bug. Let’s open the famous isEmptyNoError source code in Products.validation.validators.EmptyValidator:

class EmptyValidator:
    __implements__ = IValidator

    def __init__(self, name, title='', description='', showError=True):
        self.name = name
        self.title = title or name
        self.description = description
        self.showError = showError

    def __call__(self, value, *args, **kwargs):
        isEmpty  = kwargs.get('isEmpty', False)
        instance = kwargs.get('instance', None)
        field    = kwargs.get('field', None)

        # XXX: This is a temporary fix. Need to be fixed right for AT 2.0
        #      content_edit / BaseObject.processForm() calls
        #      widget.process_form a second time!
        if instance and field:
            widget  = field.widget
            request = getattr(instance, 'REQUEST', None)
            if request and request.form:
                form   = request.form
                result = widget.process_form(instance, field, form,
                                             empty_marker=_marker,
                                             emptyReturnsMarker=True)
                if result is _marker or result is None:
                    isEmpty = True

        if isEmpty:
            return True
        elif value == '' or value is None:
            return True
        else:
            if getattr(self, 'showError', False):
                return ("Validation failed(%(name)s): '%(value)s' is not empty." %
                       { 'name' : self.name, 'value': value})
            else:
                return False

There is my WTF. Or XXX – thanks for the kisses. My guess is that because KSS validation is executed in special context, the magical REQUEST might not be there. The “is sufficient” validator fails because of the some sort of god forgotten magic and thus all custom validators fail in KSS when the field is not required.

The workaround: I add my own greetings to the code:

        # XXX: This is a temporary fix. Need to be fixed right for AT 2.0
        #      content_edit / BaseObject.processForm() calls
        #      widget.process_form a second time!
        if instance and field:
            widget  = field.widget
            request = getattr(instance, 'REQUEST', None)

            # XXX: Whatever this all does, it does not work for KSS validation
            # requests and we should ignore this
            if "fieldname" in request:
              return False

            if request and request.form:
                form   = request.form
                result = widget.process_form(instance, field, form,
                                             empty_marker=_marker,
                                             emptyReturnsMarker=True)
                if result is _marker or result is None:
                    isEmpty = True

If Zope 3 drives you smoking Plone 3 drives me drinking. No wonder newbies steer away from Plone – if you hadn’t been armed with 8 years of web development experience you would never have figured out what’s going on with such a simple thing as adding a custom validator. A comment added to the bug tracker.