Putting views, like sitemap, into Plone content tree using Easy Template add-on

Plone has two kind of pages

• Content pages which have a path and will appear in the navigation and in the sitemap. These are stored in the database.
• View based pages and template based pages which usually present an action  (accessibility, sitemap, contact info form). They do not appear in the navigation. They are stored as source code on the file system. You cannot navigate to view based pages and just click edit. To change them you need to use various customization methods (add-on product, Zope management interface) to modify the code.

Sometimes it is desirable, for the sake of uniformness, to put view based pages (accessibility, sitemap) into the content tree. For example, one could want to have the sitemap link appearing only in the navigation tree under the site section “About this site”.

Plone add-on product Easy Template provides an easy method to show any Plone view(s) on a normal page. Easy Template uses Django like template syntax (Jinja 2 engine). It gives you great power to drop dynamic content easily on pages. Easy Template also has some security awarness ensuring the members using it cannot escape from their sandbox.

Easy Template works in WYSIWYG and non-WYSIWYG modes

• You can directly mix templates into text in Visual Editor (Kupu). This is mostly useful for non-HTML aware content editors, who use WYSIWYG editor and can use snippets from a reference card prepared by a developer. Note: Visual Editor has some limitations or undesired behavior. Sometimes it tries to put arbitary HTML tags into text (  which breaks the template code).
• You can write templatized HTML source code in “raw” mode. You can write source code on “Template” schemata in Edit view.

Example how to show a sitemap on an arbitary Plone page

1. Install Easy Template (if you are a developer I suggest you to try trunk version)
2. Create a Templated document content
3. Write some arbitary text in Kupu
4. Put in the code snippet {{ view(“sitemap”, “createSiteMap”) }} which triggers the sitemap view rendering
5. Save and view the document in View mode

It turns out to be:

There is no such thing as a “views reference” for Plone. View names and functions can be figured out by searching and reading through ZCML and Python files in Plone source tree. Some developer insight is needed. For example. for sitemap we can do the grep search:

grep -Ri --include="*.zcml" sitemap *

Then read Products/CMFPlone/browser/configure.zcml and Products/CMFPlone/browser/sitemap.py.

The same thing works in portlets. Use Templated Portlet portlet type. See Easy Template PyPi homepage for the full reference of the product’s potential.

About the author Mikko Ohtamaa

• LinkedIn
• Twitter

Setup.py sdist not including all files

Setuptools has many silent failure modes. One of them is failure to include all files in sdist release (well not exactly a failure, you could RTFM, but the default behavior is unexpected). This post will serve as a google-yourself-answer for this problem, until we get new, shinier, Distribute solving all of our problems.

I b0rked the release for plonetheme.twinapex. Version 1.0 package didn’t include media assets and ZCML configuration files. Luckily Python community reacted quickly and I got advised how to fix it.

By default, setuptools include only *.py files. You need to explicitly declare support for other filetypes in MANIFEST.in file.

Example MANIFEST.in (plonetheme, built in PyDev):

recursive-include plonetheme *
recursive-include docs *
global-exclude *pyc
global-exclude .project
global-exclude .pydevproject

About the author Mikko Ohtamaa

• LinkedIn
• Twitter

XHTML mobile profile transformer and cleaner for Python

Mobile phones, and especially mobile site validators, are very picky about the validy of XHTML. It must not be any XHTML, but special mobile profile XHTML. Also, search engines like Google, will punish you in the mobile search results if your site fails to conform to mobile profile.

This is especially troublesome if you display external content (RSS feeds, ATOM feeds) on your mobile site. Incoming HTML cannot be guaranteed to follow any specification.

To solve this problem, we have created gomobile.xhtmlmp Python library which helps you to transform any HTML to content to valid XHTML MP. The library is piloted on plonecommunity.mobi site which  uses aggregated content from varying sources. The library is based on lxml.html.Cleaner. The library is part of GoMobile project which aims to create world class Python mobile web development tools.

Highlights

• Turn any incoming HTML/XHTML to mobile profile compatible
• Enforce ALT text on images – especially useful for external tracking images (feedburner tracker). ALT texts are required by XHTML MP.
• Protect against Cross-Site Scripting Attacks (XSS) and other nastiness, as provided by lxml.xhtml.clean
• Unicode compliant – eats funky characters

As an example we integrated gomobile.xhtmlmp  to Feedfeeder Plone add-on product.

Enjoy.

Helping people and writing Plone developer API manual – killing two birds with one stone

Plone developer API documentation could be better. For this very reason, people ask questions regarding how to do a simple thing or two in Plone code. Asking questions is often the only way to fight through the monstrous codebase. Luckily these questions usually receive answers from the active community in product-developers@ list and IRC.  See also prior mailing list dicussion.

However, asking the same questions again and again, or asking the question in the first place, is undesirable way to proceed. A proper way to fix the problem would be have a proper documentation voiding the need to ask the question in the first place. Since documenting Plone API is a huge task, I propose the following to get things bootstrapped:

1. Wherever and whenever someone asks a question regarding Plone code development do not answer directly

2. Instead, write the example code snippet and commit it to https://svn.plone.org/svn/collective/collective.developermanual/trunk (collective commit access needed)

3. Give the link to the SVN trunk file as an answer This way we should slowly start building up a “developer reference” which covers the most common API use cases.

https://svn.plone.org/svn/collective/collective.developermanual/trunk is a mess. Do not care about this little detail. Just toss in your .txt files. The docteam and I will properly proof-read and structure it in the future. As soon as we code snippets keep flowing in I am happy!

Example

Here is how I helped some unlucky person struggling with DataGridFrield and created some documentation during the process.

Private email question:

I have been using your excellent DataGridField add-on on a projet, where as webmanager I add variable lists of information within an objet.  However now I would like to build a script that would insert new rows and content automatically : let’s say a User calls a specific objet ‘A’ from time to time, I would like to have a DataGridField that will record incrementally these calls (with columns like “when”, “how long”, “what result”) each time they are made, without the User having to do anything.

I tried to find an answer in the code but I’m not fluent enough in Python yet to understand everything I read.  Could you tell me how I could build that script or where shall I look for clues ?

Private email answer:

Please also read some guidelines I tried to come up for documentation.

Debugging Plone with Eclipse and PyDev remote debugger

Plone is a very heavy Python CMS containing thousands of modules. PyDev is Python extension for Eclipse IDE. Plone doesn’t run very well in PyDev debug mode:

• Debug mode is many times slower than normal Python execution
• At least on  some Linux system OS limitations prevent proper running [1]

But you can still use Eclipse and PyDev remote debugger to debug your software.

• All communications happen over sockets so you can debug software running on test server besides local machine
• Downsite is that you need to manually place pydevd.settrace() call to your code instead of just right clicking the line and choosing Set Breakpoint from menu

In both cases you need to use a custom non-forking Plone launch script from Plone and Eclipse tutorial which prevents Zope to escape itself into a child process and keeps the debug process information intact.

PyDev Extension remote debugger

Note that you need the commercial PyDev Extension license, this does not work with the basic plug-in. Free evaluation trial available. Get it here.

Benefits

• GUI stepping support
• Inspect Python run-time environment and variables using GUI
• Better integration with Eclipse console

Setting up local remote debugger

0. Install Eclipse. Do a local install, do not trust your Linux distribution package installation. http://johnpaulett.com/2009/06/26/install-eclipse-galileo-3-5-on-ubuntu-jaunty-9-04/

1. Set up idelauncher.py for your Zope application server

2. Right-click your project in Eclipse. Put pydevd in your PyDev – PYTHONPATH -> External libraries. Mine (local Eclipse 3.5 install) is /home/moo/eclipse3.5/plugins/org.pyhon.pydev.debug.XXX/pysrc

3. Switch to Eclipse Debug perspective. Start pydevd server by clicking the icon with a bug and P letter.

3. Start your Plone instance / unit tests idelauncher.py using Run (play icon – not debug). The debug server stays active even if you terminate the application launcher run.

4. When your code runs to *import pydevd ; pydevd.settrace()* (no underscore in settrace()) you should be able to start examining your site in the debugger

Now you can

• Examine and edit local variables in Variables view
• Execute Python console commands
• Examine different threads (kind of pointless with Plone…)

Note: Python console behaves funnily. You need to enter the commands to Console -> Display selected console (computer icon) -> Debug server view and output will appear  Console -> Display selected console (computer icon) -> your launcher name view.

Remote debugging not enabled

The following exception means that your pydevd server is not running. Click the P bug icon before running the launcher.

Traceback (most recent call last):
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd.py", line 624, in trace_dispatch
    return dbFrame.trace_dispatch(frame, event, arg)
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd_frame.py", line 107, in trace_dispatch
    raise
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd_frame.py", line 102, in trace_dispatch
    self.doWaitSuspend(thread, frame, event, arg)
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd_frame.py", line 25, in doWaitSuspend
    self._args[0].doWaitSuspend(*args, **kwargs)
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd.py", line 532, in doWaitSuspend
    def doWaitSuspend(self, thread, frame, event, arg): #@UnusedVariable
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd.py", line 539, in doWaitSuspend
    self.writer.addCommand(cmd)

Pydevd and Linux Resource temporarily unavailable

[1] I get the following Resource temporarily unavailable traceback when I try to run PyDev debugger against Plone – I tried to pindown the reason using strace, but no luck. Applies both for unit testing and for the actual instance launch.

Installing PloneLanguageTool ... done (0.174s)
Running Products.SitsHospital.tests.rememberbase.RememberProfileLayer tests:
  Set up Products.PloneTestCase.layer.ZCML
Traceback (most recent call last):
...
  File "/home/moo/sits/parts/zope2/lib/python/zope/testing/testrunner.py", line 688, in setup_layer
    setup_layer(base, setup_layers)
  File "/home/moo/sits/parts/zope2/lib/python/zope/testing/testrunner.py", line 692, in setup_layer
    layer.setUp()
  File "/home/moo/sits/parts/plone/PloneTestCase/layer.py", line 17, in setUp
...
  File "/usr/lib/python2.4/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.4/xml/sax/expatreader.py", line 207, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.4/xml/sax/expatreader.py", line 348, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/moo/sits/parts/zope2/lib/python/zope/configuration/xmlconfig.py", line 349, in endElementNS
    self.context.end()
  File "/home/moo/sits/parts/zope2/lib/python/zope/configuration/config.py", line 544, in end
    self.stack.pop().finish()
  File "/home/moo/sits/parts/zope2/lib/python/zope/configuration/config.py", line 692, in finish
    actions = self.handler(context, **args)
  File "/home/moo/sits/parts/plone/PlacelessTranslationService/patches.py", line 27, in compile_translations
    func(*args, **kwargs)
  File "/home/moo/sits/eggs/zope.i18n-3.7.0-py2.4.egg/zope/i18n/zcml.py", line 57, in registerTranslations
    for language in os.listdir(path):
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/home/moo/sits/parts/instance/etc/site.zcml", line 5.2-5.37
    ZopeXMLConfigurationError: File "/home/moo/sits/parts/zope2/lib/python/Products/Five/configure.zcml", line 6.2-6.30
    ZopeXMLConfigurationError: File "/home/moo/sits/parts/zope2/lib/python/Products/Five/i18n.zcml", line 24.4-24.52
    OSError: [Errno 11] Resource temporarily unavailable: '/home/moo/sits/parts/zope2/lib/python/zope/app/locales'

Scripting Google analytics for multidomain site

We are running few Plone sites which use top level domain (TLD) to identify the site language.

Like many other CMSs out there,  Plone has only one box to enter Google Analytics script snippet.  It is often desirable to use different tracker for different domain and different language combinations, but Google itself doesn’t provide any fancy generator to create complex page tracking code.

Page tracker code, though looks little difficult when spit out by Google Analytics, is just normal Javascript.  You can make the condition to choose the appropriate page tracker id in Javascript itself using document.location property and this way you don’t need to mess with your page templates to create separate tracking Javascript snippet slots.

Here is an example what you can toss in to Plone site setup -> site -> JavaScript for web statistics support:

<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>

<script type="text/javascript">
try {
// Choose page tracker id according to domain
  var domains = document.location.hostname.split(".");
  var tld = domains[domains.length-1];
if(tld == "fi") {
  // .fi
  pageTracker = _gat._getTracker("UA-8819100-1");
} else {
  // .com
  pageTracker = _gat._getTracker("UA-8819100-4");
}
pageTracker._trackPageview();
} catch(err) {
}

</script>

This is used on www.twinapex.com and www.twinapex.fi sites.

Use console.log(err) to output possible Javascript in catch {} errors using Firebug.

Autoconfiguring dual monitors on Ubuntu

The following shell script is a helper script for laptop users who connect an external monitor now and then to their laptop. Since Ubuntu does not provide clever ways to arrange desktop or detect connected displays, you need to run the script from terminal when you change your monitor configuration.

• detect the numbers of monitors you have attached
• Activate the second (external) monitor if there are two monitors
• Use the external monitory as the primary display, otherwise use internal display as a primary display
• Move your taskbars to the primary display

It is based on disper tool by Willem van Engen. For now (2009), it’s nVidia only. ATI support could be possible.

#!/bin/sh
#
# Detect displays and move panels to the primary display
#
# Copyright 2009 Twinapex Research
#
# Author <mikko.ohtamaa@twinapex.com>
#

# disper command will detect and configure monitors
disper --displays=auto -e

# parse output from disper tool how many displays we have attached
# disper prints 2 lines per displer
lines=`disper -l|wc -l`

display_count=$((lines / 2))

echo $display_count

echo "Detected display count:" $display_count

# Make sure that we move panels to the correct display based
# on the display count
if [ $display_count = 1 ] ; then
   echo "Moving panels to the internal LCD display"
   gconftool-2 \
        --set "/apps/panel/toplevels/bottom_panel_screen0/monitor" \
        --type integer "0"
   gconftool-2 \
        --set "/apps/panel/toplevels/top_panel_screen0/monitor" \
        --type integer "0"
else
   echo "Moving panels to the external display"
   gconftool-2 \
        --set "/apps/panel/toplevels/bottom_panel_screen0/monitor" \
        --type integer "1"
   gconftool-2 \
        --set "/apps/panel/toplevels/top_panel_screen0/monitor" \
        --type integer "1"
fi

How to optimize your mobile site visibility

SEOptimize has an interesting post containing lots of resources for mobile internet growth, mobile site search engine optimizations and mobile web design. Keep this under your pillow, mobile folks!

The state of mobile browsers

Very interesting presentation from Peter-Paul Koch of Quirksmode.

For fancy smartphones, only?

During the spring and summer we have been busy with several projects where our Apex Vertex mobile publishing platform has been used for mobile marketing micro sites, integrated with several other content publishing solutions and databases etc.

In most of the cases there has been discussions with the clients about the end users. Is there enough users and what kind of mobile phones they do use and what should be the expectations for the coming services and how them should be taken even further in the near future.

Well, I don’t go into all those things now but just found out from my archives one a bit old but good “snapshot” of mobile internet use. One of the most interesting things is that 70%  of those mobile internet users are not using smartphones but more “normal” mobile phones (well, some might want to define what that standard means…).