Cannot sort custom content item in Plone folder listing

Bug: Plone folder manual sorting does not move items even though you try all tricks. The first suspect would be a Javascript bug, but it isn’t.

It is bug 8161.

Your custom content meta_type must not contain spaces.

You can fix this on-line by editing meta type in portal_types in ZMI and remove all spaces from meta type name.

Subversion global-ignores and .egg-info in Python/Plone development

Subversion does a good job by ignoring most of build/temporary/unwanted files by default.

However, there is one exception still existing at least in Subversion 1.6: Python egg folders. All folders whose name ends up with .egg-info should not committed or considered in version controlling actions. your.package.name.egg-info folder is generated inside your Python egg source folder when you run setup.py / setuptools.

If you are working with Python source code eggs, add the following line to your ~/.subversion/config

global-ignores = *.o *.lo *.la #*# .*.rej *.rej .*~ *~ .#* .DS_Store *.egg-info *.pyc *.pyo .project .pydevproject

Otherwise development tools like Mr. Developer might get confused.

Plone Developer Manual, take #0.1

The first public version of  Plone developer manual is available here.

It is still very much draft, but I assure you will find it useful. You will find it even more useful after you put in the answers for your own problems.

In my previous Plone developer documentation rant my flow of though was little abstract and I couldn’t clearly explain how I want the community to maintain this crucial piece of documentation.  This time I made a comic.

* How to get support

** How to update Plone Developer Manual

Packing and copying Data.fs from production server for local development

These instructions help you to copy and transfer production server  ZODB database (Data.fs) to your local computer for development and testing. This allows you to do the testing against the copy of real data and the production server Plone instance set up.

See the original tip by cguardia.

Data.fs is ZODB file storage for transactional database. Journal history takes quite a lot of disk space there. Packing, i.e. removing the journal history,  usually reduces the size file considerably, making the file lighter for wire transfer. Depending on the database age the packed copy is less than 10% of the original size.

These instructions apply for Ubuntu/Debian based Linux systems. Apply to your own system using the operating system best practices.

We need ZODB Python package to work with the database. To use it, we’ll create virtualenv Python installation in /tmp. In virtualenv installation, installed Python packages do not pollute or break the system wide setup. Note that you might use easy-install-2.4 depending on the OS. The latest stable ZODB can be picked from PyPi listing. Plone 3.x default is ZODB 3.7.x, which is not available as Python egg, but you can use ZODB 3.8.x.

sudo easy-install virtualenv
cd /tmp
virtualenv packer
/tmp/packer/bin/easy_install ZODB=3.8.3

Data.fs cannot be modified in-place. You must create a copy of it to work with it. Data.fs copy can be created from a running system without the fear of corrupting the database, since ZODB is append only database.

cp /yoursite/var/filestorage/Data.fs /tmp/Data.fs.copy

Then create the following script snippet /tmp/pack.py using your favorite terminal editor.

import time
import ZODB.FileStorage
import ZODB.serialize

storage=ZODB.FileStorage.FileStorage('/tmp/Data.fs.copy')
storage.pack(time.time(),ZODB.serialize.referencesf)

And run it using virtualenv’ed Python setup with ZODB installed:

/tmp/packer/bin/python /tmp/pack.py

Lots of patience here… packing may take a while, but it’s still definitely faster than your Internet connection transfer rate.

Verify that the file is succesfully packed:

ls -lh Data.fs.copy
-rw-r--r-- 1 user user 30M 2009-09-01 13:24 Data.fs.copy

Woohoo 1 GB was shrunk to 30 MB. Then copy the file to your local computer using scp and place it to your development buildout.

scp user@server:/tmp/Data.fs.copy ~/mybuildout/var/filestorage/Data.fs

You just saved about 30-90 minutes of waiting of file transfer.

SEO tips: query strings, multiple languages, forms and other content management system issues

This post is collection of search engine optimization tips for content management systems, especially for Plone.

Do not index query strings

It is often desirable to make sure that query string pages (http://yoursite/page?query_string_action=something) do not end up into the search indexes. Otherwise search bots might index pages like site’s own search engine results  (yoursite/search?SearchableText=…) lowering the visibility of  actual content pages.

GoogleBot has regex support in robots.txt and can be configured to ignore any URL ? in it. See the example below.

Query string indexing causes the crawler crawl things like

• Various search results (?SearchableText)
• Keyword lists (?Subject)
• Language switching code (?set_language)… making set_language appear as the document in the search results

Also, “almost” human readable query strings look ugly in the address bar…

Top level domains and languages

Using top level domain name (.fi for Finland, .uk for United Kingdoms, and so on.) to make distinction between different languages and areas is optimal solution from the SEO point of view. Search engines use TLD information to reorder the search results based on where  the search query is performed  (there is difference between google.com and google.fi results).

Plone doesn’t use any query strings for content pages. Making robots to ignore query strings is especially important if you are hosting multilingual site and you use top level domain name (TLD) to separate languages: if you don’t configure robots.txt to ignore ?set_language links only one of your top level domains (.com, .fi, .xxx) will get proper visibility in the search results. For example we had situation where our domain www.twinapex.fi did not get proper visibility because Google considered www.twinapex.com?set_language=fi as the primary content source (accessing Finnish content through English site and  language switching links).

Shared forms

Plone has some forms (send to, login) which can appear on any content page. These must be disallowed or otherwise you might have a search result where the link goes to the form page instead of the actual content page.

Hidden content and content excluded from the navigation

Any content excluded from the sitemap navigation  should be put under disallowed in robots.txt. E.g. if you check “exclude from navigation” for Plone folder remember to update robots.txt also.

In our case, our internal image bank must not end up being indexed, though images themselves are visible on the site. Otherwise you get funny search result: if you search by person’s name the photo will be the first hit instead of biography.

Sitemap protocol

Crawlers use Sitemap protocol to help determining the content pages on your site (note: sitemap seems to be used for hinting only and it is not authoritative).  Since version 3.1 Plone can automatically generate sitemap.xml.gz. You still need to register sitemap.xml.gz in Google webmaster tools manually.

There exists a sitemap protocol extension for mobile sites.

Webmaster tools

Google Webmaster tools enable you to monitor your site visibility in Google and do some search engine specific tasks like submitting sitemaps.

I do not know what kind of similar functionality other search provides have. Please share your knowledge in the blog comments regarding this.

HTML <head> metadata

Search engines mostly ignore <meta> tags besides title so there is no point of trying fine-tune them.

Example robots.txt

Here is our optimized robots.txt for www.twinapex.com:

# Normal robots.txt body is purely substring match only
# We exclude lots of general purpose forms which are available in various mount points of the site
# and internal image bank which is hidden in the navigation tree in any case
User-agent: *
Disallow: set_language
Disallow: login_form
Disallow: sendto_form
Disallow: /images

# Googlebot allows regex in its syntax
# Block all URLs including query strings (? pattern) - contentish objects expose query string only for actions or status reports which
# might confuse search results.
# This will also block ?set_language
User-Agent: Googlebot
Disallow: /*?*
Disallow: /*folder_factories$

# Allow Adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

Useful resources

• How Google treats HTML metatags. This is especially interesting considering Plone’s support for Dublin core metadata (title, description, keywords). Note: <meta> keywords are not indexed.
• Plone SEO add-on product
• SEO with robots.txt
• Sitemaps mobile protocol

Putting views, like sitemap, into Plone content tree using Easy Template add-on

Plone has two kind of pages

• Content pages which have a path and will appear in the navigation and in the sitemap. These are stored in the database.
• View based pages and template based pages which usually present an action  (accessibility, sitemap, contact info form). They do not appear in the navigation. They are stored as source code on the file system. You cannot navigate to view based pages and just click edit. To change them you need to use various customization methods (add-on product, Zope management interface) to modify the code.

Sometimes it is desirable, for the sake of uniformness, to put view based pages (accessibility, sitemap) into the content tree. For example, one could want to have the sitemap link appearing only in the navigation tree under the site section “About this site”.

Plone add-on product Easy Template provides an easy method to show any Plone view(s) on a normal page. Easy Template uses Django like template syntax (Jinja 2 engine). It gives you great power to drop dynamic content easily on pages. Easy Template also has some security awarness ensuring the members using it cannot escape from their sandbox.

Easy Template works in WYSIWYG and non-WYSIWYG modes

• You can directly mix templates into text in Visual Editor (Kupu). This is mostly useful for non-HTML aware content editors, who use WYSIWYG editor and can use snippets from a reference card prepared by a developer. Note: Visual Editor has some limitations or undesired behavior. Sometimes it tries to put arbitary HTML tags into text (&nbsp; which breaks the template code).
• You can write templatized HTML source code in “raw” mode. You can write source code on “Template” schemata in Edit view.

Example how to show a sitemap on an arbitary Plone page

1. Install Easy Template (if you are a developer I suggest you to try trunk version)
2. Create a Templated document content
3. Write some arbitary text in Kupu
4. Put in the code snippet {{ view(“sitemap”, “createSiteMap”) }} which triggers the sitemap view rendering
5. Save and view the document in View mode

It turns out to be:

There is no such thing as a “views reference” for Plone. View names and functions can be figured out by searching and reading through ZCML and Python files in Plone source tree. Some developer insight is needed. For example. for sitemap we can do the grep search:

grep -Ri --include="*.zcml" sitemap *

Then read Products/CMFPlone/browser/configure.zcml and Products/CMFPlone/browser/sitemap.py.

The same thing works in portlets. Use Templated Portlet portlet type. See Easy Template PyPi homepage for the full reference of the product’s potential.

About the author Mikko Ohtamaa

• LinkedIn
• Twitter

Helping people and writing Plone developer API manual – killing two birds with one stone

Plone developer API documentation could be better. For this very reason, people ask questions regarding how to do a simple thing or two in Plone code. Asking questions is often the only way to fight through the monstrous codebase. Luckily these questions usually receive answers from the active community in product-developers@ list and IRC.  See also prior mailing list dicussion.

However, asking the same questions again and again, or asking the question in the first place, is undesirable way to proceed. A proper way to fix the problem would be have a proper documentation voiding the need to ask the question in the first place. Since documenting Plone API is a huge task, I propose the following to get things bootstrapped:

1. Wherever and whenever someone asks a question regarding Plone code development do not answer directly

2. Instead, write the example code snippet and commit it to https://svn.plone.org/svn/collective/collective.developermanual/trunk (collective commit access needed)

3. Give the link to the SVN trunk file as an answer This way we should slowly start building up a “developer reference” which covers the most common API use cases.

https://svn.plone.org/svn/collective/collective.developermanual/trunk is a mess. Do not care about this little detail. Just toss in your .txt files. The docteam and I will properly proof-read and structure it in the future. As soon as we code snippets keep flowing in I am happy!

Example

Here is how I helped some unlucky person struggling with DataGridFrield and created some documentation during the process.

Private email question:

I have been using your excellent DataGridField add-on on a projet, where as webmanager I add variable lists of information within an objet.  However now I would like to build a script that would insert new rows and content automatically : let’s say a User calls a specific objet ‘A’ from time to time, I would like to have a DataGridField that will record incrementally these calls (with columns like “when”, “how long”, “what result”) each time they are made, without the User having to do anything.

I tried to find an answer in the code but I’m not fluent enough in Python yet to understand everything I read.  Could you tell me how I could build that script or where shall I look for clues ?

Private email answer:

Please also read some guidelines I tried to come up for documentation.

Debugging Plone with Eclipse and PyDev remote debugger

Plone is a very heavy Python CMS containing thousands of modules. PyDev is Python extension for Eclipse IDE. Plone doesn’t run very well in PyDev debug mode:

• Debug mode is many times slower than normal Python execution
• At least on  some Linux system OS limitations prevent proper running [1]

But you can still use Eclipse and PyDev remote debugger to debug your software.

• All communications happen over sockets so you can debug software running on test server besides local machine
• Downsite is that you need to manually place pydevd.settrace() call to your code instead of just right clicking the line and choosing Set Breakpoint from menu

In both cases you need to use a custom non-forking Plone launch script from Plone and Eclipse tutorial which prevents Zope to escape itself into a child process and keeps the debug process information intact.

PyDev Extension remote debugger

Note that you need the commercial PyDev Extension license, this does not work with the basic plug-in. Free evaluation trial available. Get it here.

Benefits

• GUI stepping support
• Inspect Python run-time environment and variables using GUI
• Better integration with Eclipse console

Setting up local remote debugger

0. Install Eclipse. Do a local install, do not trust your Linux distribution package installation. http://johnpaulett.com/2009/06/26/install-eclipse-galileo-3-5-on-ubuntu-jaunty-9-04/

1. Set up idelauncher.py for your Zope application server

2. Right-click your project in Eclipse. Put pydevd in your PyDev – PYTHONPATH -> External libraries. Mine (local Eclipse 3.5 install) is /home/moo/eclipse3.5/plugins/org.pyhon.pydev.debug.XXX/pysrc

3. Switch to Eclipse Debug perspective. Start pydevd server by clicking the icon with a bug and P letter.

3. Start your Plone instance / unit tests idelauncher.py using Run (play icon – not debug). The debug server stays active even if you terminate the application launcher run.

4. When your code runs to *import pydevd ; pydevd.settrace()* (no underscore in settrace()) you should be able to start examining your site in the debugger

Now you can

• Examine and edit local variables in Variables view
• Execute Python console commands
• Examine different threads (kind of pointless with Plone…)

Note: Python console behaves funnily. You need to enter the commands to Console -> Display selected console (computer icon) -> Debug server view and output will appear  Console -> Display selected console (computer icon) -> your launcher name view.

Remote debugging not enabled

The following exception means that your pydevd server is not running. Click the P bug icon before running the launcher.

Traceback (most recent call last):
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd.py", line 624, in trace_dispatch
    return dbFrame.trace_dispatch(frame, event, arg)
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd_frame.py", line 107, in trace_dispatch
    raise
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd_frame.py", line 102, in trace_dispatch
    self.doWaitSuspend(thread, frame, event, arg)
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd_frame.py", line 25, in doWaitSuspend
    self._args[0].doWaitSuspend(*args, **kwargs)
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd.py", line 532, in doWaitSuspend
    def doWaitSuspend(self, thread, frame, event, arg): #@UnusedVariable
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd.py", line 539, in doWaitSuspend
    self.writer.addCommand(cmd)

Pydevd and Linux Resource temporarily unavailable

[1] I get the following Resource temporarily unavailable traceback when I try to run PyDev debugger against Plone – I tried to pindown the reason using strace, but no luck. Applies both for unit testing and for the actual instance launch.

Installing PloneLanguageTool ... done (0.174s)
Running Products.SitsHospital.tests.rememberbase.RememberProfileLayer tests:
  Set up Products.PloneTestCase.layer.ZCML
Traceback (most recent call last):
...
  File "/home/moo/sits/parts/zope2/lib/python/zope/testing/testrunner.py", line 688, in setup_layer
    setup_layer(base, setup_layers)
  File "/home/moo/sits/parts/zope2/lib/python/zope/testing/testrunner.py", line 692, in setup_layer
    layer.setUp()
  File "/home/moo/sits/parts/plone/PloneTestCase/layer.py", line 17, in setUp
...
  File "/usr/lib/python2.4/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.4/xml/sax/expatreader.py", line 207, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.4/xml/sax/expatreader.py", line 348, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/moo/sits/parts/zope2/lib/python/zope/configuration/xmlconfig.py", line 349, in endElementNS
    self.context.end()
  File "/home/moo/sits/parts/zope2/lib/python/zope/configuration/config.py", line 544, in end
    self.stack.pop().finish()
  File "/home/moo/sits/parts/zope2/lib/python/zope/configuration/config.py", line 692, in finish
    actions = self.handler(context, **args)
  File "/home/moo/sits/parts/plone/PlacelessTranslationService/patches.py", line 27, in compile_translations
    func(*args, **kwargs)
  File "/home/moo/sits/eggs/zope.i18n-3.7.0-py2.4.egg/zope/i18n/zcml.py", line 57, in registerTranslations
    for language in os.listdir(path):
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/home/moo/sits/parts/instance/etc/site.zcml", line 5.2-5.37
    ZopeXMLConfigurationError: File "/home/moo/sits/parts/zope2/lib/python/Products/Five/configure.zcml", line 6.2-6.30
    ZopeXMLConfigurationError: File "/home/moo/sits/parts/zope2/lib/python/Products/Five/i18n.zcml", line 24.4-24.52
    OSError: [Errno 11] Resource temporarily unavailable: '/home/moo/sits/parts/zope2/lib/python/zope/app/locales'

Plone goes mobile

FYI

Plone GoMobile project.

plonecommunity.mobi demo site.

More to come.