Preface

Please do not take this post too seriously if I am ranting tongue-in-cheek. This thinking does not reflect the official opinion of Sauna Sprint team any way, it is just my personal rambling.

Background

Plone is having popularity issues. This can be deducted by:

• PHP CMS are more popular (Google trends and so on)
• Plone lacks love
• By discussing with Python programmers outside Plone community: they see Plone as monster

Plone is a superior CMS product (feature wise, polish wise, tech wise, and so on). So how did this happen?

My theory, you are free to disagree, is that the community has been driving “wrong” ideologies

• Plone’s “buyers” are not users, but developers
• There has been no driving community factor to make Plone accessible for more developers. Plone community has been working to make Plone work for the existing community members, not new members. The consulting companies have been too busy to extract money from their existing big clients. It might even be that it’s the best interest of these companies to keep the barrier of entry high, as this way they protect the Plone business for themselves. Note that this strategy will fail in the point there is no longer Plone business.

The big boys are telling that customers are abandoning Plone, because there are no developers. I know a lot of Plone developers, looking for more Plone work.  I’ll translate this that there are not enough affordable Plone developers entering the market.

I am also not buying arguments like

• “don’t compare apples to oranges (Django vs. Plone)”
• “its history and evolved that way”
• “big tool must be complex”

The result

Plone is easy to use, but hard to develop. If you need to use code generators and boiler-plate in Python code, you are doing it wrong.

Plone is a soup of different, non-mainstream, technologies. To be a Plonista you need master: Python (ok docs), Plone user mode (good docs), Zope component architecture (some docs), ZCML (no useful docs), GenericSetup (no docs), TAL (some docs), Archetypes (basically no docs), buildout (aaaaaarghs) … You have too much to learn from material which does not exist. Today the material effectively does not exist if it’s not your first Google hit – I am against book.

New developers are not getting in. It is easier to duplicate basic Plone functionality in a hacky in-house Django CMS than customize Plone for your use case. Plone does not give any more added value to the new developers. There is no business motivation to adapt Plone as a technology platform in new ventures.

The quest

Let’s fix it, as I see the developer acceptance the most crucial challenge Plone community is facing if the community wants to survive.

What we can do here is

• Make Plone easier for developers
• Make more, better, learning material available
• Eliminate and protect against factors in Plone which hinder the goals of two above bullets (do not include new technologies or modules to core if they make developers sad)

To accomplish this we can

1. Promote “easy” technologies inside and outside Plone community (Dexterity over Archetypes, Grok over ZCML)
2. Making a standard workflow available for them (Django style tutorial that do 1, 2, 3 and you are a certified Plone developer)
3. Kill technologies which make the stack unnecessary complex and are not necessarily needed in newbie use cases (buildout, I am looking at you)
4. Replace old, badly documented and difficult solutions with easier ones (could we actually replace TAL with something more popular?)

Operational plan

Here are some concrete tasks what we could tackle in Sauna Sprint.

Make it easy department

• Plone auto-restart team: provide working and fast auto-restart á la Django or Tornado (hint: plone.reload doesn’t cut it – you need real restarts process-wise)
• Default custom product team: ship Plone with a file-system based custom product skeleton, so that when you need to include your first custom Python module you don’t need to learn about buildout and paster first . You actually can copy-paste in Python code and it just works.
• Through-the-web is happy again team: Make it possible to code and execute Python through-the-web so that it actually works.: a working replacement for RestrictedPython

Fix user-wise broken things department

• Buildout team: make buildout more user friendly: spit out error messages which are actually useful, have newbie buildout tutorial from the developer perspective. Nothing bad should happen when buildout is run. Make buildout to confirm version updates, etc.
• Anti-buildout team: how to install eggs without buildout. Why did we need the buildout in the first place?

This is how you do it department

• Tutorial team: Make “gapless” hand-holding tutorial for creating your first view, form and content-type. Make it so that you do not need to look outside the tutorial or learn anything unnecessary (don’t make me think)
• Picture team: draw big pictures of Plone architecture, how different parts are connected, so that new developers get hang on things easily
• Document generator team: Integrate collective.developermanual with Sphinx tool that generate the reference  documentation from the source code: views, viewlets, templates, CSS classes, portlets, etc.

Hasta la vista babe department

• Kill it with fire team: Hunt down all bad learning material and make it redirect to new learning material (no ExternalMethods, no Install.py, no plone_skins, etc.)
• Nuke it from orbit team: I am not sure what this team could do, but I like the feeling of the name

We hope that novice  developers attend the sprint. This way we can use you as guinea pigs and document the pain points on a road to become Plone developer. Don’t worry – we’ll inject you with proper medicine to numb the pain and forget bad memories.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Please update your buildout pindowns for Dexterity 1.0:

extends =

http://dist.plone.org/release/4.1rc2/versions.cfg

http://good-py.appspot.com/release/dexterity/1.0?plone=4.1rc2

Please report all the issues in the issue tracker.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

If you get stuck in a point please ask questions on the gomobile-dev discussion forum.

Prerequisites

• Basic Plone development skills
• Basic ZopeSkel knowledge
• Basic five.grok knowledge (Grok framework integration for Plone)

This tutorial is little rough on details, but links to authoritative sources for more in-depth information. I hope to further polish this tutorial based on the user comments.

Note: currently the installation process does not work from China, as the local access to code.google.com and appspot.com servers is limited.

Installation

Install Web and Mobile to your buildout.

Possibly fight with lxml to get it running.

Create a Plone site (if you don’t already have an existing site).

Then install following add-ons, in this order in the site control panel

• Go Mobile
• Go Mobile Default Theme

Now your website should have a basic mobile experience with a default Web and Mobile theme.

Test mobile site

You can visit the mobile version of your site by putting a m.localhost domain in your /etc/hosts file. You don’t need a mobile browser, as the mobile site is detected by a domain name by default. Your mobile site should be available from http://m.localhost:8080/Plone

If you access the website with a mobile browser you will be automatically redirected to a mobile site.

Creating a mobile theme

Install the ZopeSkel based mobile theme template to your buildout.cfg.

parts =
   ...
   paster

[paster]
recipe = zc.recipe.egg
eggs =
  PasteScript
  gomobile.templates
  ${instance:eggs}

Rerun buildout to enable bin/paster command:

bin/buildout

Create a theme skeleton using paster. We use a Python package named gomobiletheme.yourcompanyname here as an example name for the theme package. If it asks for “override files” answer yes:

cd src
 ../bin/paster create -t gomobile_theme gomobiletheme.yourcompanyname

List your new theme egg in buildout in develop-eggs. (Alternatively: use mr.developer  if you indent to put it directly under version control.)

eggs =
    gomobiletheme.yourcompany
develop =
   src/gomobiletheme.yourcompan

Rerun buildout.

Test theme skeleton

Test the theme installs and works. Restart Plone and install your theme add-on in the add-on installer.

Navigate to m.localhost again and see that the mobile site still opens.

Theme theory

The generated mobile theme includes five.grok based viewlets.py and view.py files where you are to place

• Mobile theme viewlets (note: mobile theme does not use viewlet managers)
• Mobile specific view overrides
• Mobile specific page template overrides

All visual code registered against gomobiletheme.yourcompany.interfaces.IThemeLayer is available and used when your site is browser in mobile mode.

Changing logo

Change the generated gomobile.yourcompany.viewlets.Logo class.  In the example below we set it to use logo image file from our web theme.

class Logo(base.Logo):

 # Change logo URI here
 def getLogoPath(self):
 return "++resource++plonetheme.yourwebtheme.images/logo.png"

If you are using logo from the web theme resource directory, make sure <resourceDirectory> in theme browser/configure.zcml does not have <resourceDirectory layer=”"> set. If it’s set the resource directory is not activate for the mobile theme and the mobile theme cannot access the logo file.

The logo image is resized automatically by gomobiletheme.basic.viewlets.Logo base class for different mobile screen resolutions.

Changing footer text

In viewlets.py add:

class FooterText(base.FooterText):
 """ Free-form HTML text at the end of the page """

 # the presence of this viewlet includes a new page template
 # and overrides default footertext

Then include the page template itself. Add new file to the your theme package templates/footertext.pt:

<div id="footer-text" i18n:domain="isleofback.app">
 Your text goes here.
</div>

Change the default colours and CSS

Mobile theme static CSS and Javascript resources are provided by five.grok static folder mechanism. Transformations document provides some in-sight for different CSS files used by different handsets: common.css is served to all mobile phones, highend.css for Webkit and Firefox based browsers, lowend.css for limited browsers.

The default gomobiletheme.basic CSS resources are included by default with the theme skeleton. You need override only the CSS rules you specifically want to change for your own  theme. This can be done by adding rules to common.css, lowend.css and highend.css files which you can find from the generated theme package.

Further info

• Read theming manual in Web and Mobile documentation section.
• See Web and Mobile presentation in Plone Conference 2010
• See plonecommunity.app package for an open source example
• SVN trunk version of the theming manual for the latest instructions
• Subscribe to Web and Mobile discussion group for further questions

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

To have the über-experience of customer engagement with mobile push notifications you need

• A mobile application (iOS, Android 2.2)
• RSS feed to notifications middleware server (our solution)
• RSS feeds themselves
• Windows/UNIX server running the middleware

How it is put together

Tornado web server is used to handle incoming HTTP requests in scalable manner.

feedparser library fetches RSS feeds and processes them to client notifications.

BitReader (post, source) library is used to create messages to interact with Apple push notification service (APNs). The protocol is bit based protocol running directly over TCP/IP. Apple service has been designed to handle high volumes of traffic – it does not use anything like stateless HTTP to waste bandwidth.

Django models are used to store the state of each individual subscriber. Django’s ORM abstraction allows us to use the same middleware for small distributions (< 1000 clients, SQLite database) or big ones (millions of clients, MySQL database). The stored state information includes the subscriber id and the current badge number – the red circle on the app icon showing the count unread posts. When the application is launched, it can decrease its badge number by doing a HTTP call to the server.

Django settings are used to put together required certificates and whether the application is run in sandbox mode.

Walkthrough

There is a core IO loop, running in a separate process, called stream observer. This loops updates fetches RSS feeds’ status and passes updates to Tornado server over HTTP.  With this arrangement, any HTTP capable client can send push notifications.

Tornado handles incoming updates, updates the related subscribe status – how many unread notifications, etc. through exposed Django views. The notification is formatted according to the variables available on the subscriber mobile platform. In Apple’s case, the notification message gets title, badge, sound and a launch image. Payload is checked against hard 256 byte limit.

Then the payload is pushed to Apple servers over TCP/IP protocol. SSL certificaties needed.

A subscriber is registered  when the mobile application is launched. The application asks a subscriber id from Apple servers. Then, this subscriber id is delivered to our middleware over normal HTTP call.

The middleware also handles feedback service which gives you list of devices which have unsubscribed from your service. This way you can cut off notifications from unsubscribed clients. This is also done using BitReader and TCP/IP.

Future

The architecture is built so that different push backends can be included in the service. Android support is on the roadmap and we probably will have Blackberry and Meego support (when/if Nokia announces such a service).

We have currently tested this solution with RSS streams from WordPress and Plone.

We may release source code when it’s ready.

More info

• http://blog.boxedice.com/2009/07/10/how-to-build-an-apple-push-notification-provider-server-tutorial/

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

• Plone 4 is up to 5x times faster than  Joomla!, Drupal or WordPress systems
• Plone is secure – it has no known site breaches – it even runs on fbi.gov and cia.gov
• Plone has the best user experience and easiest to use of open source CMSes – Alexander Limi,  the usability experts, also works on Firefox on Mozilla corporation

Vote Plone 4 release news on reddit.

Yeah, yeah… it is a linkadvertisement

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

I had this curious piece on conversation on #joomla channel on freenode. Though it is an individual case, I hope it will bring some light to the fact how people perceive Plone outside Plone community and what Plone should to do fix it.

I think it would be beneficial for Plone to finally close mailman for the site administration / user support and move to real web forums / Google Groups / whatever which would be usable.

Also, there is an example how unprofessionalism is not good for the community.

[20:23:29] x: Biggest problem so far is finding competent ("I will deliver on this schedule) joomla consulting experts. Second biggest problem is security, our site has been hacked 3 times in the past 6 months
[20:25:18] me: have you considered any alternative CMS with better security track record?
[20:25:50] x: moo: we moved from Plone to Joomla. 3 years on Plone with no hacks.
[20:26:04] x: Problem with plone is no forums with email support
[20:26:21] me: you pay for support
[20:26:30] z: did you do basic joomla security guidelines?
[20:26:39] me: also check http://plone.org/support
[20:26:40]  Title: Support options for Plone — Plone CMS: Open Source Content Management (at plone.org)
[20:26:51] x: moo: I'm fine with paying for support. We're paying SiteGround $200-$300/month on average when you add the support costs.
[20:26:58] z: ie using a key to access admin, changing default sql prefix
[20:27:01] AngryPerson: who cares about plone
[20:27:10] AngryPerson: its an ancient cms thats clearly past its time
[20:27:20] AngryPerson: its only privately supported with little community support
[20:28:34] AngryPerson: Moo^_^: why are you even in here?
[20:28:40] AngryPerson: you just want to piss on joomla?
[20:28:43] me: we do business on drupal, joomla and plone
[20:28:48] me: different tool for different job
[20:28:51] -*- y shrugs
[20:29:01] AngryPerson: just seems to me like you want to push ppl away from joomla
[20:29:11] x: z: After 6 months I'm still a Joomla noob. I need a consulting services company that will do the security patching, maintenance, service on the site, and host it.
[20:29:25] me: not true
[20:29:36] AngryPerson: Moo^_^: well regardless of what you say, tahts how it seems to me
[20:29:54] z: actually I've had good luck just following a few blog posts
[20:30:10] me: I don't defend myself, as I don't want to engage such a conversation with you
[20:30:22] AngryPerson: thats good, why dont you fuck off too
[20:30:22] <-* jools has kicked y from #joomla (Please watch your language) [20:30:22] --> y (dgdf@unaffiliated/anti-mttr/x-9384728) has joined #joomla
[20:30:27] z: nothing is impervious, but you drastically reduce your attractiveness to hackers by a few simple steps
[20:30:32] AngryPerson: stop giving ppl your shitty advice

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

The idea is that we take three first sentences and use them as a description.

Since Zope uses RestrictedPython for through-the-web created scripts, the user of this script cannot breach the server security (they cannot make Python calls they have no permission for). This sets some limitations for automating tasks like this, but we don’t hit those limitations in our use case.

def create_automatic_description(content, text_field_name="text"):
    """ Creates an automatic description from HTML body by taking three first sentences. 

    Takes the body text

    @param content: Any Plone contentish item (they all have description)

    @param text_field_name: Which schema field is used to supply the body text (may very depending on the content type)
    """

    # Body is Archetype "text" field in schema by default.
    # Accessor can take the desired format as a mimetype parameter.
    # The line below should trigger conversion from text/html -> text/plain automatically using portal_transforms
    field = content.Schema()[text_field_name]

    # Returns a Python method which you can call to get field's
    # for a certain content type. This is also security aware
    # and does not breach field-level security provded by Archetypes
    accessor = field.getAccessor(content)

    # body is UTF-8
    body = accessor(mimetype="text/plain")

    # Now let's take three first sentences or the whole content of body
    sentences = body.split(".")

    if len(sentences) > 3:
       intro = ".".join(sentences[0:3])
       intro += "." # Don't forget closing the last sentence
    else:
       # Body text is shorter than 3 sentences
       intro = body

    content.setDescription(intro)

# context is the reference of the folder where this script is run
for id, item in context.contentItems():
     # Iterate through all content items (this ignores Zope objects like this script itself)

     # Use RestrictedPython safe logging.
     # plone_log() method is permission aware and available on any contentish object
     # so we can safely use it from through-the-web scripts
     context.plone_log("Fixing:" + id)

     # Check that the description has never been saved (None)
     # or it is empty, so we do not override a description someone has
     # set before automatically or manually
     desc = context.Description() # All Archetypes accessor method, returns UTF-8 encoded string

     if desc is None or desc.strip() == "":
          # We use the HTML of field called "text" to generate the description
          create_automatic_description(item, "text")

# This will be printed in the browser when the script completes succesfully
return "OK"

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Do not index query strings

It is often desirable to make sure that query string pages (http://yoursite/page?query_string_action=something) do not end up into the search indexes. Otherwise search bots might index pages like site’s own search engine results  (yoursite/search?SearchableText=…) lowering the visibility of  actual content pages.

GoogleBot has regex support in robots.txt and can be configured to ignore any URL ? in it. See the example below.

Query string indexing causes the crawler crawl things like

• Various search results (?SearchableText)
• Keyword lists (?Subject)
• Language switching code (?set_language)… making set_language appear as the document in the search results

Also, “almost” human readable query strings look ugly in the address bar…

Top level domains and languages

Using top level domain name (.fi for Finland, .uk for United Kingdoms, and so on.) to make distinction between different languages and areas is optimal solution from the SEO point of view. Search engines use TLD information to reorder the search results based on where  the search query is performed  (there is difference between google.com and google.fi results).

Plone doesn’t use any query strings for content pages. Making robots to ignore query strings is especially important if you are hosting multilingual site and you use top level domain name (TLD) to separate languages: if you don’t configure robots.txt to ignore ?set_language links only one of your top level domains (.com, .fi, .xxx) will get proper visibility in the search results. For example we had situation where our domain www.twinapex.fi did not get proper visibility because Google considered www.twinapex.com?set_language=fi as the primary content source (accessing Finnish content through English site and  language switching links).

Shared forms

Plone has some forms (send to, login) which can appear on any content page. These must be disallowed or otherwise you might have a search result where the link goes to the form page instead of the actual content page.

Hidden content and content excluded from the navigation

Any content excluded from the sitemap navigation  should be put under disallowed in robots.txt. E.g. if you check “exclude from navigation” for Plone folder remember to update robots.txt also.

In our case, our internal image bank must not end up being indexed, though images themselves are visible on the site. Otherwise you get funny search result: if you search by person’s name the photo will be the first hit instead of biography.

Sitemap protocol

Crawlers use Sitemap protocol to help determining the content pages on your site (note: sitemap seems to be used for hinting only and it is not authoritative).  Since version 3.1 Plone can automatically generate sitemap.xml.gz. You still need to register sitemap.xml.gz in Google webmaster tools manually.

There exists a sitemap protocol extension for mobile sites.

Webmaster tools

Google Webmaster tools enable you to monitor your site visibility in Google and do some search engine specific tasks like submitting sitemaps.

I do not know what kind of similar functionality other search provides have. Please share your knowledge in the blog comments regarding this.

HTML <head> metadata

Search engines mostly ignore <meta> tags besides title so there is no point of trying fine-tune them.

Example robots.txt

Here is our optimized robots.txt for www.twinapex.com:

# Normal robots.txt body is purely substring match only
# We exclude lots of general purpose forms which are available in various mount points of the site
# and internal image bank which is hidden in the navigation tree in any case
User-agent: *
Disallow: set_language
Disallow: login_form
Disallow: sendto_form
Disallow: /images

# Googlebot allows regex in its syntax
# Block all URLs including query strings (? pattern) - contentish objects expose query string only for actions or status reports which
# might confuse search results.
# This will also block ?set_language
User-Agent: Googlebot
Disallow: /*?*
Disallow: /*folder_factories$

# Allow Adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

Useful resources

• How Google treats HTML metatags. This is especially interesting considering Plone’s support for Dublin core metadata (title, description, keywords). Note: <meta> keywords are not indexed.
• Plone SEO add-on product
• SEO with robots.txt
• Sitemaps mobile protocol

This is especially troublesome if you display external content (RSS feeds, ATOM feeds) on your mobile site. Incoming HTML cannot be guaranteed to follow any specification.

To solve this problem, we have created gomobile.xhtmlmp Python library which helps you to transform any HTML to content to valid XHTML MP. The library is piloted on plonecommunity.mobi site which  uses aggregated content from varying sources. The library is based on lxml.html.Cleaner. The library is part of GoMobile project which aims to create world class Python mobile web development tools.

Highlights

• Turn any incoming HTML/XHTML to mobile profile compatible
• Enforce ALT text on images – especially useful for external tracking images (feedburner tracker). ALT texts are required by XHTML MP.
• Protect against Cross-Site Scripting Attacks (XSS) and other nastiness, as provided by lxml.xhtml.clean
• Unicode compliant – eats funky characters

As an example we integrated gomobile.xhtmlmp  to Feedfeeder Plone add-on product.

Enjoy.