| .gitignore for Python developersPosted on July 23, 2010 by Mikko OhtamaaFiled Under git, plone, python, technology, web development, zope If you are using Git for version control for your Python egg and buildout development below are is a sample which you might want to put into your .gitignore file. *.mo *.egg-info *.egg *.EGG *.EGG-INFO bin build develop-eggs downloads eggs fake-eggs parts dist .installed.cfg .mr.developer.cfg .hg .bzr .svn *.pyc *.pyo *.tmp* Suggestions for new ignores are welcome.
Plone Go Mobile – preparing a releasePosted on April 1, 2010 by Mikko OhtamaaFiled Under blackberry, iphone, maemo, n900, plone, technology, web development, zope Enter the mobile world with Plone! Go mobile for Plone add-on is now “feature complete” and we are preparing a release. For Easter holidays we bring you a trunk release instructions. Note: you might want to change checkout protocol from “https” to “http” in the buildout cfg. The most able and most inpatient might want to test this now when it is hot. There exists three production site where you can see the product in action
See the feature set which will beat crap out of other mobile CMSs. What’s missing for the real release is
It has not been tested on Plone 4. Thanks Karl Horak for bringing up the issue. Eclipse plug-in for Plone/Zope/buildout based developmentPosted on March 29, 2010 by Mikko OhtamaaFiled Under buildout, eclipse, plone, pydev, python, zope Few days ago Fabio announced Django plug-in for Eclipse. There also exists an effort to make Eclipse integrate smoother with Plone / Zope / Buildout world. Please see PyPi page here. It is not nearly as polished as Fabio’s work, as Fabio being main author of PyDev and started as a collection of company internal tools. However if you mare making many buildouts, write code > 5 Python modules at once and don’t want to start terminal everytime you need to run buildout, this plug-in is for you. Also, it is not yet a real plug-in. It is a collection of PythonMonkey scripts. This means that to deploy the scripts you simply copy .py files them to scripts folder of any of your Eclipse projects. If you need to work on the codebase, you don’t need to start a separate Eclipse instance, but you can do it interactively through Eclipse console while you do other development. The script source code is well commented, so if you want to tune them for your own habits it should be easy. Integrating and theming Wordpress with your CMS site using XDVPosted on March 28, 2010 by Mikko OhtamaaFiled Under Wordpress, apache, cms, css, plone, python, technology, web development, xdv, zope IntroductionXDV is an external HTML theming engine, a.k.a. theming proxy, which allows you to mix and match HTML and CSS from internal and external sites by using simple XML rules. It separates the theme development from the site development, so that people with little HTML and CSS knowledge can create themes without need to know underlying Python, PHP or whatever. It also enables integration of different services and sites to one, unified, user experience. For example, XDV is used by plone.org <http://plone.org> to integrate Plone CMS and Trac issue tracker. XDV compiles theming rules to XSL templates, which has been a standard XML based templates language since 1999. XSL has good support in every programming language and web server out there. Example backends to perform XSL transformation include
XDV theming can be used together with Plone where enhanced support is provided by collective.xdv package package. Technically, collective.xdv adds Plone settings panel and does XSL transformation in Zope’s post-publication hook using lxml library. XDV can be used standalone with XDV package to theme any web site, let it be Wordpress, Joomla, Drupal or custom in-house PHP solution from year 2000. XDV is based on Deliverance specification The difference between XDV and Deliverance reference implementation is that XDV internally compiles themes to XSL templates, when Deliverance relies on processing HTML in Python. Currently XDV approach seems to be working better, as we had many problems trying to apply Deliverance for Wordpress site (redirects didn’t work, HTTP posts didn’t work, etc.). Setting up XDV development toolsXDV tools are deployed as Python eggs. You can use tools like buildout <http://www.buildout.org/> configuration and assembly tool or easy_install to get XDV on your development computer and the server. If you are working with Plone you can integrate XDV to your site existing buildout. If you are not working with Plone, XDV home page has instructions how to deploy XDV command standalone. XDV RulesRules (rules.xml) will tell how to fit content from external source to your theme HTML. It provides straightforward XML based syntax to manipulate HTML easily
Rules XML syntax is documented at XDV homepage. Rules will be compiled to XSL template (theme.xsl) by xdvcompiler command. The actual theming is done by one of the XSL backends listed above, by taking HTML as input and applying XSL transformations on it. Note that currently rules without matching selectors are silently ignored and there is no bullet-proof way to debug what happens inside XSL transformation, except by looking into compiled theme.xsl. Using XDV to theme and integrate a Wordpress siteBelow are instructions how to integrate a Wordpress site to your CMS. In this example CMS is Plone, but it could be any other system. We will create XDV theme which will theme Wordpress site to match our CMS site in the fly.
Wordpress theme using built with XDV and using a live Plone web page as a theme template. This way Wordpress theme inherits “live data” from Plone site, like top tabs (portal sections), footer, CSS and other stuff which can be changed in-the-fly and reflecting changes to two separaet theming products would be cumbersome. Benefits using Wordpress for blogging instead of main CMS
Benefits of using XDV theming instead of creating native Wordpress theme are
Theme elementsThe theme will consist of following pieces
CMS page templateThis explains how to create a Plone page template where Wordpress content will be dropped in. This step is not necessary, as we could do this without touching the Plone. However, it makes things more straightforward and explicit when we known that Wordpress theme uses a certain template and we explicitly define slots for Wordpress content there. Example: <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
xmlns:tal="http://xml.zope.org/namespaces/tal"
xmlns:metal="http://xml.zope.org/namespaces/metal"
xmlns:i18n="http://xml.zope.org/namespaces/i18n"
lang="en"
metal:use-macro="here/main_template/macros/master"
i18n:domain="plone">
<body>
<div metal:fill-slot="content">
<div id="wordpress-content">
<!-- Your Wordpress "left column" will go there -->
</div>
</div>
</body>
</html>
Theming rulesFollowing are XDV rules (rules.xml) how we will fit Wordpress site to Plone frame. It will integrate
rules.xml: <?xml version="1.0" encoding="UTF-8"?>
<rules xmlns="http://namespaces.plone.org/xdv"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:css="http://namespaces.plone.org/xdv+css">
<!-- Remove Wordpress CSS by filtering out <style> tags-->
<drop css:content="style" />
<!-- Make sure that Wordpress metadata is present in <head> section -->
<append css:content="head link" css:theme="head" />
<!-- note: replace does not seem to handle multiple meta tags very well -->
<drop css:theme="meta" />
<append css:content="head meta" css:theme="head" />
<!-- Use blog title instead of Plone page title -->
<replace css:content="title" css:theme="title" />
<!-- Put Wordpress sidebar to Plone's portlets section -->
<append css:content="#r_sidebar" css:theme="#portal-column-one .visualPadding" />
<!-- Place wordpress content into our theme content area -->
<copy css:content="#contentleft" css:theme="#wordpress-content" />
<!-- This mixes in Wordpress specific CSS sheet which is applied for pages
served from Wordpress only and does not concern Plone CMS.
This stylesheet will theme Wordpress specific tags,
like blog posts and comment fields.
We keep this file in Plone, but this could be served from elsewhere. -->
<append css:theme="head">
<style type="text/css">
@import url(http://mfabrik.com/++resource++plonetheme.mfabrik/wordpress.css);
</style>
</append>
<!-- This stylesheet is used by special spam protection plug-in NoSpamNX -->
<append css:theme="head">
<link rel="stylesheet" href="http://blog.mfabrik.com/wp-content/plugins/nospamnx/nospamnx.css" type="text/css" />
</append>
<!-- Remove Google Analytics script used for CMS site -->
<drop css:theme="#page-bottom script" />
<!-- Rebuild our Google Analytics code, using a different tracker id this time
which is a specific to our blog.
-->
<append css:theme="#page-bottom">
<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>
<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-8819100-2");
pageTracker._trackPageview();
} catch(err) {
}
</script>
</append>
</rules>
Wordpress specific CSSThis CSS has styles which are applied only to Wordpress pages. They are mainly corner case fixes where Wordpress and CMS styles must match. The CSS file is loaded when rules.xml injects it to <head> section. wordpress.css: /* Font and block style fixes */
#wordpress-content h1 {
border: 0;
}
#wordpress-content .post-end {
margin-bottom: 60px;
}
#wordpress-content pre {
width: 600px;
overflow: auto;
background: white;
border: 1px solid #888;
}
#wordpress-content ul {
margin-left: 20px;
}
#wordpress-content .post-info-date,
#wordpress-content .post-info-categories,
#wordpress-content .post-info-tags {
font-size: 80%;
color: #888;
}
/* Make sure that posts and comments look sane in our theme */
#wordpress-content .post {
margin-top: 15px;
}
#wordpress-content .commentlist li {
margin: 20px;
background: white;
padding: 10px;
}
#wordpress-content .commentlist li img {
float: left;
margin-right: 20px;
margin-bottom: 20px;
}
#wordpress-content #commentform {
margin: 20px;
}
#wordpress-content {
margin-left: 20px;
margin-right: 20px;
}
/* Make Wordpress "sidebaar" look like Plone "portlets */
.template-wordpress_listing #portal-column-one ul {
list-style: none;
margin-bottom: 40px;
}
.template-wordpress_listing #portal-column-one ul#Recent li {
margin-bottom: 8px;
}
.template-wordpress_listing #portal-column-one ul#Categories a {
line-height: 120%;
}
.template-wordpress_listing #portal-column-one h2 {
background: transparent;
border: 0;
font-weight:normal;
line-height:1.6em;
padding:0;
text-transform:none;
font-size: 16px;
color: #9b9b9b;
border-bottom:4px solid #CDCDCD;
}
Helper scriptThe following Python script (xdv.py) makes it easy for us
xdv.py: """
This command line Python script compiles your rules.xml to XDV XSL
Modify it for your own needs.
It assumes your buildout.cfg has xdv section and generated XDV
commands under bin/
To compile, execute in the buildout folder::
python src/plonetheme.mfabrik/xdv.py
To build test HTML::
python src/plonetheme.mfabrik/xdv.py --test
To build test HTML and preview it in browser, execute in buildout folder::
python src/plonetheme.mfabrik/xdv.py --preview
"""
import getopt, sys
import os
import webbrowser
# rules XML for theming
RULES_XML = "src/plonetheme.mfabrik/deliverance/etc/rules.xml"
# Which XSL file to generate for compiled XDV
OUTPUT_FILE = "theme.xsl"
# Which file to generate applied theme test runs
TEST_HTML_FILE = "test.html"
# Our "theme.html" is a remote template served for each request.
# Because we are doing live integrattion, this is a HTTP resource,
# not a local file.
THEME="http://mfabrik.com/news/wordpress_listing/"
#
# External site you are theming.
# Note: must have ending slash (lxm cannot handle redirects)
#
SITE="http://blog.twinapex.fi/"
try:
opts, args = getopt.getopt(sys.argv[1:], "pt", ["preview", "test"])
except getopt.GetoptError, err:
# print help information and exit:
print str(err) # will print something like "option -a not recognized"
# Convert options to simple list
opts = [ opt for opt, value in opts ]
print "Compiling transformation"
value = os.system("bin/xdvcompiler -o " + OUTPUT_FILE + " " + RULES_XML +" " + THEME)
if value != 0:
print "Compilation failed"
sys.exit(1)
if "-p" in opts or "--preview" in opts or "-t" in opts or "--test" in opts:
print "Generating test HTML page"
value = os.system("bin/xdvrun -o " + TEST_HTML_FILE + " " + OUTPUT_FILE + " " + SITE)
if value != 0:
print "Page transformation failed"
sys.exit(1)
if "-p" in opts or "--preview" in opts:
# Preview the result in a browser
# NOTE: OSX needs Python >= 2.5 to make this work
# Make sure test run succeeded
url = "file://" + os.path.abspath(TEST_HTML_FILE)
print "Opening:" + url
# We prefer Firefox for preview for its superious
# Firebug HTML debugger and XPath rule generator
try:
browser = webbrowser.get("firefox")
except webbrowser.Error:
# No FF on the system, or OSX which can't find its browsers
browser = webbrowser.get()
browser.open_new_tab(url)
Compiling the themeThis will generate XSL templates to do theming transform. It will compile rules XML with some boilerplate XSL. Running our compile script: python src/plonetheme.mfabrik/xdv.py Since Plone usually does not use any relative paths or relative resources in HTML, we do not give the parameter “Absolute prefix” to the compilation stage. In Plone, everything is mapped through a virtual hosting aware resource locator: portal_url and VirtualHostMonster. For more information see Testing the themeThe following command will apply theme for an example external page: bin/xdvrun -o theme.html theme.xsl http://blog.twinapex.fi firefox theme.xhtml … or we can use shortcut provided by our script … python src/plonetheme.mfabrik/xdv.py --preview Applying the theme in Apache production environmentThese steps tell how to apply the integration theme for Wordpress when Wordpress is running under Apache virtualhost. Installing dependenciesWe use Apache and mod_transform. Instructions how to set up modules for Apache are available on XDV homepage. Some hand-build modules must be used, but instructions to set them up for Ubuntu / Debian are available. Apache 2 supports filter chains which allow you to perform magic on HTTP response before sending it out. This corresponds Python’s WSGI middleware. We’ll use special built of mod_transform and mod_depends which are known to working. These modules were forked from their orignal creations to make them XDV compatible, as the orignal has not been updated since 2004 (here you can nicely see how open source guarantees “won’t run out of support” freedom). Example: sudo -i apt-get install libxslt1-dev libapache2-mod-apreq2 libapreq2-dev apache2-threaded-dev wget http://html-xslt.googlecode.com/files/mod-transform-html-xslt.tgz wget http://html-xslt.googlecode.com/files/mod-depends-html-xslt.tgz tar -xzf mod-transform-html-xslt.tgz tar -xzf mod-depends-html-xslt.tgz cd mod-depends-html-xslt ; ./configure ; make ; make install ; cd .. cd mod-transform-html-xslt ; ./configure ; make ; make install ; cd .. Enable built-in Apache modules: a2enmod filter a2enmod ext_filter For modules depends and transform you need to manually add them to the end of Apache configuration, as they do not provide a2enmod stubs for Debian. Edit /etc/apache2/apache.conf: LoadModule depends_module /usr/lib/apache2/modules/mod_depends.so LoadModule transform_module /usr/lib/apache2/modules/mod_transform.so You need to hard reset Apache to make the new modules effective: /etc/init.d/apache2 force-reload Virtual host configurationBelow is our virtualhost configuration which runs Wordpress and PHP. Transformation filter chain has been added in. /etc/apache/sites-enabled/blog.mfabrik.com: <VirtualHost *>
ServerName blog.mfabrik.com
ServerAdmin info@mfabrik.com
LogFormat combined
TransferLog /var/log/apache2/blog.mfabrik.com.log
# Basic Wordpress setup
Options +Indexes FollowSymLinks +ExecCGI
DocumentRoot /srv/www/wordpress
<Directory /srv/www/wordpress>
Options FollowSymlinks
AllowOverride All
</Directory>
AddType application/x-httpd-php .php .php3 .php4 .php5
AddType application/x-httpd-php-source .phps
# Theming set-up
# This chain is used for public web pages
FilterDeclare THEME
FilterProvider THEME XSLT resp=Content-Type $text/html
TransformOptions +ApacheFS +HTML
# This is the location of compiled XSL theme transform
TransformSet /theme.xsl
# This will make Apache not to reload transformation every time
# it is performed. Instead, a compiled version is hold in the
# virtual URL declared above.
TransformCache /theme.xsl /srv/plone/twinapex.fi/theme.xsl
# We want to apply theme only for
# 1. public pages (otherwise Wordpress administrative interface stops working)
<Location "/">
FilterChain THEME
</Location>
# 2. Admin interface and feeds should not receive any kind of theming
<LocationMatch "(wp-login|wp-admin|wp-includes)">
# The following resets the filter chain
# http://httpd.apache.org/docs/2.2/mod/mod_filter.html#filterchain
FilterChain !
</LocationMatch>
</VirtualHost>
Running itAfter Apache has all modules enabled and your virtualhost configuration is ok, you should see Wordpress through your new theme by visiting at the site served through Apache: Automatically reflecting CMS changes back to XDV themeThe theme should be recompiled every time
This is because the compilation will hard-link resources and template snippets to resulting the theme.xsl file. If hard-linked resources change on the Plone site, the transformation XSL file does not automatically reflect back the changes. It could be possible to use Plone events automatically to rerun theme compilation when concerned resources change. However, the would be quite complex. For now, we are satisfied with a scheduled task which will recompile the theme now and then. Alternatively, mod_transforms could be run in non-cached mode with some performance implications. Here is a shell script, update-wordpress-theme.sh, which will perform the recompilation and make Apache’s transformation cache aware of changes: #!/bin/sh # # Periodically update Wordpress theme to reflect changes on CMS site # # Recompile theme sudo -H -u twinapex /bin/sh -c cd /srv/plone/twinapex.fi ; python src/plonetheme.mfabrik/xdv.py # Make Apache aware of theme changes sudo apache2ctl graceful Then we call it periodically in cron job, every 15 minutes in /etc/cron.d/update-wordpress: # Make Wordpress XDV theme to reflect changes on CMS 0,15,30,45 * * * * /srv/plone/twinapex.fi/update-wordpress-theme.sh Updating Wordpress settingsNo changes on Wordpress needed if the domain name is not changed in the theme transformation process. Site URLUnlike Plone, Wordpress does not have decent virtual hosting machinery. It knowns only one URL which is uses to refer to the site in the external context (e.g. RSS feeds). This setting can be overridden in
Here is an example how we override this in our wp-config.php: // http://codex.wordpress.org/Editing_wp-config.php#WordPress_address_.28URL.29
define('WP_HOME','http://blog.mfabrik.com');
define('WP_SITEURL','http://blog.mfabrik.com');
HTTP 404 Not Found special caseHttp 404 Not Found responses are not themed by Apache filter chain. This is not possible due to order of pipeline in Apache. As a workaround you can set up a custom HTTP 404 page in Wordpress which does not expose the old theme.
For more information see Roll-out checklistBelow is a checklist you need to go to through to confirm that the theme integration works on your production site
Userland templates for Plone – template engine abstraction layer for PythonPosted on November 9, 2008 by Mikko OhtamaaFiled Under Plone (old), python, zope I have been working with collective.easytemplate product which allows users to use template tags on various places on Plone site. Currently supporting
The users can place ${title}, ${object_url} and other template in the edit mode. These variables which are directly mapped from Archetypes fields when the content is viewed/sent. Also, one can register custom snippet generators like $list_folder_content. I hope Easy Template to cover some more actions in the future. I have noted PloneFormGen and Singing & Dancing product authors that we could add some mixed in functionality together. Currently Easy Template uses Cheetah template backend. Cheetah is not Zope security friendly and exposing templated actions should be allowed only to trusted members. I am not huge fan of Plone’s TAL template language which is based on XML attributes and thus suitable only be used in XML context – this language is aimed only for hardcore hackers and software designers and ordinary folk really cannot wrap their minds around it. Because I am not sure which will be the chosen template backend in the future I chose to abstract the template engine layer away. I created collective.templateengines product. It is a bunch of Zope interfaces and utility functions to abstract away common template actions like
Currently collective.templateengines supports Cheetah and Django templates. So, dear audience, what do you think of all this? What template engine would you suggest which would be Kupu friendly – you can edit the template language in WYSIWYG editor? Do you see any other usages for collective.templateengines? Which other projects could adopt template engine abstraction layer? Python code management & deployment – a glance at zc.buildout and few othersPosted on September 2, 2008 by Tuukka MustonenFiled Under development tools, django, python, zope We’ve been using zc.buildout for Plone deployment and it’s working out great. A few days ago implemented a buildout recipe for Django project deployment, automatic web configuration, symlinking, media-folder structuring etc. and while I got it working, I came up with twisted feelings. Buildout is from the creators of Zope (I suppose) so you can expect a powerful project code management tool. The question is, however, whether or not it suits your needs. In my case I found out it too heavy. I mean, to add even a simple task you have to create a new “recipe” (a package) that does the tricks. Of course some recipes are generic (found from PyPi) and you can just run them with your own INI options, but in my case I had to do some custom implementation. Creating a new python package isn’t that hard for sure I found out that zc.buildout has some nice features like:
The problems?
There’s no denying zc.buildout is powerful, but I wouldn’t use it for projects which need reasonable amount of customization. It’s just plain easier and quicker to write shell scripts and while those won’t provide you with any sort of ready tools you won’t propably need them. For bringing up somewhat static environment, where you don’t need to hack things (like that for Plone) it’s quite a decent option, however. I also explored alternatives to zc.buildout. I’ve been reading about earlier virtualenv but haven’t really tried it out until now. It looks very promising and creates a more flexible environment compared to zc.buildout. Of course their goals are not exactly the same. Also, there are a few other alternatives out there, among them a new Python code management tool called Paver (just look at that cool logo.. it does remind you of Indiana Jones, does it not?). I glanced through the Paver docs and it looks like it might be the way to go (Paver also supports virtualenv), but didn’t quite get the grasp of the benefits just yet. Anyway, if you are still interested in code management and deployment, I’d recommend you to read the Paver release announcement and also Paver forewords. They should clear things up. Zope Zeo vs. standalone setupsPosted on July 7, 2008 by Tuukka MustonenFiled Under Plone (old), Red innovation, apache, database, linux, performance, ubuntu, zope We do some Plone development here at Redi. As known, Plone is a powerful, but unfortunately quite a heavy CMS which is best suited for Intranets. Thus, we are always looking for speed increase. Enter Zeo cluster – a feature that nowadays comes bundled with Zope and allows one database (practically Data.fs) to be used by multiple Zope instances, or more accurately Zeo clients. In standalone installation only one CPU / CPU core can be used for processing requests (as Zope / Python implementation is single-threaded AFAIK). So if there are any concurrent requests the database (ZODB, the Zope Object Database) usually has to wait for the request processing before it is asked for the data and only part of the processing power is used as requests are queued. Using Zeo server-client architecture however, each Zeo client can do the processing on their own CPU/core (thus efficiently using the whole CPU prosessing power available) and also minimize the hard disk idle time by asking for data in an ~asynchronous manner (in separate queues). Actually ZODB even serves the same object simultaneously to different client processes for performance reasons. This might raise database ConflictErrors, which are nothing to fear of, however, as noted some paragraphs below. Similarly, you could also deploy Zeo clients on different computers in local network (or wherever you want), but that’s not the scope of this article. Having clients running on different machines is a similar case with the same performance basis, but there are connection lags, bandwith limits and such that decrease performance. Theory vs. practiceDeploying a Zeo cluster instead of standalone Zope instance should theoretically increase the performance by factor of extra available CPUs / CPU cores. There might be some overheads from this setup though, so we tested it out using ApacheBenchmark – the benchmarking module that comes bundled with Apache nowadays. But first something about… Setting up Zeo & converting from standalone modeIn the easiest scenario, setting Zeo up is rather easy: the unified installer supports Zeo-server setup out of the box (=there is a recipe for it). Just run the unified installer like: $ ./install.sh zeo Luckily, the unified installer uses buildout from Plone 3.1 onwards. Thus, converting your current buildout instances to Zeo cluster is nothing but change of buildout configuration. Where you would normally need ‘instance’ section in your buildout.cfg you will now need the following: [zeoserver]
recipe = plone.recipe.zope2zeoserver
zope2-location = ${zope2:location}
zeo-address = 127.0.0.1:12000
#effective-user = __EFFECTIVE_USER__
[client1]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
zeo-client = true
zeo-address = ${zeoserver:zeo-address}
# The line below sets only the initial password. It will not change an
# existing password.
user = admin:mysecretpassword
http-address = 12001
#effective-user = __EFFECTIVE_USER__
#debug-mode = on
#verbose-security = on
# If you want Zope to know about any additional eggs, list them here.
# This should include any development eggs you listed in develop-eggs above,
# e.g. eggs = ${buildout:eggs} ${plone:eggs} my.package
eggs =
${buildout:eggs}
${plone:eggs}
# If you want to register ZCML slugs for any packages, list them here.
# e.g. zcml = my.package my.other.package
zcml =
products =
${buildout:directory}/products
${productdistros:location}
${plone:products}
To add more clients (which is quite the point here), append as many times the extra client sections like this: [client2]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
zeo-client = true
zeo-address = ${zeoserver:zeo-address}
user = ${client1:user}
http-address = 12002
#effective-user = __EFFECTIVE_USER__
#debug-mode = on
#verbose-security = on
eggs = ${client1:eggs}
zcml = ${client1:zcml}
products = ${client1:products}
That minimizes the need for retyping user names, password etc. These examples were taken from Plone unified installer buildout.cfg with ports changed. Starting, stopping & restartingNow, to start your Zeo-powered Plon clients you could type: bin/zeoserver start bin/client1 start bin/client2 start ...same for all the clients... However, the unified installer has a recipe which automatically generates nice and simple shell scripts to control your cluster. In the end of your buildout.cfg, add: [unifiedinstaller]
recipe = plone.recipe.unifiedinstaller
user = ${client1:user}
primary-port = ${client1:http-address}
That should generate the scripts. In fact, it propably does also something else, something which I’m not aware of. However, I didn’t bump into any problems, yet bin/startcluster.sh And that does it (it start server and the clients). Shut it down via: bin/shutdowncluster.sh And restart: bin/restartcluster.sh ConflictErrors – not that errerousAs noted before, in Zeo mode the ZODB might serve the same objects to two more clients at the same time. If one client manipulates the object before others (ie. edits values and saves changes) the other requests will propably fail. This raises ConflicError which looks like this: ConflictError: database conflict error (oid 0x0f39, class HelpSys.HelpSys.ProductHelp) In this case ZODB tries to reprocess the failed requests. This should be common database approach and thus a feature, not a bug (although Zope might want to tell that in error message!). For more accurate explanation see Plone discussion. Parsing it together with web serverThe Zeo components (server and clients) talk to each other via standard Internet protocols (TCP or UDP, not sure). In the default setup, the Zeo server listens to port 8100 and Zeo clients to 8080, 8081, etc. Thus, to access the separate clients as ‘one site’ we need to serve the requests to multiple clients. This can be achieved with load balancers. Apache has at least one: mod_proxy_balancer which should do exactly what we need. Apache isn’t the best choice for achieving high requests per second values, but it will do for our tests (compare to more lightweight but also more limited lighttpd). Just remember that there are other alternatives/methods available, like using squid as load balancer. Our configuration is as follows (inside VirtualHost-directive): <Proxy balancer://lb>
BalancerMember http://127.0.0.1:12001/
BalancerMember http://127.0.0.1:12002/
BalancerMember http://127.0.0.1:12003/
BalancerMember http://127.0.0.1:12004/
</Proxy>
<Location /balancer-manager>
SetHandler balancer-manager
Order Deny,Allow
Allow from all
</Location>
ProxyPass /balancer-manager !
ProxyPass / balancer://lb/http://localhost/VirtualHostBase/http/www.mydomain.com:80/plonesite/VirtualHostRoot/
ProxyPassReverse / balancer://lb/http://localhost/VirtualHostBase/http/www.mydomain.com:80/plonesite/VirtualHostRoot/
This setup also allows us to use the balancer-manager (accessible at /balancer-manager) that comes with mod_proxy_balancer. It’s useful for checking if the configuration is working and balancer is dividing the requests equally. In my setup the balancer is using the default Request Counting -algorithm which divides the requests numerically equally between the instances, but you might want to also try Weighted Traffic Counting, which should be for actual use. In our test only the frontpage is accessed however, so each request’s data transfer is equal and the weighted traffic counting isn’t of use. The testThe server machine
The setup
The tests where run locally in development environment to minimize the network lag (was 0-1ms). The test commandsApacheBenchmark commands: $ ab -n N -c C myurl where N was either 1000 or 9000 (requests) and C 1, 10, 100 or 1000 (concurrent requests). The resultsYou can download the more in-depth test sheet Plone Standalone vs. Zeo installation (PDF). To put it simple: theory and practise meet well – Zeo server is a lot more powerful with concurrent requests. On non-concurrent requests the results are about the same. Having as many Zeo clients as CPUs / CPU cores can boost the performance up to number of extra CPUs/cores. For example, in our quad-core server with Zeo setup we gained nearly 4 times the requests per second of standalone installation (~370% to be accurate). Increasing Zeo clients to 6 didn’t help any as there’s no processing power left from 4 heavily stressed client processes. Also to be noted is that the waiting times for clients nearly tripled (median jumped from 126 to 305 ms) when raising concurrency from 1 to 10. This isn’t bad though – those are still low figures compared to standalone’s median of 1215 ms! Only when raising concurrency to 100 we began to see some 3,6 seconds waiting times (6 seconds for standalone). Increasing concurrency didn’t bring down the requests/second rates much (less than 5%) as expected. Overall, the results were expected, but now we have evidence of it: under concurrent request load Zeo server is a good option to multiply the performance of your site. With very low traffic sites which rarely get more than 1 request at time this doesn’t matter. One bad word about the resource requirements though: The used RAM increase for 6 client Zeo setup (standard Plone 3.1.2 + 12 additional Products) was whopping 621 MB (1132 MB -> 1753 MB). That means about 100 MB per Zeo client as the Zeo server memory intake was only about 12-15 MB. Thus, only use as many Zeo clients as absolutely necessary or you might find your beloved server machine under very serious Zope flu! The good, the bad and the ZopePosted on April 18, 2008 by Mikko OhtamaaFiled Under python, zope I want to use Zope 3 interface package to write component architecture i.e. have a plug-ins easily in Python. Zope 3 interfaces are very handy and, which cannot be conducted from the name, are available outside Zope too. From my prior experiences I know that Zope 3 interfaces package is one of the best and most underrated Python packages out there. It even influenced to the new design of Python 3k. Well then… I haven’t used Zope 3 interfaces standalone before, so the first thing what I do is writing “zope 3 interfaces” into my Google search this. It’s horrible – the very reason I write this quick blog entry. Some notes below (I have written things from the point of external visitor – I have hands deep in Zope myself, so you don’t need to clarify these things for me or teach anything)
In the post “No, you are not smart enough for Zope” Martjin Faassen highlights some problems of Zope community. “It’s hard to get good content written” Martjin claims. I disagree. Whoever created the page originally could have thought what people coming to the page want. They don’t want to decrypt the brain core dump of hardcore Zope developer. They want to know what is this thing, how this thing is beneficial for them, how do I get started with it and how do I use it. You all know how Internet works. You all have visit on web pages. You all are customers for the same thing you also produce. So writing a basic web page is not something you couldn’t do. Hints:
Pardon me the tone of this post. Zope is the 23th best thing out there, but the Zope community has stagnated badly in some aspects. Some things were acceptable ten years ago when web was still young and Python developers hardcore, but if you don’t keep with the pace you lose all the mindshare. Tuning file system performance for Plone developmentPosted on October 24, 2007 by Mikko OhtamaaFiled Under Plone (old), ubuntu, zope I recently read this article about tuning Ext3 file system for better performance. I was doing a fresh Ubuntu 7.10 install on my laptop, so I decided to see how much this would affect to my every day Plone development. On Linux, every time a file is read, its access time attribute is rewritten. This causes a lot of unnecessary writes to file system. Since there are only few rare application needing this feature, turning of the feature can give a nice performance boost on systems dealing with large amount of files. Plone 3.0 has 10000 files. A lot of them are read during the start-up. Maybe I am getting somewhere here… When you are doing Plone development, you need to restart Plone often. I used this highly scientific method to measure Plone start-up time from issuing zopectl fg to getting the front page load completed in Firefox. I warmed the file system cache beforehand by doing two dry runs. I also did some simple front page bombing with ab tool. System setup
Out-of-the-box filesystem Lap 1: 23s Lap 2: 22s Lap 3: 22s ab stats: Concurrency Level: 10 Tuned file system Lap 1: 21s Lap 2: 22s Lap 3: Didn’t bother to do it… ab stats: Concurrency Level: 10 Conclusion “Hooray.” Though Plone/Zope crawls through of thousands of files during the start up (and thus touches their access times), the slow start-up process seem to be CPU bound. Magic file system tricks won’t make your everyday Plone development more effective. A quick tryout: Documentation Generating for Plone productsPosted on September 3, 2007 by Tuukka MustonenFiled Under Plone (old), development tools, python, zope Plone is a modular CMS, which can be expanded with additional products. That means new features are easy to install, and also to customize. However, quickly understanding code that other people wrote, might turn tricky as there are as each coder uses his own style. Therefore, it might be useful to get an overall picture of the system before diving into details. Documentation generators are useful for giving a comprehensive view on code. These are applications that traverse through code and extract information out of it. They use the structured information then to produce a nice looking reference of the code. Ever heard about API? Yep. Ever seen that sort of documentation among any 3rd party Plone product? At least I haven’t. Luckily, there a few choices suitable for Plone/Python: Parsers: doxygen (generic), epydoc (defines ‘epytext’, parses also others), docutils (defines and parses ‘reStructuredText’) (Plone API’s at api.plone.org use epydoc btw.) Out of these, I quickly tested doxygen on a Plone product called EasyShop. The result was interesting but without use. EasyShop does only little subclassing and therefore the documentation doxygen produced was basically listings of separate classes and methods. Doxygen uses graphviz to build graphical visualizations of class relations, but those were out of use also. The problem here is that Plone products are not common python packages: they have adapters, utilities, views, events, subscribers and such. Creating dececnt API out of these would need a specific solution targeted at the platform. Documentation generating seems interesting, however, and graphviz the most providing out of the whole bunch. Unfortunately, I couldn’t produce anything useful on my first few tries, but the subject just needs a little more research. After all, think about it: an API-like documentation with UML-like graphs of any Plone product, wouldn’t that be nice? |
but there’s of course some learning curve, so the real question is should you spend time to learn it or not?