Contact Us

If you are interested in our services leave your contact details below and our sales representatives will contact you.

The organization which you represent
Email address we will use to contact you
Longer contact form…
 
  • About

    mFabrik Blog is about mobile and web software development, open source and Linux. We tell exciting tales where business, technology, web and mobile convergence.

    Creative Commons License
    This work is licensed under a Creative Commons Attribution 3.0 Unported License.

Plone 4 released – the best open source CMS of 2010?



The long awaited version 4 of enterprise grade Plone CMS has been released.

Vote Plone 4 release news on reddit.

Yeah, yeah… it is a linkadvertisement :)

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

How people perceive Plone outside Plone community



Our company does business with multiple CMS systems, like Joomla, Plone and Drupal.  They all have their advantages, they all have their disadvantages. We do not want to make CMS a religion. It’s a tool. You can argue with the client which tool is a right job for a task. Joomla is lightweight solution for non-critical systems, Plone is good with lots of content, editors and workflows flying around. etc. etc.

I had this curious piece on conversation on #joomla channel on freenode. Though it is an individual case, I hope it will bring some light to the fact how people perceive Plone outside Plone community and what Plone should to do fix it.

I think it would be beneficial for Plone to finally close mailman for the site administration / user support and move to real web forums / Google Groups / whatever which would be usable.

Also, there is an example how unprofessionalism is not good for the community.

[20:23:29] x: Biggest problem so far is finding competent ("I will deliver on this schedule) joomla consulting experts. Second biggest problem is security, our site has been hacked 3 times in the past 6 months
[20:25:18] me: have you considered any alternative CMS with better security track record?
[20:25:50] x: moo: we moved from Plone to Joomla. 3 years on Plone with no hacks.
[20:26:04] x: Problem with plone is no forums with email support
[20:26:21] me: you pay for support
[20:26:30] z: did you do basic joomla security guidelines?
[20:26:39] me: also check http://plone.org/support
[20:26:40]  Title: Support options for Plone — Plone CMS: Open Source Content Management (at plone.org)
[20:26:51] x: moo: I'm fine with paying for support. We're paying SiteGround $200-$300/month on average when you add the support costs.
[20:26:58] z: ie using a key to access admin, changing default sql prefix
[20:27:01] AngryPerson: who cares about plone
[20:27:10] AngryPerson: its an ancient cms thats clearly past its time
[20:27:20] AngryPerson: its only privately supported with little community support
[20:28:34] AngryPerson: Moo^_^: why are you even in here?
[20:28:40] AngryPerson: you just want to piss on joomla?
[20:28:43] me: we do business on drupal, joomla and plone
[20:28:48] me: different tool for different job
[20:28:51] -*- y shrugs
[20:29:01] AngryPerson: just seems to me like you want to push ppl away from joomla
[20:29:11] x: z: After 6 months I'm still a Joomla noob. I need a consulting services company that will do the security patching, maintenance, service on the site, and host it.
[20:29:25] me: not true
[20:29:36] AngryPerson: Moo^_^: well regardless of what you say, tahts how it seems to me
[20:29:54] z: actually I've had good luck just following a few blog posts
[20:30:10] me: I don't defend myself, as I don't want to engage such a conversation with you
[20:30:22] AngryPerson: thats good, why dont you fuck off too
[20:30:22] <-* jools has kicked y from #joomla (Please watch your language) [20:30:22] --> y (dgdf@unaffiliated/anti-mttr/x-9384728) has joined #joomla
[20:30:27] z: nothing is impervious, but you drastically reduce your attractiveness to hackers by a few simple steps
[20:30:32] AngryPerson: stop giving ppl your shitty advice

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Peek-a-boo – Python logo spotted in outdoor advertisement!



This caught my eye when coming home from the work.

The finger points straight to the snake…

Kudos to Nokia for this. Nokia’s phones are the best platform if you wish to have pleasant rapid mobile application development in Python – just stay away from Symbian and Series 40 models.

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Python Interest Group Finland (PIG FI) – join now



I have started a mailing list / Google group for Python Finnish community.

http://groups.google.com/group/pigfi

As far as I know such thing didn’t exist before. We discussed about the establishment of Python Finnish user group with several Python activists in Sauna Sprint few days ago (the event arranged by EESTEC and Plonistans, mid term report, final report). There used to exist Plone-related mailing list, but that’s pretty much dead now. If there are other parallel efforts I am not aware of them, but I really like to merge with the existing work whether it exists. I know at least Nokia, Finnish universities and several small enterprises are very active Python users in Finland.

Our goal is to establish a vibrant, self-sustaining, Python community of Finland. We want to promote the sharing of Python knowledge across organizational borders, the adoption of Python in corporate and hobbyist cultures and have jolly good time with other Python enthusiasts. This probably means discussion (in Finnish), events and  lots of, lots of, sauna. After all, the community will take the shape what suits it for the best, it is not forced externally.

The short term goal is to gather enough participants to have a meaningful community. So, Join now!. You can use web interface, email or RSS to follow the discussion. Spread the word to make sure that the message reaches everyone. Tweet it. IRC it. Email it to your friends. SMS it to your parents. Go to streets and cry aloud “Rakastan Pythonia.”

For most of us being Python programmer is a choice you have made  or would like to take. Let’s together create an environment where this choice is fun and will generate great personal wealth for all of us.

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Applying monkey patches in Django middleware layer



Monkey patching is a technique to modify module functions or class methods in Python and other dynamic languages run-time. It differs from the traditional source code patching that it does not need separate utility or compilation process to become effective. This means that you can deploy patches to codebase not under your control with your application without extra effort. Monkey patching has been made famous by Plone/Zope community where there is even collective.monkeypatcher add-on for managed monkey patching.

Because monkey patches do not need a compilation stage, the patch will work with the future versions of the application, assuming the patched function or method is not changed. So you can “safely” update the patched software and the patch will apply to the new version without need to go to command-line to perform some cumbersome commands. However, it is the best practice of open source community to report the bugs and submit the fixing patches, as source code patch, in the corresponding issue trackers.

In Django context, you can use monkey patching to

  • Fix bugs or modify features of Django core without touching the source code
  • Fix bugs or modify features of Django plug-ins (TinyMCE, filebrowser, Django CMS) without touching the source code

Patches are usually applied when Python does module imports. You have a special module called “monkeypatches.py” and when that is imported, it applies the patches when the module body level code runs. However, it is difficult to find stable import point in Django to run monkey patching. Django does some really evil magic to initialize INSTALLED_APPS, database models and stuff and doing any kind of work during import causes headache.

So I figured out that you can apply monkey patches using middleware. Middleware applies the monkey patch when the first HTTP request hits the process (note that if you run preforked web server like FCGI every process has its own run-time code in memory). This technique, of course, cannot be used to monkey-patch things that happen before middleware processing, but it is not often needed.

Below is an example how to monkey-patch Django CMS to normalize its unicode output. There was an issue with unicode characters and this is a stop-gap measure to fix it. (I think the proper fix would be fix related Cufon font renderin Javascript library).

We add our monkey patcher to loaded middleware in settings.py.

MIDDLEWARE_CLASSES = (
    ...
    'foobar.middleware.FixCufonUnicodeNormalization',
)

The actual monkey patching happens by fiddling with the class code in process_request(). Note that in this particular case we only need to transform the output of the original function, we can simply hold a reference into it, call it and perform our transformation on the result. This way our monkey patch do not hinder the orignal function and is update safe (it does not matter if the code of PlaceholderNode.render method changes).

import unicodedata

# Orignal function we monkey-patched away
_orignal_render = None

def _patched_render(self, context):
    """ Normalize all unicode output of Placeholder nodes in Django templates """

    content = _orignal_render(self, context)

    # http://docs.python.org/library/unicodedata.html
    return unicodedata.normalize("NFC", content)

class FixCufonUnicodeNormalization(object):
    """ Fix issue with Cufon, user-generated HTML and unicode decomposed characters.

http://github.com/sorccu/cufon/issues/issue/133

    """ 

    def process_request(self, request):
        """ Install monkey-patch on demand.

        If monkey-patch has not been run in for this process (assuming multiple preforked processes),
        then do it now.

        """
        from cms.templatetags.cms_tags import PlaceholderNode

        global _orignal_render, _patched_render

        if not _orignal_render:
            # replace one of the class's method with own fixed version
            _orignal_render = PlaceholderNode.render
            PlaceholderNode.render = _patched_render

As far as I know, monkey patching is something PHP cannot do :)

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Running 32-bit chroot on 64-bit Ubuntu server to reduce Python memory usage



Here are documented brief instructions how to run 32-bit chroot’ed environment on 64-bit Ubuntu server. chroot means that you run re-rooted and jailed system inside another system.

What we do here is enabling 32-bit chroot’ed userland on 64-bit server. 32-bit userland and 32-bit Python environment reduces the memory usage of heavy website applications we are running (read: Plone/Zope), since 32-bit has only half of the pointer size and object-oriented programming is all about pointers. Zope is especially memory hungry, because it uses ZODB object database. The developer does not need to worry about when doing queries, updates or caches that much  as the persistent site state is transparent to Python (objects are automatically loaded from database or cache when they are referred). Easy persistency means that almost everything is in the database and you need to have big object cache. Plone has huge client-side, in-process, cache for persistent objects. The default setting is for the cache size 5000 objects. (sidenote: since ZODB cache is in-process and Python does not do threading too well, running big Plone sites means that you need run several processes to handle parallel requests – having multiple processes with big in-process caches means loads of memory consumption)

32-bit userland is especially useful if you need to run Plone on 64-bit VPS (virtual private server) with low amount of available memory (512 MB or 1 GB).

There are some brief measurements about 32-bit vs. 6-bit Python memory usage at the end of this article.

Unless otherwise specified, all command here should be run as the root user of the host system. Commands here are for example only and you need to know what you are doing. If you lack advanced UNIX administration skills we gladly arrange you some commercial training or hosting support.

Installing

Basic schroot installation instructions for Ubuntu can be found here https://help.ubuntu.com/community/DebootstrapChroot . We also install ZopeSkel in the chroo’ed environment for starting creating Plone sites. Note that we are using Ubuntu 8.04 which still ships with Python 2.4 – for later Ubuntus you need to compile Python 2.4 from the scratch.

apt-get install debootstrap
apt-get install schroot

# Old schroot uses global schroot.conf, new versions have
# chroot.d directory

# This is a heredoc, but use what ever editor you like
# to create the configuration
cat <<EOF > /etc/schroot/chroot.d/hardy_i386.conf
[hardy_i386]
description=Ubuntu 8.04 Hardy for i386
location=/srv/chroot/hardy_i386
personality=linux32
root-users=bob
run-setup-scripts=true
run-exec-scripts=true
type=directory
users=bob,john,alice,ploneuser
EOF

mkdir -p /srv/chroot/hardy_i386
debootstrap --variant=buildd --arch i386 hardy /srv/chroot/hardy_i386 \

http://archive.ubuntu.com/ubuntu/

# Check that the chroot is created and working
schroot -l

# Enter the chroot (logged in as bob)
schroot -c hardy_i386 -u root

# Once inside, install python2.4-dev and other needed tools
# Installing PIL with easy_install didn't work for some
# reason, so we use python-imaging package.
apt-get install python2.4-dev python-setuptools python-imaging
easy_install-2.4 ZopeSkel

And that’s about it.

Chroot’ed environment will have

  • It’s own application binaries (and userland bitness)

Chroot’ed environment will share with the host system

  • Ports
  • Processes
  • User accounts

Chroot’ed users

  • Can’t list filesystem outside chroot

…giving additional safety for shared hosting in the case of chroot’ed environment is compromised.

Entering chroot’ed environment as a specific user

Try

schroot -c hardy_i386 -u root

…or…

schroot -c hardy_i386 -u plone_user # After you have set-uped normal user for chroot'ed environment

All background processes you leave running in chroot’ed environment are terminated when you exit this environment, unless you create sessions as described below.

Creating chroot sessions

Sessions enable running commands in chroot without the need to have it constantly open.

# Create a new schroot session
schroot --chroot=hardy_i386 --user=ploneuser \
--session-name=plonesession \
--begin-session

# Run commands in the created session
schroot --chroot=plonesession --user=ploneuser --run-session \
/srv/plone/yourplonesite/bin/instance start

# Ending session
schroot --chroot=plonesession --user=ploneuser --end-session

Note that --chroot parameter takes in both actual chroot installations and session ids.

Doing resets for chrooted environment

The session processes exist as long as the session exist. Unless you explicitly start a new session with –begin-session the processes are terminated as soon as you log out from the chroot’ed environent.
chroot’ed environment is temporary unless you explicitly specify it not be
Thus if you want to run daemonized services in chrooted environment you need to take care of session handling manually.
Here is an example how do you construct a session (as a real root user) and then launch a shell script which will take care of launching applications inside the chroot.

# Terminate previous session if any
schroot --chroot=hardy_i386 --user=plone_user --session-name=plone_session --end-session
#Start the session (again)
schroot --chroot=hardy_i386 --user=plone_user --session-name=plone_session --begin-session
# Run a start script inside the chroot'ed environment which will start Plone
# NGINX and other necessary 32-bit services
schroot --chroot=plone_session --user=plone_user --run-session /srv/plone/myplonesite/restart-all.sh

Running sessions at startup

You can add schroot bootstrap in real /etc/rc.local:

schroot --chroot=hardy_i386 --user=ploneuser \
--session-name=plonesession --begin-session \
&& schroot --chroot=plonesession --run-session \
/srv/plone/inst/bin/instance start

Remember that the users have to be created outside the chrooted environment. If you set the home directory to something that exists only in the chrooted environment, use something like this

adduser --no-create-home --home HOMEDIR_IN_CHROOT ploneuser

Then to create the directory inside the chroot and set its ownership to the newly created user and group.

32-bit vs 64-bit memory consumption

Reason why we even tried this is that some python applications, like Zope, use references heavily and moving from 32bit to 64bit references increases memory usage. (J Stahl 2010)

Memory figures from a development Zope/Plone 3.3.5 server

32-bit 64-bit
After startup 112 MiB RES
116 MiB VIRT
175 MiB RES
342 MiB VIRT
After normal usage 159 MiB RES
194 MiB VIRT
236 MiB RES
487 MiB VIRT

This is far from a complete study, but it would seem that the chroot does pay off even though it has to load 32bit versions of basic libraries along. If running more than one instance on same server memory savings should increase.

More information

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

.gitignore for Python developers



If you are using Git for version control for your Python egg and buildout development below are is a sample which you might want to put into your .gitignore file.

*.mo
*.egg-info
*.egg
*.EGG
*.EGG-INFO
bin
build
develop-eggs
downloads
eggs
fake-eggs
parts
dist
.installed.cfg
.mr.developer.cfg
.hg
.bzr
.svn
*.pyc
*.pyo
*.tmp*

Suggestions for new ignores are welcome.

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Zoho integration for Python and Plone CMS



Zoho is a web application provider competing with Google Docs, Microsoft Office and Live.

Zoho provides a very wide set of browser based applications from text editing and spreadsheets to project management and customer relationship management (highlighted items should ring a bell for small software development companies). Especially the last one, CRM, is a very attractive deal as you get a hosted complex CRM application with API services for very affordable or free price. Small organizations are not necessarily rich enough to go for Salesforce API supported edition which would be 135 € / month / user.

mFabrik has been working on Zoho Python bindings as we use Zoho internally.

Zoho API is HTTP GET/POST based.

  • Authentication, which is called a ticket in Zoho language, is HTTP POST with custom plain-text responses. The same authentication mechanism works in-browser (Javascript) and for machine-to-machine communication as far as I know
  • Most functions can be performed as HTTP POST or GET. If you need to input compex data (like CRM leads), you’ll do it as HTTP POST of custom XML payload
  • Some functions expose the output as JSON for HTTP GET, so that it can be directly consumed inside browser Javascript

mfabrik.zoho is a GPL’ed Python library which provides basic facilities for Zoho API calls. Currently the feature set is very CRM weighted, though it can be easily expanded for other Zoho applications.

mfabrik.plonezohointegration is a Plone CMS add-on product which marry Plone and Zoho together. The add-on provides a control panel where you can enter Zoho API key details for Plone. Forms for CRM lead generation are provided as standalone and as a portlet (you can see them in action on our web and blog site).

The source is hosted on Github, so you can easily start tailoring it for your own organization needs. I happily accept all merge requests, providing that unit tests for new features are included. If you do not feel comfortable with Python programming, but still want to integrate Zoho to your systems, please contact us for further help.

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Easily install all Python versions under Linux and OSX using collective.buildout.python



Here are short instructions how to install all versions (2.4, 2.5, 2.6, 2.7 and 3.1) of Python interpreters on UNIX system. The instructions were tested on Ubuntu 10.04 Lucid Lynx Linux but should work on other systems as is. The installation is based of downloading, compiling and installing different Pythons and their libraries using buildout tool. A buildout configuration for doing this is maintained by a Plone community.

This buildout is especially useful to get Python 2.4 properly running under the latest Ubuntu 10.04 Lucid Lynx. This is because Ubuntu repositories won’t ship with Python 2.4 packages anymore.

The installation will also include static compilation of some very popular libraries. These are dependencies for other Python packages including, but not limited, to

  • libjpeg
  • Python imaging library
  • readline

Prerequisites

  • Some Python version is installed (OS default)
  • GCC compiler is installed (sudo apt-get install build-essential)
  • Subversion tool is installed (sudo apt-get install subversion)

Running it

svn co http://svn.plone.org/svn/collective/buildout/python/
cd python
python bootstrap.py
bin/buildout

Using it

All Pythons are under virtualenv installations. This means that you can activate one Python configuration for your shell once easily (python command will run under different Python versions).

Activating Python 2.4

source python/python-2.4/bin/activate
(python-2.4)moo@murskaamo:~/code$ python -V
Python 2.4.6

Check that Python Imaging Library works

python
Python 2.4.6 (#1, Jul 16 2010, 10:31:46)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import PIL

(No exceptions raised, Python Imaging Library works well).

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Automatically generating description based on body text



Below is a sample script to automatically generate descriptions based on page body text. It is for Plone CMS, but should be applicable to any Python based CMS with some modifications.

The idea is that we take three first sentences and use them as a description.

Use case: People are lazy to write descriptions (descriptions as in Dublin Core metadata). You can generate some kind of description by taking the few first sentences of the text. This is not perfect, but this is way better than empty description. Also, the script comes with good comments which should be helpful for beginner Plone programmers.
Please comment if you have other simple ideas to generate descriptions.
Usage
  • Add  Script (Python) item through Zope Management interface to any Plone folder
  • Put in the code payload below
  • Hit Test tab or type in Script URL manually – note that the operation is one shot only
  • The script iterates through all content items in that folder
  • The script will provide logging output to standard Plone log (var/log and stdout if Plone is run in the debug mode).

Since Zope uses RestrictedPython for through-the-web created scripts, the user of this script cannot breach the server security (they cannot make Python calls they have no permission for). This sets some limitations for automating tasks like this, but we don’t hit those limitations in our use case.

def create_automatic_description(content, text_field_name="text"):
    """ Creates an automatic description from HTML body by taking three first sentences. 

    Takes the body text

    @param content: Any Plone contentish item (they all have description)

    @param text_field_name: Which schema field is used to supply the body text (may very depending on the content type)
    """

    # Body is Archetype "text" field in schema by default.
    # Accessor can take the desired format as a mimetype parameter.
    # The line below should trigger conversion from text/html -> text/plain automatically using portal_transforms
    field = content.Schema()[text_field_name]

    # Returns a Python method which you can call to get field's
    # for a certain content type. This is also security aware
    # and does not breach field-level security provded by Archetypes
    accessor = field.getAccessor(content)

    # body is UTF-8
    body = accessor(mimetype="text/plain")

    # Now let's take three first sentences or the whole content of body
    sentences = body.split(".")

    if len(sentences) > 3:
       intro = ".".join(sentences[0:3])
       intro += "." # Don't forget closing the last sentence
    else:
       # Body text is shorter than 3 sentences
       intro = body

    content.setDescription(intro)

# context is the reference of the folder where this script is run
for id, item in context.contentItems():
     # Iterate through all content items (this ignores Zope objects like this script itself)

     # Use RestrictedPython safe logging.
     # plone_log() method is permission aware and available on any contentish object
     # so we can safely use it from through-the-web scripts
     context.plone_log("Fixing:" + id)

     # Check that the description has never been saved (None)
     # or it is empty, so we do not override a description someone has
     # set before automatically or manually
     desc = context.Description() # All Archetypes accessor method, returns UTF-8 encoded string

     if desc is None or desc.strip() == "":
          # We use the HTML of field called "text" to generate the description
          create_automatic_description(item, "text")

# This will be printed in the browser when the script completes succesfully
return "OK"

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Next Page →