|
Installing Python Imaging Library (PIL) under virtualenv or buildoutPosted on November 19, 2009 by Mikko OhtamaaFiled Under plone, python, technology I have greatly struggled to have PIL library support in isolated Python environments like virtualenv –no-site-packages. For example, when installing Satchmo shop under virtualenv:
Though it clearly is there, installed by easy_install PIL command: ls ../lib/python2.5/site-packages/PIL-1.1.7-py2.5-linux-x86_64.egg ArgImagePlugin.py ExifTags.py GimpGradientFile.pyc... Does anyone know if this problem is with PIL itself, eggified PIL or something else? In any case, there is an easy workaround: use system-wide PIL (sudo apt-get install python-imaging) and symlink PIL from your site-wide installation under the isolated Python environment: (satchmo-py25)mulli% pwd /srv/plone/mmaspecial/satchmo-py25/lib/python2.5/site-packages (satchmo-py25)mulli% ln -s /usr/lib/python2.4/PIL . That works for now, but I’d like to learn how to make virtualenv and buildout install PIL egg bullet-proof way.
Subversion global-ignores and .egg-info in Python/Plone developmentPosted on October 3, 2009 by Mikko OhtamaaFiled Under plone, python, technology Subversion does a good job by ignoring most of build/temporary/unwanted files by default. However, there is one exception still existing at least in Subversion 1.6: Python egg folders. All folders whose name ends up with .egg-info should not committed or considered in version controlling actions. your.package.name.egg-info folder is generated inside your Python egg source folder when you run setup.py / setuptools. If you are working with Python source code eggs, add the following line to your ~/.subversion/config global-ignores = *.o *.lo *.la #*# .*.rej *.rej .*~ *~ .#* .DS_Store *.egg-info *.pyc *.pyo .project .pydevproject Otherwise development tools like Mr. Developer might get confused. Building a mobile site and applications with Django and PythonPosted on September 30, 2009 by Mikko OhtamaaFiled Under Business, django, iphone, linux, mobile, pys60, python, technology Recently we created a mobile site for an interactive bicycle tour. oulugo.mobi (you need to use mobile browser to access the site or you’ll get a redirect) is a multimedia enriched bicycle tour through the historic parts of the city of Oulu. All content is provided by OnGo. The route, which you can bicycle through is drawn on Google Maps. There are nine action points where the user can listen to streaming audio clips, with still images, in his/her mobile phone. This is sort of augmented reality experience: The user sees the real world (where he/she is now bicycling) combined with the historic events (audio playback narrative). For example, at Linnansaari (a location on the route) you’ll see the actual 17th century castle ruins and the narrator tells how the castle exploded when fire, caused by a lighting, reached gunpowder warehouse… boom. The explosion caused stones fly over 400 meters. Alternatively, the clips are available as podcasts from Oulu Tourism pages. You can download them into your iPod for offline listening and use in conjuction with a paper map. This demostrates interesting mix of multichannel publishing: paper, web, mobile and podcasts. The tour is bilingual in Finnish and English. There exists unreleased iPhone application, based on PhoneGap, which allows the user to track his/her location real-time on the web page. We didn’t see it worth of trouble to go through Apple iPhone application review process. When location based service support comes for the browser this feature is indended to be included as the standard HTML5 feature of the service. There also exists Nokia Series 60 mobile application, based on PyS60 and Series 60 BrowserControl API, which allows the user to track his/her location in real-time. The application provides wrapper around Series 60 WebKit control and allows Javascript to access phone native functions (GPS) over localhost socket communication. Like with Apple, we didn’t see real-time tracking feature interesting enough to go through Symbian Signed process to get our application released. Also, BrowserControl had seriousquality problems and we didn’t consider it stable enough for the end users. Some work is available in PyS60 Community Edition repository. The service is hosted on Python specific virtual server on Twinapex services server farm. Features
Software stack
Development effortDevelopment time: Around 100 hours. Three different developers where involved. Used development tools: Eclipse, PyDev, Subclipse, Subversion. There were around five meetings between the content provider and the technology provider. Few beta testing rounds using iPhone application were performed by bicycling in -10 celcius degrees weather (north and so on…). No polar bears were harmed during the creation of this mobile service. The service is linked in from Oulu Tourism pages and thousands of paper brochures printed for Oulu summer season 2009. About the author Mikko Ohtamaa Putting views, like sitemap, into Plone content tree using Easy Template add-onPosted on July 30, 2009 by Mikko OhtamaaFiled Under plone, python, technology Plone has two kind of pages
Sometimes it is desirable, for the sake of uniformness, to put view based pages (accessibility, sitemap) into the content tree. For example, one could want to have the sitemap link appearing only in the navigation tree under the site section “About this site”. Plone add-on product Easy Template provides an easy method to show any Plone view(s) on a normal page. Easy Template uses Django like template syntax (Jinja 2 engine). It gives you great power to drop dynamic content easily on pages. Easy Template also has some security awarness ensuring the members using it cannot escape from their sandbox. Easy Template works in WYSIWYG and non-WYSIWYG modes
Example how to show a sitemap on an arbitary Plone page
It turns out to be:
There is no such thing as a “views reference” for Plone. View names and functions can be figured out by searching and reading through ZCML and Python files in Plone source tree. Some developer insight is needed. For example. for sitemap we can do the grep search: grep -Ri --include="*.zcml" sitemap * Then read Products/CMFPlone/browser/configure.zcml and Products/CMFPlone/browser/sitemap.py. The same thing works in portlets. Use Templated Portlet portlet type. See Easy Template PyPi homepage for the full reference of the product’s potential. About the author Mikko Ohtamaa Setup.py sdist not including all filesPosted on July 24, 2009 by Mikko OhtamaaFiled Under plone, python, technology Setuptools has many silent failure modes. One of them is failure to include all files in sdist release (well not exactly a failure, you could RTFM, but the default behavior is unexpected). This post will serve as a google-yourself-answer for this problem, until we get new, shinier, Distribute solving all of our problems. I b0rked the release for plonetheme.twinapex. Version 1.0 package didn’t include media assets and ZCML configuration files. Luckily Python community reacted quickly and I got advised how to fix it. By default, setuptools include only *.py files. You need to explicitly declare support for other filetypes in MANIFEST.in file. Example MANIFEST.in (plonetheme, built in PyDev): recursive-include plonetheme * recursive-include docs * global-exclude *pyc global-exclude .project global-exclude .pydevproject About the author Mikko Ohtamaa XHTML mobile profile transformer and cleaner for PythonPosted on July 23, 2009 by Mikko OhtamaaFiled Under mobile, plone, python, technology Mobile phones, and especially mobile site validators, are very picky about the validy of XHTML. It must not be any XHTML, but special mobile profile XHTML. Also, search engines like Google, will punish you in the mobile search results if your site fails to conform to mobile profile. This is especially troublesome if you display external content (RSS feeds, ATOM feeds) on your mobile site. Incoming HTML cannot be guaranteed to follow any specification. To solve this problem, we have created gomobile.xhtmlmp Python library which helps you to transform any HTML to content to valid XHTML MP. The library is piloted on plonecommunity.mobi site which uses aggregated content from varying sources. The library is based on lxml.html.Cleaner. The library is part of GoMobile project which aims to create world class Python mobile web development tools. Highlights
As an example we integrated gomobile.xhtmlmp to Feedfeeder Plone add-on product. Enjoy. pygame goes mobilePosted on December 17, 2008 by Mikko OhtamaaFiled Under mobile, python, technology Pygame, the easiest way to make computer games in the world, has just reached your pocket. Check our announcement at http://discussion.forum.nokia.com/forum/showthread.php?t=152969 Userland templates for Plone – template engine abstraction layer for PythonPosted on November 9, 2008 by Mikko OhtamaaFiled Under Plone (old), python, zope I have been working with collective.easytemplate product which allows users to use template tags on various places on Plone site. Currently supporting
The users can place ${title}, ${object_url} and other template in the edit mode. These variables which are directly mapped from Archetypes fields when the content is viewed/sent. Also, one can register custom snippet generators like $list_folder_content. I hope Easy Template to cover some more actions in the future. I have noted PloneFormGen and Singing & Dancing product authors that we could add some mixed in functionality together. Currently Easy Template uses Cheetah template backend. Cheetah is not Zope security friendly and exposing templated actions should be allowed only to trusted members. I am not huge fan of Plone’s TAL template language which is based on XML attributes and thus suitable only be used in XML context – this language is aimed only for hardcore hackers and software designers and ordinary folk really cannot wrap their minds around it. Because I am not sure which will be the chosen template backend in the future I chose to abstract the template engine layer away. I created collective.templateengines product. It is a bunch of Zope interfaces and utility functions to abstract away common template actions like
Currently collective.templateengines supports Cheetah and Django templates. So, dear audience, what do you think of all this? What template engine would you suggest which would be Kupu friendly – you can edit the template language in WYSIWYG editor? Do you see any other usages for collective.templateengines? Which other projects could adopt template engine abstraction layer? How to unit test security declarations in Plone and ZopePosted on October 3, 2008 by Mikko OhtamaaFiled Under Plone (old), python Security is hard. Unit testing security in Plone seems to be even harder. Here is a fool proof example how to do it. After comments I plan to release this as plone.org How to. I hope some of these ideas could get into PloneTestCase itself, so there wouldn’t be need to reinvent the wheel on every product. Since 2004, when I was first introduced to Plone, it has been great mystery to me how to properly unit test your content type and workflow security declarations. Archetypes itself uses ugly hack where it creates secure Python Scripts from strings in Zope and then executes them. There had to be something better, but after asking questions no one seem to know what. Function security declarations (security.declareProtected & co.) are only effective when Python is run in restricted mode. Entering to “restricted Zope Python” has not been very well documented anywhere, until RestrictedPython package Read me got revamped. This finally gave a clue how to one could hit Unauthorized exceptions in unit testing. To enter the promised world of sandboxed Python you need to do following
Zope get_safe_globals() will overwrite __getattr__ with guarded_getattr, etc. providing automatic code execution level security. This information is not usable only for unit testing, but for scripting purposes also – it is a developer heaven to be able to give a sandboxed template environment to the users to play around withoutworry that they can escalate privileges. But getting into restricted mode was not enough… after that all kind of kinks started to hit me. Namely, in some places of Plone items are cached over the request lifecycle. Since unit tests do not create new requests, the cache will contain invalid values. Here borg.localroles bit me badly – I had to dig through the security management layers manually to see why the unit test code was giving bad results. Maybe it would be wise to have a flag for caches and disable them when running on a test layer? Below is the my example code for normal Document content type and simple_publication_workflow. All sandboxed code are declared in independend functions, but it is easy to pass arguments for them. If there is no need to reuse the sandboxed functions, I recommend use Python lambda: function declaration. Functions which should succesfully pass sandbox testing are evaluated using self.execUntrusted(). Functions which are expect to fail are evaluated using self.assertUnauthorized(). import unittest
# Zope security imports
from AccessControl import getSecurityManager
from AccessControl.SecurityManagement import newSecurityManager
from AccessControl.SecurityManagement import noSecurityManager
from AccessControl.SecurityManager import setSecurityPolicy
from AccessControl import ZopeGuards
from AccessControl.ZopeGuards import guarded_getattr, get_safe_globals
from AccessControl.ImplPython import ZopeSecurityPolicy
from AccessControl import Unauthorized
# Restricted Python imports
from RestrictedPython import compile_restricted
from RestrictedPython.Guards import safe_builtins
from RestrictedPython.SafeMapping import SafeMapping
from zope.component import getUtility, getMultiAdapter, getSiteManager
from Products.CMFCore.tests.base.security import UserWithRoles
from Products.CMFCore.WorkflowCore import WorkflowException
from Products.CMFCore.utils import getToolByName
__docformat__ = "epytext"
__author__ = "Mikko Ohtamaa <mikko@redinnovation.com>"
__license__ = "BSD"
class WorkflowTestCase(PloneTestCase):
""" Test workflow access rights. """
def afterSetUp(self):
self.workflow = getToolByName(self.portal, 'portal_workflow')
self.acl_users = getToolByName(self.portal, 'acl_users')
self.types = getToolByName(self.portal, 'portal_types')
self.registration = getToolByName(self.portal, 'portal_registration')
self.membership = getToolByName(self.portal, 'portal_membership')
# Create a normal registered portal member
# to be used in tests
self.registration.addMember("testmember", "secret", ["Member",], properties={ 'username': "testmember", 'email' : "foobar@foobar.com" })
# Set verbose security policy, making debugging Unauthorized
# exceptions great deal easier in unit tests
setSecurityPolicy(ZopeSecurityPolicy(verbose=True))
def clearLocalRolesCache(self):
""" Clear borg.localroles cache.
borg.localroles check role implementation caches user/request combinations.
If we edit the roles for a user we need to clear this cache,
"""
from zope.annotation.interfaces import IAnnotations
ann = IAnnotations(self.app.REQUEST)
for key in ann.keys():
del ann[key]
def loginAsPortalMember(self, id):
''' Login as a normal portal member.
@param id. username
'''
self.login(id)
def _execUntrusted(self, debug, func, *args, **kwargs):
""" Sets up a sandboxed Python environment with Zope security in place.
Calls func() in an sandboxed environment. The security mechanism
should catch all unauthorized function calls (declared
with a class SecurityManager).
Security is effective only inside the function itself -
The function security declarations themselves are ignored.
@param func: Function object
@param args: Parameters delivered to func
@param kwargs: Parameters delivered to func
@param debug: If True, break into pdb debugger just before evaluation
@return: Function return value
"""
# Create global variable environment for the sandbox
globals = get_safe_globals()
globals['__builtins__'] = safe_builtins
globals['_getattr_'] = guarded_getattr
# Create variable context available in the restricted Python
data = { "func" : func,
"args" : ZopeGuards.SafeIter(args),
"kwargs" : kwargs } # TODO: Do we need to map this to SafeMappings?
globals.update(data)
# Our magic code
body = """func(*args, **kwargs)"""
# The following will replace all function calls
# in the code with Zope call guard proxies
code = compile_restricted(body, "<string>", "eval")
# Here is a good place to break in
# if you need to do some ugly permission debugging
if debug:
import pdb
pdb.set_trace()
return eval(code, globals)
def execUntrusted(self, func, *args, **kwargs):
""" Sets up a sandboxed Python environment with Zope security in place. """
return self._execUntrusted(False, func, *args, **kwargs)
def execUntrustedDebug(self, func, *args, **kwargs):
""" Sets up a sandboxed Python debug environment with Zope security in place. """
return self._execUntrusted(True, func, *args, **kwargs)
def assertUnauthorized(self, func, *args, **kwargs):
""" Check that calling func with currently effective roles will raise Unauthroized error. """
try:
self.execUntrusted(func, *args, **kwargs)
except Unauthorized, e:
return
raise AssertionError, 'Unauthorized exception was expected'
def test_document_workflow_access(self):
""" Check that anonymous users cannot access diagnosis in unwanted state. """
def check_set_access(doc, text="foobar"):
""" This is executed as RestrictedPython, print might not be available """
# Try do a call which should hit Zope and Archetypes field security mechanisms
doc.setText(text)
def check_read_access(doc):
""" This is executed as RestrictedPython, print might not be available """
# Try do a call which should hit Zope and Archetypes field security mechanisms
return doc.getText()
def check_workflow_action(portal, action):
""" Publish the document.
Stresses secure workflow execution
"""
portal.portal_workflow.doActionFor(portal.doc, action)
# Login as a manager and create
# an item which is initially private page to play around with
self.loginAsPortalOwner()
self.portal.invokeFactory("Document", "doc")
doc = self.portal.doc
# Item is private by default and editably by creator
self.execUntrusted(check_set_access, doc)
self.logout()
# Anonymous cannot access the document when it's private
self.assertUnauthorized(check_read_access, doc)
self.assertUnauthorized(check_set_access, doc)
# Relogin as a normal member and see we cannot access the item
self.loginAsPortalMember("testmember")
self.assertUnauthorized(check_set_access, doc)
self.logout()
# Now relogin as the manager and share manager role with a member
self.loginAsPortalOwner()
self.membership.setLocalRoles(obj=doc,
member_ids=["testmember"],
member_role="Owner",
reindex=True)
# IMPORTANT: This is a very invisible feature of Plone 3.1 -
# setLocalRoles is ineffective in unit tests unless the cache is cleared
self.clearLocalRolesCache()
self.logout()
# Relogin as a normal member and now we should be able to edit the document
self.loginAsPortalMember("testmember")
doc = self.portal.doc
# Rich text is automatically paragraphed unless it
# begins with HTML element
self.assertEqual(self.execUntrusted(check_read_access, doc), "<p>foobar</p>")
self.execUntrusted(check_set_access, doc)
self.execUntrusted(check_workflow_action, self.portal, "submit")
# Only site manager can publish items
try:
self.execUntrusted(check_workflow_action, self.portal, "publish")
raise AssertionError("Publishing as normal member should not be possible")
except WorkflowException:
# WorkflowException: No workflow provides the '${action_id}' action.
pass
self.logout()
# Now the portal owner publishes the document
self.loginAsPortalOwner()
self.execUntrusted(check_workflow_action, self.portal, "publish")
self.logout()
# Anonymous should now have read access/no edit
self.execUntrusted(check_set_access, doc)
self.assertEqual(self.execUntrusted(check_read_access, doc), "<p>foobar</p>")
# Member should be still able to read and edit the document
self.loginAsPortalMember("testmember")
self.assertEqual(self.execUntrusted(check_read_access, doc), "<p>foobar</p>")
self.logout()
def test_suite():
suite = unittest.TestSuite()
suite.addTest(unittest.makeSuite(WorkflowTestCase))
return suite
Mysterious buildout error – missing docs/HISTORY.txt filePosted on October 1, 2008 by Mikko OhtamaaFiled Under Plone (old), python I was getting the following error with Plone buildout … Develop: '/home/moo/workspace/collective.easytemplate'
Traceback (most recent call last):
File "/tmp/tmp_G8621", line 11, in ?
File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 655, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 931, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 919, in run_setup
run_setup(setup_script, args)
File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 26, in run_setup
DirectorySandbox(setup_dir).run(
File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 63, in run
return func()
File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 29, in <lambda>
{'__file__':setup_script, '__name__':'__main__'}
File "setup.py", line 9, in ?
return open(os.path.join(os.path.dirname(__file__), *rnames)).read()
File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 166, in _open
return _open(path,mode,*args,**kw)
IOError: [Errno 2] No such file or directory: 'docs/HISTORY.txt'
An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 1477, in main
getattr(buildout, command)(args)
File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 324, in install
installed_develop_eggs = self._develop()
File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 556, in _develop
zc.buildout.easy_install.develop(setup, dest)
File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/easy_install.py", line 868, in develop
assert os.spawnl(os.P_WAIT, executable, _safe_arg (executable), *args) == 0
AssertionError
My product had docs folder. HISTORY.txt was there properly. This made me scratch my head for a while. Buildout calls easy_install as an external process. If easy_install eggs have dependencies in their setup.py easy_install tries to download and install these eggs. There is no reported progress what eggs are installed in easy_install process created from buildout. Looks like buildout verbosity (-v) switch does not reach easy_install. So the problem was not in my product, but in its dependency. However the debug output did not reveal that we were dealing with a dependency. Is there easy means to solve this kind of problems? I bluntly put debug prints inside my server wide setuptools Python files to known which was the faulty dependency. It turned out that easy_install was trying to execute setup.py against a downloaded source distribution (.tar.gz). I had the same egg as a local source code copy. The source code contains docs folder, the egg doesn’t. The solution was to change buildout.cfg develop directive to be the same as the flattened dependency order of the eggs (dependencies come top). This way setup.py was evaluated correctly against the source code folder. |
