Userland templates for Plone – template engine abstraction layer for Python

I have been working with collective.easytemplate product which allows users to use template tags on various places on Plone site. Currently supporting

• Kupu
• Outgoing email actions (Content rules ones)

The users can place ${title}, ${object_url} and other template in the edit mode. These variables which are directly mapped from Archetypes fields when the content is viewed/sent. Also, one can register custom snippet generators like $list_folder_content.

I hope Easy Template to cover some more actions in the future. I have noted PloneFormGen and Singing & Dancing product authors that we could add some mixed in functionality together.

Currently Easy Template uses Cheetah template backend. Cheetah is not Zope security friendly and exposing templated actions should be allowed only to trusted members. I am not huge fan of Plone’s TAL template language which is based on XML attributes and thus suitable only be used in XML context – this language is aimed only for hardcore hackers and software designers and ordinary folk really cannot wrap their minds around it.

Because I am not sure which will be the chosen template backend in the future I chose to abstract the template engine layer away. I created collective.templateengines product. It is a bunch of Zope interfaces and utility functions to abstract away common template actions like

• Applying a template
• Adding a template context variables
• Registering custom template tags

Currently collective.templateengines supports Cheetah and Django templates.

So, dear audience, what do you think of all this? What template engine would you suggest which would be Kupu friendly – you can edit the template language in WYSIWYG editor? Do you see any other usages for collective.templateengines? Which other projects could adopt template engine abstraction layer?

How to unit test security declarations in Plone and Zope

Security is hard. Unit testing security in Plone seems to be even harder. Here is a fool proof example how to do it. After comments I plan to release this as plone.org How to. I hope some of these ideas could get into PloneTestCase itself, so there wouldn’t be need to reinvent the wheel on every product.

Since 2004, when I was first introduced to Plone, it has been great mystery to me how to properly unit test your content type and workflow security declarations. Archetypes itself uses ugly hack where it creates secure Python Scripts from strings in Zope and then executes them. There had to be something better, but after asking questions no one seem to know what.

Function security declarations (security.declareProtected & co.) are only effective when Python is run in restricted mode. Entering to “restricted Zope Python” has not been very well documented anywhere, until RestrictedPython package Read me got revamped. This finally gave a clue how to one could hit Unauthorized exceptions in unit testing.

To enter the promised world of sandboxed Python you need to do following

• Create a globals dictionary containing secured version of all __builtin__ functions and accessable objects
• Compile your Python code through RestrictedPython compiler
• Evaluate the result

Zope get_safe_globals() will overwrite __getattr__ with guarded_getattr, etc. providing automatic code execution level security. This information is not usable only for unit testing, but for scripting purposes also – it is a developer heaven to be able to give a sandboxed template environment to the users to play around withoutworry that they can escalate privileges.

But getting into restricted mode was not enough…  after that all kind of kinks started to hit me. Namely, in some places of Plone items are cached over the request lifecycle. Since unit tests do not create new requests, the cache will contain invalid values. Here borg.localroles bit me badly – I had to dig through the security management layers manually to see why the unit test code was giving bad results. Maybe it would be wise to have a flag for caches and disable them when running on a test layer?

Below is the my example code for normal Document content type and simple_publication_workflow. All sandboxed code are declared in independend functions, but it is easy to pass arguments for them. If there is no need to reuse the sandboxed functions, I recommend use Python lambda: function declaration.

Functions which should succesfully pass sandbox testing are evaluated using self.execUntrusted(). Functions which are expect to fail are evaluated using self.assertUnauthorized().

import unittest

# Zope security imports
from AccessControl import getSecurityManager
from AccessControl.SecurityManagement import newSecurityManager
from AccessControl.SecurityManagement import noSecurityManager
from AccessControl.SecurityManager import setSecurityPolicy
from AccessControl import ZopeGuards
from AccessControl.ZopeGuards import guarded_getattr, get_safe_globals
from AccessControl.ImplPython import ZopeSecurityPolicy
from AccessControl import Unauthorized

# Restricted Python imports
from RestrictedPython import compile_restricted
from RestrictedPython.Guards import safe_builtins
from RestrictedPython.SafeMapping import SafeMapping

from zope.component import getUtility, getMultiAdapter, getSiteManager
from Products.CMFCore.tests.base.security import UserWithRoles
from Products.CMFCore.WorkflowCore import WorkflowException
from Products.CMFCore.utils import getToolByName

__docformat__ = "epytext"
__author__ = "Mikko Ohtamaa <mikko@redinnovation.com>"
__license__ = "BSD"

class WorkflowTestCase(PloneTestCase):
    """ Test workflow access rights. """

    def afterSetUp(self):
        self.workflow = getToolByName(self.portal, 'portal_workflow')
        self.acl_users = getToolByName(self.portal, 'acl_users')
        self.types = getToolByName(self.portal, 'portal_types')
        self.registration =  getToolByName(self.portal, 'portal_registration')
        self.membership =  getToolByName(self.portal, 'portal_membership') 

        # Create a normal registered portal member
        # to be used in tests
        self.registration.addMember("testmember", "secret", ["Member",], properties={ 'username': "testmember", 'email' : "foobar@foobar.com" })

        # Set verbose security policy, making debugging Unauthorized
        # exceptions great deal easier in unit tests
        setSecurityPolicy(ZopeSecurityPolicy(verbose=True))

    def clearLocalRolesCache(self):
        """ Clear borg.localroles cache.

        borg.localroles check role implementation caches user/request combinations.
        If we edit the roles for a user we need to clear this cache,
        """
        from zope.annotation.interfaces import IAnnotations
        ann = IAnnotations(self.app.REQUEST)
        for key in ann.keys():
            del ann[key]

    def loginAsPortalMember(self, id):
        ''' Login as a normal portal member.

        @param id. username
        '''
        self.login(id)

    def _execUntrusted(self, debug, func, *args, **kwargs):
        """ Sets up a sandboxed Python environment with Zope security in place.

        Calls func() in an sandboxed environment. The security mechanism
        should catch all unauthorized function calls (declared
        with a class SecurityManager).

        Security is effective only inside the function itself -
        The function security declarations themselves are ignored.

        @param func: Function object
        @param args: Parameters delivered to func
        @param kwargs: Parameters delivered to func
        @param debug: If True, break into pdb debugger just before evaluation
        @return: Function return value
        """

        # Create global variable environment for the sandbox
        globals = get_safe_globals()
        globals['__builtins__'] = safe_builtins
        globals['_getattr_'] = guarded_getattr

        # Create variable context available in the restricted Python
        data = { "func" : func,
                "args" : ZopeGuards.SafeIter(args),
                "kwargs" : kwargs } # TODO: Do we need to map this to SafeMappings?

        globals.update(data)

        # Our magic code
        body = """func(*args, **kwargs)"""

        # The following will replace all function calls
        # in the code with Zope call guard proxies
        code = compile_restricted(body, "<string>", "eval")

        # Here is a good place to break in
        # if you need to do some ugly permission debugging
        if debug:
            import pdb
            pdb.set_trace()

        return eval(code, globals)

    def execUntrusted(self, func, *args, **kwargs):
        """ Sets up a sandboxed Python environment with Zope security in place. """
        return self._execUntrusted(False, func, *args, **kwargs)

    def execUntrustedDebug(self, func, *args, **kwargs):
        """ Sets up a sandboxed Python debug environment with Zope security in place. """
        return self._execUntrusted(True, func, *args, **kwargs)

    def assertUnauthorized(self, func, *args, **kwargs):
        """ Check that calling func with currently effective roles will raise Unauthroized error. """
        try:
            self.execUntrusted(func, *args, **kwargs)
        except Unauthorized, e:
            return

        raise AssertionError, 'Unauthorized exception was expected'

    def test_document_workflow_access(self):
        """ Check that anonymous users cannot access diagnosis in unwanted state. """

        def check_set_access(doc, text="foobar"):
            """ This is executed as RestrictedPython, print might not be available """

            # Try do a call which should hit Zope and Archetypes field security mechanisms
            doc.setText(text)

        def check_read_access(doc):
            """ This is executed as RestrictedPython, print might not be available """

            # Try do a call which should hit Zope and Archetypes field security mechanisms
            return doc.getText()

        def check_workflow_action(portal, action):
            """ Publish the document.

            Stresses secure workflow execution
            """
            portal.portal_workflow.doActionFor(portal.doc, action)

        # Login as a manager and create
        # an item which is initially private page to play around with
        self.loginAsPortalOwner()
        self.portal.invokeFactory("Document", "doc")
        doc = self.portal.doc
        # Item is private by default and editably by creator
        self.execUntrusted(check_set_access, doc)
        self.logout()

        # Anonymous cannot access the document when it's private
        self.assertUnauthorized(check_read_access, doc)
        self.assertUnauthorized(check_set_access, doc)       

        # Relogin as a normal member and see we cannot access the item
        self.loginAsPortalMember("testmember")
        self.assertUnauthorized(check_set_access, doc)
        self.logout()

        # Now relogin as the manager and share manager role with a member
        self.loginAsPortalOwner()
        self.membership.setLocalRoles(obj=doc,
                      member_ids=["testmember"],
                      member_role="Owner",
                      reindex=True)
        # IMPORTANT: This is a very invisible feature of Plone 3.1 -
        # setLocalRoles is ineffective in unit tests unless the cache is cleared
        self.clearLocalRolesCache()
        self.logout()

        # Relogin as a normal member and now we should be able to edit the document
        self.loginAsPortalMember("testmember")
        doc = self.portal.doc
        # Rich text is automatically paragraphed unless it
        # begins with HTML element
        self.assertEqual(self.execUntrusted(check_read_access, doc), "<p>foobar</p>")
        self.execUntrusted(check_set_access, doc)
        self.execUntrusted(check_workflow_action, self.portal, "submit")
        # Only site manager can publish items
        try:
            self.execUntrusted(check_workflow_action, self.portal, "publish")
            raise AssertionError("Publishing as normal member should not be possible")
        except WorkflowException:
            # WorkflowException: No workflow provides the '${action_id}' action.
            pass
        self.logout()           

        # Now the portal owner publishes the document
        self.loginAsPortalOwner()
        self.execUntrusted(check_workflow_action, self.portal, "publish")
        self.logout()

        # Anonymous should now have read access/no edit
        self.execUntrusted(check_set_access, doc)
        self.assertEqual(self.execUntrusted(check_read_access, doc), "<p>foobar</p>")      

        # Member should be still able to read and edit the document
        self.loginAsPortalMember("testmember")
        self.assertEqual(self.execUntrusted(check_read_access, doc), "<p>foobar</p>")
        self.logout()

def test_suite():
    suite = unittest.TestSuite()
    suite.addTest(unittest.makeSuite(WorkflowTestCase))
    return suite