| Python Interest Group Finland (PIG FI) – join nowPosted on August 13, 2010 by Mikko OhtamaaFiled Under django, plone, python, technology I have started a mailing list / Google group for Python Finnish community. http://groups.google.com/group/pigfi As far as I know such thing didn’t exist before. We discussed about the establishment of Python Finnish user group with several Python activists in Sauna Sprint few days ago (the event arranged by EESTEC and Plonistans, mid term report, final report). There used to exist Plone-related mailing list, but that’s pretty much dead now. If there are other parallel efforts I am not aware of them, but I really like to merge with the existing work whether it exists. I know at least Nokia, Finnish universities and several small enterprises are very active Python users in Finland. Our goal is to establish a vibrant, self-sustaining, Python community of Finland. We want to promote the sharing of Python knowledge across organizational borders, the adoption of Python in corporate and hobbyist cultures and have jolly good time with other Python enthusiasts. This probably means discussion (in Finnish), events and lots of, lots of, sauna. After all, the community will take the shape what suits it for the best, it is not forced externally. The short term goal is to gather enough participants to have a meaningful community. So, Join now!. You can use web interface, email or RSS to follow the discussion. Spread the word to make sure that the message reaches everyone. Tweet it. IRC it. Email it to your friends. SMS it to your parents. Go to streets and cry aloud “Rakastan Pythonia.” For most of us being Python programmer is a choice you have made or would like to take. Let’s together create an environment where this choice is fun and will generate great personal wealth for all of us.
Applying monkey patches in Django middleware layerPosted on August 12, 2010 by Mikko OhtamaaFiled Under django, plone, python, technology Monkey patching is a technique to modify module functions or class methods in Python and other dynamic languages run-time. It differs from the traditional source code patching that it does not need separate utility or compilation process to become effective. This means that you can deploy patches to codebase not under your control with your application without extra effort. Monkey patching has been made famous by Plone/Zope community where there is even collective.monkeypatcher add-on for managed monkey patching. Because monkey patches do not need a compilation stage, the patch will work with the future versions of the application, assuming the patched function or method is not changed. So you can “safely” update the patched software and the patch will apply to the new version without need to go to command-line to perform some cumbersome commands. However, it is the best practice of open source community to report the bugs and submit the fixing patches, as source code patch, in the corresponding issue trackers. In Django context, you can use monkey patching to
Patches are usually applied when Python does module imports. You have a special module called “monkeypatches.py” and when that is imported, it applies the patches when the module body level code runs. However, it is difficult to find stable import point in Django to run monkey patching. Django does some really evil magic to initialize INSTALLED_APPS, database models and stuff and doing any kind of work during import causes headache. So I figured out that you can apply monkey patches using middleware. Middleware applies the monkey patch when the first HTTP request hits the process (note that if you run preforked web server like FCGI every process has its own run-time code in memory). This technique, of course, cannot be used to monkey-patch things that happen before middleware processing, but it is not often needed. Below is an example how to monkey-patch Django CMS to normalize its unicode output. There was an issue with unicode characters and this is a stop-gap measure to fix it. (I think the proper fix would be fix related Cufon font renderin Javascript library). We add our monkey patcher to loaded middleware in settings.py. MIDDLEWARE_CLASSES = (
...
'foobar.middleware.FixCufonUnicodeNormalization',
)
The actual monkey patching happens by fiddling with the class code in process_request(). Note that in this particular case we only need to transform the output of the original function, we can simply hold a reference into it, call it and perform our transformation on the result. This way our monkey patch do not hinder the orignal function and is update safe (it does not matter if the code of PlaceholderNode.render method changes). import unicodedata
# Orignal function we monkey-patched away
_orignal_render = None
def _patched_render(self, context):
""" Normalize all unicode output of Placeholder nodes in Django templates """
content = _orignal_render(self, context)
# http://docs.python.org/library/unicodedata.html
return unicodedata.normalize("NFC", content)
class FixCufonUnicodeNormalization(object):
""" Fix issue with Cufon, user-generated HTML and unicode decomposed characters.
http://github.com/sorccu/cufon/issues/issue/133
"""
def process_request(self, request):
""" Install monkey-patch on demand.
If monkey-patch has not been run in for this process (assuming multiple preforked processes),
then do it now.
"""
from cms.templatetags.cms_tags import PlaceholderNode
global _orignal_render, _patched_render
if not _orignal_render:
# replace one of the class's method with own fixed version
_orignal_render = PlaceholderNode.render
PlaceholderNode.render = _patched_render
As far as I know, monkey patching is something PHP cannot do
Building a mobile site and applications with Django and PythonPosted on September 30, 2009 by Mikko OhtamaaFiled Under Business, django, iphone, linux, mobile, pys60, python, technology Recently we created a mobile site for an interactive bicycle tour. oulugo.mobi (you need to use mobile browser to access the site or you’ll get a redirect) is a multimedia enriched bicycle tour through the historic parts of the city of Oulu. All content is provided by OnGo. The route, which you can bicycle through is drawn on Google Maps. There are nine action points where the user can listen to streaming audio clips, with still images, in his/her mobile phone. This is sort of augmented reality experience: The user sees the real world (where he/she is now bicycling) combined with the historic events (audio playback narrative). For example, at Linnansaari (a location on the route) you’ll see the actual 17th century castle ruins and the narrator tells how the castle exploded when fire, caused by a lighting, reached gunpowder warehouse… boom. The explosion caused stones fly over 400 meters. Alternatively, the clips are available as podcasts from Oulu Tourism pages. You can download them into your iPod for offline listening and use in conjuction with a paper map. This demostrates interesting mix of multichannel publishing: paper, web, mobile and podcasts. The tour is bilingual in Finnish and English. There exists unreleased iPhone application, based on PhoneGap, which allows the user to track his/her location real-time on the web page. We didn’t see it worth of trouble to go through Apple iPhone application review process. When location based service support comes for the browser this feature is indended to be included as the standard HTML5 feature of the service. There also exists Nokia Series 60 mobile application, based on PyS60 and Series 60 BrowserControl API, which allows the user to track his/her location in real-time. The application provides wrapper around Series 60 WebKit control and allows Javascript to access phone native functions (GPS) over localhost socket communication. Like with Apple, we didn’t see real-time tracking feature interesting enough to go through Symbian Signed process to get our application released. Also, BrowserControl had seriousquality problems and we didn’t consider it stable enough for the end users. Some work is available in PyS60 Community Edition repository. The service is hosted on Python specific virtual server on Twinapex services server farm. Features
Software stack
Development effortDevelopment time: Around 100 hours. Three different developers where involved. Used development tools: Eclipse, PyDev, Subclipse, Subversion. There were around five meetings between the content provider and the technology provider. Few beta testing rounds using iPhone application were performed by bicycling in -10 celcius degrees weather (north and so on…). No polar bears were harmed during the creation of this mobile service. The service is linked in from Oulu Tourism pages and thousands of paper brochures printed for Oulu summer season 2009. About the author Mikko Ohtamaa Putting views, like sitemap, into Plone content tree using Easy Template add-onPosted on July 30, 2009 by Mikko OhtamaaFiled Under plone, python, technology Plone has two kind of pages
Sometimes it is desirable, for the sake of uniformness, to put view based pages (accessibility, sitemap) into the content tree. For example, one could want to have the sitemap link appearing only in the navigation tree under the site section “About this site”. Plone add-on product Easy Template provides an easy method to show any Plone view(s) on a normal page. Easy Template uses Django like template syntax (Jinja 2 engine). It gives you great power to drop dynamic content easily on pages. Easy Template also has some security awarness ensuring the members using it cannot escape from their sandbox. Easy Template works in WYSIWYG and non-WYSIWYG modes
Example how to show a sitemap on an arbitary Plone page
It turns out to be:
There is no such thing as a “views reference” for Plone. View names and functions can be figured out by searching and reading through ZCML and Python files in Plone source tree. Some developer insight is needed. For example. for sitemap we can do the grep search: grep -Ri --include="*.zcml" sitemap * Then read Products/CMFPlone/browser/configure.zcml and Products/CMFPlone/browser/sitemap.py. The same thing works in portlets. Use Templated Portlet portlet type. See Easy Template PyPi homepage for the full reference of the product’s potential. About the author Mikko Ohtamaa Userland templates for Plone – template engine abstraction layer for PythonPosted on November 9, 2008 by Mikko OhtamaaFiled Under Plone (old), python, zope I have been working with collective.easytemplate product which allows users to use template tags on various places on Plone site. Currently supporting
The users can place ${title}, ${object_url} and other template in the edit mode. These variables which are directly mapped from Archetypes fields when the content is viewed/sent. Also, one can register custom snippet generators like $list_folder_content. I hope Easy Template to cover some more actions in the future. I have noted PloneFormGen and Singing & Dancing product authors that we could add some mixed in functionality together. Currently Easy Template uses Cheetah template backend. Cheetah is not Zope security friendly and exposing templated actions should be allowed only to trusted members. I am not huge fan of Plone’s TAL template language which is based on XML attributes and thus suitable only be used in XML context – this language is aimed only for hardcore hackers and software designers and ordinary folk really cannot wrap their minds around it. Because I am not sure which will be the chosen template backend in the future I chose to abstract the template engine layer away. I created collective.templateengines product. It is a bunch of Zope interfaces and utility functions to abstract away common template actions like
Currently collective.templateengines supports Cheetah and Django templates. So, dear audience, what do you think of all this? What template engine would you suggest which would be Kupu friendly – you can edit the template language in WYSIWYG editor? Do you see any other usages for collective.templateengines? Which other projects could adopt template engine abstraction layer? Python code management & deployment – a glance at zc.buildout and few othersPosted on September 2, 2008 by Tuukka MustonenFiled Under development tools, django, python, zope We’ve been using zc.buildout for Plone deployment and it’s working out great. A few days ago implemented a buildout recipe for Django project deployment, automatic web configuration, symlinking, media-folder structuring etc. and while I got it working, I came up with twisted feelings. Buildout is from the creators of Zope (I suppose) so you can expect a powerful project code management tool. The question is, however, whether or not it suits your needs. In my case I found out it too heavy. I mean, to add even a simple task you have to create a new “recipe” (a package) that does the tricks. Of course some recipes are generic (found from PyPi) and you can just run them with your own INI options, but in my case I had to do some custom implementation. Creating a new python package isn’t that hard for sure I found out that zc.buildout has some nice features like:
The problems?
There’s no denying zc.buildout is powerful, but I wouldn’t use it for projects which need reasonable amount of customization. It’s just plain easier and quicker to write shell scripts and while those won’t provide you with any sort of ready tools you won’t propably need them. For bringing up somewhat static environment, where you don’t need to hack things (like that for Plone) it’s quite a decent option, however. I also explored alternatives to zc.buildout. I’ve been reading about earlier virtualenv but haven’t really tried it out until now. It looks very promising and creates a more flexible environment compared to zc.buildout. Of course their goals are not exactly the same. Also, there are a few other alternatives out there, among them a new Python code management tool called Paver (just look at that cool logo.. it does remind you of Indiana Jones, does it not?). I glanced through the Paver docs and it looks like it might be the way to go (Paver also supports virtualenv), but didn’t quite get the grasp of the benefits just yet. Anyway, if you are still interested in code management and deployment, I’d recommend you to read the Paver release announcement and also Paver forewords. They should clear things up. Updated: Eclipse web developer plug-in memoPosted on July 14, 2008 by Mikko OhtamaaFiled Under Plone (old), Uncategorized, eclipse, python Below are my personal notes what plug-ins are needed to get “perfect” Eclipse web development set-up. Basically they are just my own notes so that I don’t need to Google everything all over again every time I reinstall. I hope the readers can find new pearls here or suggest improvements. This post is update to previous Eclipse web developer plug-in memo post. New versions are available and some plug-ins have become deprecated. This blog post reflects those changes. These instructions are good for:
Choosing Eclipse distribution
sudo apt-get install sun-java6 eclipse
EasyEclipse bundles some of the stuff listed here with it – when using EasyEclipse you don’t need to have separate PyDev and Subclipse downloads. Eclipse for 64-bit Linux has various problems. You might want to run 32-bit Eclipse (another relevant blog post). When you use Linux distribution specific Eclipse install, all your personal Eclipse files go to .eclipse folder under your home folder. Installing plug-insEclipse has internal updater/web installer. All plug-ins are downloaded as ZIP files and extracted to Eclipse folder or installed through the internal updater. Paste Eclipse update site URLs to menu Help -> Software updates -> Find and Install, New Remote Location. PythonPyDev is a plug-in for Python and Jython development. It has enhanced commercial extensions for professional developers with more intelligent autocomplete and debugger. Site URL: http://pydev.sourceforge.net PyDev Eclipse update URL: http://pydev.sourceforge.net/updates/ PyDev extensions Eclipse update URL (this commercial, but worth of every penny): http://www.fabioz.com/pydev/updates PDTPDT download provides Eclipse, HTML editor, PHP editor and CSS editor. Site URL: http://www.eclipse.org Eclipse update site URL: http://download.eclipse.org/tools/pdt/updates/ JavaIf you need to do J2EE development use IBM’s Web Tools Platform. If you don’t need Java capabilities don’t install these, since they just bloat Eclipse and make the start up time worse. SubclipseSubclipse provides Subversion version control integration to Eclipse. Eclipse update site URL: http://subclipse.tigris.org/update_1.4.x/ In the installer, uncheck the integration modules checkbox or the installer will complain about missing modules. Aptana Studio
Aptana Studio is state-of-the-art Web 2.0 development suite for Eclipse. It has Javascript, CSS and HTML editors. It supports various Javascript libraries out of the box and has support for Firefox and IE in-browser Javascript debugging. Eclipse update site URL: http://update.aptana.com/update/studio/3.2/site.xml ShellEdSyntax coloring for Unix shell scripts Project site: http://sourceforge.net/projects/shelled SQL ExplorerSQL terminal and SQL editor with some GUI capabilities. Eclipse update site URL http://eclipsesql.sourceforge.net/ SQL Explorer needs MySQL JDBC driver. Download from here. Install MySQL connector by extracting the file and adding it from SQL Explorer preferences. Debugging Django memory leak with TrackRefs and GuppyPosted on March 7, 2008 by Mikko OhtamaaFiled Under django, python I run Django in a standalone long-running application (video encoding server). It leaked memory severely. By using htop, one was seeing two gigabytes reserved for /usr/bin/python after a while. Before starting the debugging session, I had no faintest idea what could be the cause of the problem. Django is robust technology – this kind of things haven’t happened for me before. Since I was running Django in standalone mode, I suspected that some query cache does not get cleared. But random poking around the source code didn’t give any clues. It was time to do some serious memory debugging for Python. Python as is doesn’t leak memory, since it’s garbage collected virtual machine. All “leaks” are design problems in the application logic.I found a good primer here what’s going inside Python’s memory management. First I tried this nice TrackRefs class from Zope. It relies on Python’s own in-interpreter functions to monitor objects. class TrackRefs:
"""Object to track reference counts across test runs."""
def __init__(self, limit=40):
self.type2count = {}
self.type2all = {}
self.limit = limit
def update(self):
obs = sys.getobjects(0)
type2count = {}
type2all = {}
for o in obs:
all = sys.getrefcount(o)
if type(o) is str and o == '<dummy key>':
# avoid dictionary madness
continue
t = type(o)
if t in type2count:
type2count[t] += 1
type2all[t] += all
else:
type2count[t] = 1
type2all[t] = all
ct = [(type2count[t] - self.type2count.get(t, 0),
type2all[t] - self.type2all.get(t, 0),
t)
for t in type2count.iterkeys()]
ct.sort()
ct.reverse()
printed = False
logger.debug("----------------------")
logger.debug("Memory profiling")
i = 0
for delta1, delta2, t in ct:
if delta1 or delta2:
if not printed:
logger.debug("%-55s %8s %8s" % ('', 'insts', 'refs'))
printed = True
logger.debug("%-55s %8d %8d" % (t, delta1, delta2))
i += 1
if i >= self.limit:
break
self.type2count = type2count
self.type2all = type2all
You need to have Python compiled in debug mode to have sys.getobjects() method. Luckily this beefed up Python binary is availalble from Ubuntu’s stock repository: sudo apt-get install python-dbg python-mysqldb-dbg Note that native Python extensions don’t work unless they are specifically compiled against the Python debug build (python-mysqldb-dbg).. Then I add TrackRefs to my main loop: def run(self):
self.running = True
logger.info("Started worker " + self.get_worker_id_string())
# Memory leak tracking
tracker = TrackRefs()
while self.running:
self.mark_for_download()
self.process_downloads()
self.process_encodings()
tracker.update() # Dump memory here
time.sleep(settings.WORKER_POLL_DELAY) And after running a while I start getting interesting results: 7956 [2008-03-07 02:59:28,767] INFO Jobs needing sources to download 0 7956 [2008-03-07 02:59:28,768] DEBUG Processable jobs: 0 7956 [2008-03-07 02:59:29,754] DEBUG ---------------------- 7956 [2008-03-07 02:59:29,754] DEBUG Memory profiling 7956 [2008-03-07 02:59:29,754] DEBUG insts refs 7956 [2008-03-07 02:59:29,754] DEBUG <type 'int'> 150 137406 7956 [2008-03-07 02:59:29,755] DEBUG <type 'tuple'> 117 130211 7956 [2008-03-07 02:59:29,755] DEBUG <type 'dict'> 5 8331 7956 [2008-03-07 02:59:29,755] DEBUG <type 'str'> 3 27643 7956 [2008-03-07 02:59:29,755] DEBUG <type 'unicode'> 3 4606 7956 [2008-03-07 02:59:29,755] DEBUG <type 'list'> 3 3492 7956 [2008-03-07 02:59:29,756] DEBUG <type 'frame'> 1 962 7956 [2008-03-07 02:59:29,756] DEBUG <type 'cell'> 0 12948 7956 [2008-03-07 02:59:29,756] DEBUG <type 'function'> 0 9479 Woah! Who reserved 130 000 ints and tuples? No wonder that soon python gulps 1 gigabytes of memory. Since this is the only number which grows during the main loop cycling and there is no references to classes or objects debugging becomes a bit more difficult. I need to try to cross-reference the difficult tuple objects. This didn’t go well – with gc.get_referrers() recurive parsing I got some results (example below). But it became soom clearthat debug references from the system itself was impossible: the memory debugging code will always create nasty cyclic references to the system, since it needs to track the objects. I gave up. There had to be something better. 9154 [2008-03-07 04:05:23,571] DEBUG /var/lib/python-support/python2.5/MySQLdb/connections.pyc 9154 [2008-03-07 04:05:23,571] DEBUG <type 'function'> 9154 [2008-03-07 04:05:23,572] DEBUG defaulterrorhandler 9154 [2008-03-07 04:05:23,572] DEBUG <type 'function'> 9154 [2008-03-07 04:05:23,572] DEBUG string_literal 9154 [2008-03-07 04:05:23,572] DEBUG <type 'function'> 9154 [2008-03-07 04:05:23,573] DEBUG unicode_literal 9154 [2008-03-07 04:05:23,573] DEBUG <type 'function'> 9154 [2008-03-07 04:05:23,573] DEBUG string_decoder 9154 [2008-03-07 04:05:23,573] DEBUG <type 'function'> 9154 [2008-03-07 04:05:23,573] DEBUG __exit__ 9154 [2008-03-07 04:05:23,574] DEBUG <type 'function'> 9154 [2008-03-07 04:05:23,574] DEBUG begin 9154 [2008-03-07 04:05:23,574] DEBUG <type 'function'> 9154 [2008-03-07 04:05:23,574] DEBUG __init__ 9154 [2008-03-07 04:05:23,574] DEBUG <type 'function'> 9154 [2008-03-07 04:05:23,575] DEBUG show_warnings There was: Guppy. Thank you Sverker Nilsson! You saved my day. Since the API of Guppy is a little eccentric, here are some examples for you: # init heapy
heapy = guppy.hpy()
# Print memory statistics
def update():
print heapy.heap()
# Print relative memory consumption since last sycle
def update():
print heapy.heap()
heapy.setref()
# Print relative memory consumption w/heap traversing
def update()
print heapy.heap().get_rp(40)
heapy.setref()
With heapy.heap() ; heapy.setref() I got this output: Partition of a set of 12 objects. Total size = 3544 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 3 25 2244 63 2244 63 unicode
1 2 17 708 20 2952 83 types.FrameType
2 3 25 432 12 3384 95 dict (no owner)
3 3 25 120 3 3504 99 str
4 1 8 40 1 3544 100 list
One adds get_rp() travelsal magic and everything becomes clear: Reference Pattern by <[dict of] class>. 0: _ --- [-] 14 (dict (no owner) | list | str | types.FrameType | types.Gene... 1: a [-] 3 dict (no owner): 0x8c11f34*2, 0x8c1bd54*2, 0x8c1f854*2 2: aa ---- [-] 1 list: 0x833c504*18 3: a3 [-] 1 dict of django.db.backends.mysql.base.DatabaseWrapper: 0x8... 4: a4 ------ [-] 1 dict (no owner): 0x83a65d4*2 5: a5 [R] 1 guppy.heapy.heapyc.RootStateType: 0xb787c7a8L 6: a3b ----- [-] 1 django.db.backends.mysql.base.DatabaseWrapper: 0x8356a34 7: a3ba [S] 7 dict of module: ..db, ..models, ..query, ..transaction... 8: b ---- [S] 1 types.FrameType: <<lambda> at 0x8b16ecc> 9: c [-] 2 list: 0x833c504*18, 0xb7dafe6cL*5 <Type e.g. '_.more' for more.> What there could in DatabaseWrapper object which is growing and growing… query debugger. Django keeps track of all queries for debugging purposes (connection.queries). This list is reseted at the end of HTTP request. But in standalone mode, there are no requests. So you need to manually reset to queries list after each working cycle. while self.running:
self.mark_for_download()
self.process_downloads()
self.process_encodings()
tracker.update()
time.sleep(settings.WORKER_POLL_DELAY)
# Clear database connection ad reset query debugger
# between cycles to make sure that
# related resources get released
reset_queries()
connection.close()
print str(connection.queries)
But even after this fix, I got increase in tuple and int usage when monitoring with TrackRefs. But when I run heapy.heap() alone, there is no increase. So the tuple and int consumption must have been caused by TrackRef, sys.getobjects, gc, etc. magic itself. Eclipse web developer plug-in memoPosted on November 27, 2007 by Mikko OhtamaaFiled Under Plone (old), eclipse, php, python Currently I work in quite wide field of software development: Python (standalone, Plone, Zope, Django), PHP, Java, Symbian and embedded Linux. I am using Eclipse for development, since it’s pretty much the only consistent platform filling my needs. The nature of work also forces me to use different computers (Mac/Windows/Linux) with different clients. This drives me to reinstall Eclipse now and then. Below are my personal notes what plug-ins are needed to get “perfect” Eclipse set-up. Basically they are just my own notes so that I don’t need to Google everything all over again every time I reinstall. I hope the readers can find new pearls here or suggest improvements. Eclipse setupEclipse has internal updater/web installer. All plug-ins are downloaded as ZIP files and extracted to Eclipse folder or installed through the internal updater. Paste Eclipse update site URLs to menu Help -> Software updates -> Find and Install, New Remote Location. You can use dummy text as the name of update site. Eclipse WTP (Web Tools Platform)Eclipse Web Tools Platform bundles Eclipse, Java development tools, HTML editor, CSS editor and some other generic useful stuff.
PythonPyDev is a plug-in for Python and Jython development. Site URL: http://pydev.sourceforge.net Eclipse update site URL: http://pydev.sourceforge.net/updates/ PDTPDT download provides Eclipse, HTML editor, PHP editor and CSS editor. Site URL: http://www.eclipse.org Eclipse update site URL: http://download.eclipse.org/tools/pdt/updates/ SubclipseSubclipse provides Subversion version control integration to Eclipse. Eclipse update site URL: http://subclipse.tigris.org/update_1.2.x In the installer, uncheck the integration modules checkbox or the installer will complain about missing modules. JSEclipseJSEclipse provides a better editor (over WTP) for Javascript files, with impressive outlining and autofill capabilities. Download requires Adobe developer account or similar fill-in-the-fields crap. Site URL: http://labs.adobe.com/technologies/jseclipse/ ShellEdSyntax coloring for Unix shell scripts Project site: http://sourceforge.net/projects/shelled SQL ExplorerSQL editor with limited GUI capabilities. Based on Eclipse platform. Comes standalone and as Eclipse plug-in.
needs MySQL JDBC driver Technorati tags: Python Plone Django PHP Eclipse Web development Subclipse Javascript SQL |
Latest Top Shouts | Want to get mobile? |
