| Introducing Python for Series 60 Community EditionPosted on September 1, 2008 by Mikko OhtamaaFiled Under mobile, python, series 60, symbian This blog post will introduce Python for Series 60 Community Edition. Pythor for Series 60 Community Edition is a new open source effort to push Python for high quality mobile phone development. It aims to provide a maintained software stack for creating real mobile applications. The codebase is derived from the original Nokia’s Python for Series 60 codebase, but has been refactored for better integration with third party extensions and patches and commercial grade application deployment. MotivationBuilding and distributionIt is difficult to distribute Python for Series 60 applications to the end user with the current Nokia’s PyS60 distribution. You probably want to modify or extend PyS60 in some way. Since the build chain and deployment model is not designed for changes this would collide with the other PyS60 installations. Symbian Platform security prevents installing conflicting binaries. Thus, one can effectively have one Nokia PyS60 application in the phone once. There are other problems: Nokia PyS60 distribution has UIDs in Nokia protected range. Embedded SIS file cause extra installation dialog and an application manager uninstall entry. Trimming down Nokia PyS60 distribution is a little bit difficult. To overcome all these issues we created a build chain which spits out monolithic PyS60 distributions. We build only one DLL whose name and UID can be decided. Also the build chain is Scons for Symbian, scrapping the obscure, inflexible and difficult to understand Symbian ABLD once for all. Evolution towards higher qualityCurrently there is no centralized authority to co-ordinate PyS60 open source developers and maintain the repository of all the extension and patches. This effectively prevents the biggest benefit of open source: open innovation and gradual evolution of the product. It would be very nice having all those third party extensions, now scattered around the internet, under one maintained source – making PyS60 more functional out of the box. The community maintained repositories do not have the same restrictions as ones managed by a big public corporations. It is not a probable target of a trigger happy lawyer action and ungentlemanly competition: the discussion and plans can be public and due dilugence check of the code more relax. We started the project in the Launchpad. Launchpad provide a distributed version control system (Bazaar) which streamlines the process of integrating third party commits and patches. This should encourage contribution. The standard build system makes it easy to roll out applications and extensions from bare C++ source up to the end user distributable SIS files. It is yet to see what kind of co-operation possibilities between the community and Nokia exists. In the future, it should be possible to cherry pick patches from PyS60 community edition to Nokia’s own version. Showing the commercial potential of PyS60 in the mobile application developmentOn Python you can write native Series 60 applications with very little effort compared to hardcore C++ banging, lowering the barries to enter the mobile application development. We do not deny that we have an extrinct motivation called money. Of course we have also instrict motivations like thinking Python is the best programming language in the world and we all want to be most respectable gurus in it. Gurus need to eat still, though. We hope that our effort does not go unnoticed in the mobile application development world and good subcontract offers fill our inboxes. Also, there is the John McClane effect. Unless we had done it, no one had. Somebody has to save the world, despite the hangover. It runs on LinuxSince we are no longer dependend on .BAT/Perl/Windows hindered ABLD buildchain, we can (almost) crosscompile and build native Symbian binaries in Linux and Python applications. All good hackers use Linux – but currently there are still kinks and you need to use WINE for some parts – all sane Symbian developers are tied to Windows based tools for now and so are these instructions. PrerequisitementsYou need all this stuff to get things running. Install BazaarYou need Bazaar distributed version control client. We are not planning to have fixed releases for Python for Series 60 community edition any time soon. This is because 1) the most magic happens at a compiler level and we are providing a buildchain 2) we hope this fosters incoming patches. Why Bazaar?
Install ActiveState PerlSeries 60 SDK has ActiveState as a prerequisitement for running its installer. http://www.activestate.com/Products/activeperl/index.mhtml Install Series 60 SDKUse only Series 60 3.0 maintenance release. Other releases have SDK bugs preventing correct Python compilation. Get the Windows installer from http://forum.nokia.com. Forum Nokia Registration is required. Please use the default installation location C:\Symbian\9.1\S603rd_MR. Install Carbide.c++ expressCarbide.c++ comes with a Windows compiler to compile the emulator binaries. You need this only if you indend to develop and test your applications on Series 60 emulator. http://www.forum.nokia.com/info/sw.nokia.com/id/dbb8841d-832c-43a6-be13-f78119a2b4cb.html Forum Nokia Registration is required. Use Software updater in Carbide.c++ to install PyDev, Python developer extensions for Eclipse. Install Python 2.5Scons build chain and our utility scripts use Python. http://www.python.org/download/releases/2.5.2/ Use the installer EXE and the default installation location C:\Python25. If you want to use advanced Bluetooth shell (PUTools) you also need wxPython and pyserial packages. Install SConsPython for Series 60 build script are based on SCons. It is a build system using Python as a recipe langauge. http://sourceforge.net/project/showfiles.php?group_id=30337 Install SubversionInstall Subversion client for Windows. This is needed for checking out Scons for Series 60. http://www.collab.net/downloads/subversion/ Registration to CollabNet is needed to download Windows binaries. Scons for Series 60SCons for Series 60 is available as a separate project. SCons for Symbian is a build toolchain intended as a replacement for Perl and MMP files used on regular Symbian projects. SCons for Symbian is not limited to build Python – You can use it to build any Series 60 C++ application. http://code.google.com/p/scons-for-symbian/ This is later checked out during to the environment construction, so you do not need to install it now. We have included a workaround for a problem with limited command line length on Windows. Included toolsThe following tools are included in the trunk tools folder: These tools are not licensed under Apache license. Some of them are under GPL license. However, we believe that distribution them is ok, since this falls under GPL’s mere aggregation clause. However if you indent to distribute commercial applications built from PyS60 Community codebase, make sure that you understand the set of different licenses involved. Set up build environmentEnsure that Bazaar is properly in your Windows path. Create a workspace folderFirst you need to subst (make a folder appear as a driver letter) in Windows. Open command line. Go to SDK folder. C:\Symbian\9.1>subst t: S60_3rd_MR T: mkdir workspace Now choose this folder as a workspace folder in Carbide C++ and create an Empty Symbian C++ project called ”pys60” there.
Checkout PyS60 community editionThe go to this folder T: cd workspace\pys60 bzr branch lp:pys60community cd pys60community\src Preparing the buildThis needs to be done only once. We need to pacth the existing Series 60 SDK headers which have some bugs. T: cd \epoc32\include \workspace\pys60\pys60community\src\tools\patch.exe -p1 < \workspace\pys60\pys60community\src\pys60-fix-3rded-sdk.diff EPOCROOT must be set for some Series 60 SDK tools to work. We point to T: drive root. T: cd workspace\pys60\src set EPOCROOT=\ As we still have some dependencies to the legacy system, one needs to configure the build system using PyS60 setup. This will generate some files and defines for Series 60 versio 3.0. c:\Python25\python.exe setup.py configure 30 Do not run bldmake bldfiles. You need to convert legacy MMP build files to SCons based. First we need to possibly fix up PATH, since Carbide C++ might break it. set PATH=c:\program files\bazaar;c:\program files\CSL Arm Toolchain\arm-none-symbian elf\bin;c:\program files\CSL Arm Toolchain\libexec\gcc\arm-none-symbianelf\3.4.3;C:\program files\CSL Arm Toolchain\bin;t:\epoc32\gcc\bin;t:\epoc32\tools;t:\epo c32\tools;C:\program files\CSL Arm Toolchain\bin;C:\Program Files\Common Files\Symbian\Tools;C:\Perl\site\bin;C:\Perl\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem c:\Python25\python tools\mmp2scons.py ext\\calendar\\calendar.mmp:60:7: warning: no newline at end of file Creating recipe ext\miso\build.py Creating recipe ext\socket\build.py ... Done! Checkout SCons for Symbian. We assume it lives in src tree. "c:\Program Files\CollabNet Subversion"\svn checkout http://scons-for-symbian.googlecode.com/svn/trunk/ scons_symbian Running the buildNow we can execute the actual Python build script. This will create one monolithic emulator DLL which has almost all the PyS60 extensions built in – some extensions need manual building, since they rely on headers not found from standard Series 60 SDK. You might need to reset PATH to default to Carbide C++ after the previous mmp2cons step by reopening the console window. c:\Python25\Scripts\scons You should see the following output. As you can see, UIDs are being allocated dynamically as instructed in Scontruct UID_BASE argument. scons: Reading SConscript files ... EPOCROOT=\ Info: SIS creation disabled Building winscw udeb Defines [] Getting dependencies for e32socket.pyd Getting dependencies for _topwindow.pyd Getting dependencies for zlib.pyd Getting dependencies for _locationacq.pyd Getting dependencies for _location.pyd Getting dependencies for _graphics.pyd Getting dependencies for _sysinfo.pyd Getting dependencies for Python222Config.lib Getting dependencies for Python222.dll Getting dependencies for Python_appui.dll Allocated UID:0xE1000000 Getting dependencies for Python.exe Allocated UID:0xE1000001 Getting dependencies for Python_launcher.exe Allocated UID:0xE1000002 scons: done reading SConscript files. scons: Building targets ... ... scons: done building targets. Now you should be able to start a custom built Python shell in the emulator. You should see the following applications in Installation menu: helloworld, btconsole and filebrowser. Try launch helloworld and if it opens a pop up dialog the build has been succesful. Building a SIS file for mobile phonesTo build a target distribution type: scons release=urel compiler=gcce dosis=true This should yield to the result: scons: Building targets ... ensymble(["MyPythonForSymbian.sis"], []) scons: warning: no package version given, using 1.0.0 scons: warning: no certificate given, using insecure built-in one scons: done building targets. If you want to build a SIS file signed with your developer certificate: c:\python25\scripts\scons compiler=gcce release=urel dosis=true simplesis={'--privkey':'C:\\Certificates\\PrivateKeyNoPassphrase.pem','--cert':'C:\\Certificates\\MyApp.cer','--passphrase':''}
For now, installing the SIS file works only for C drive – we’ll fix this little issue soon. If you do not have a symbiansigned.com developer certificate you can sign the SIS file online for one phone (one IMEI code).
Building your own applicationThe purpose of this project is to make rolling out your PyS60 applications possible – so here we go. Currently we make a quite bad assumption that all the application live in the same source tree with PyS60 community edition due to problems with absolute file paths with Symbian build tools – we will figure out a long term solution for this later. PyS60 applications are stub Symbian executables which boostrap Python virtual machine and start the Python code execution. Executables are linked against a custom PyS60 DLL and they are restricted by capabilities given to the EXE file. PyS60 applications live in applications source tree. The source tree comes with Bluetooth console, Hello world and File browser sample applications. Scons build scripts takes as applications parameter a comma separated list which applications are included in the build. scons builtin=all applications=helloworld,filebrowser Applications consist of
See applications\helloworld folder to examine what files are needed to build an application. All application Python modules go to the private application folder (/private/myapplicationuid). Default.py must boostrap PYTHONPATH (sys.path) for this folder – PYTHONPATH defines where Python interpreter looks for the code. Application UIDs can be chosen manually or they are picked automatically by Scons for the unprotected test range. Note that Python Script Shell application is handled out of this flow due to its legacy heritage. Adding in your own extensionIf you have development an PyS60 extension you can drop in into the buildchain easily. Each extension is defined in ext subfolder. It consists of necessary CPP, H and Python files. The building structure is defined in build.py using SConstruct command PyS60Extension(). Build.py files can be automatically generated from legacy code using mmp2scons.py converter.
c:\Python25\python.exe tools\mmp2scons.py ext\calendar\calendar.mmp:60:7: warning: no newline at end of file ext\progressnotes\progressnotes.mmp:38:7: warning: no newline at end of file ext\uikludges\uikludges.mmp:37:7: warning: no newline at end of file Creating recipe ext\\socket\build.py Creating recipe ext\\glcanvas\build.py Creating recipe ext\\graphics\build.py ... ...Done!
c:\Python25\scripts\scons
>> import applicationmanager If your extension is using thread local storage (Dll::Tls()) you might need to figure out how to workaround with it. See socket module for example. You may also need to play around with the init function of the Python extension – it must be init + module name. Developing on targetIf you want to develop your application on a mobile phone, you do not need to go through the full development cycle for every little change. It is possible to update Python files on a phone without SIS installation. You can either automatically synchronize changed files from your PC to Phone (the example below) or you can edit files in-place on the Phone either using PCSuite or Series 60 SMB server. Here are short instructions how to update files using PUTools console (btconsole). PUTools is wxPython based remote Python shell which allows you to run Python console commands over a Bluetooth connection from your PC. PUTools also has a file syncrhonization feature – after editing source code on the PC changes are reflected automatically to the phone.
T:\workspace\pys60\pys60community\src\tools\putools\pcfiles>c:\python25\python.exe putools.py com5 >> import filebrowser >> filebrowser.FileBrowser.run() Release notesHere is the short summary of differences with the current PyS60 community edition and one available from Nokia. This information is also available in divergence.txt file in the source folder. 2008-08-29 Mikko Ohtamaa <mikko@redinnovation.com>
* PyS60 general
New build chain and static config generation
Migration tool for MMP -> Scons based extensions
Added several tools included in the core distribution: sisinfo, ensymble, cog, patch
Patched py2sis tool
Contains extension: applicationmanager
Contains extension: uikludges
Contains extension: progressnotes
Contains extension: miso
Contains application: Bluetooth shell
Contains example applications: filebrowser, helloworld
Changed Bluetooth console bootstrap to e:\startup.py
2008-08-15 Antti Haapala <antti@redinnovation.com>
* e32socketmodule.cpp:
socket.access_points has more information, two new
fields is given per access point: isptype and bearertype,
whose values are integers corresponding to values returned
by CApSelect::Type and CApSelect::BearerType respectively.
No symbolic constants are yet exported.
* appuifwmodule.cpp:
multi_select_list has a new argument, selected, which defaults
to None. Given a list of integers, the items with the given indices
are initially selected.
ConclusionWe hope this helps you to get started with PyS60 community edition. It’s still a bit complicated, since setting up the build environment on Windows is a such a pain. In the future, when the Linux based build system is reading settings up the development environment should be easier – all those boring steps happen automatically. This might be still too difficult for some of the readers, since a lot of prerequirement work must be done before anything useful can be done. Feel free to comment the article in this blog, but we hope that you use Answers section in Launchpad to ask help and technical questions related to PyS60 community edition. Logging with LogManPosted on June 23, 2008 by Jussi ToivolaFiled Under development tools, pys60, symbian This is my first post on our company blog and I thought I’d tell you something about LogMan, which is developer’s utility for getting logging messages from Symbian device over a USB cable. It is written by me and mostly on my own time. I started the project because I had to do a Symbian excercise for university course and I thought I’d do something useful instead of quickly tinkering something small and easy. LogMan supports both C++ and Python. With LogMan, you can send data to the same location from Python and C++ in real-time instead of using log file(s). Browsing through multiple log files can be tedious and you can’t see the debug output while using your application, because on Symbian you must read the file after the debugging session – Symbian cannot share opened files between applications. It is also possible that you create too much log and you run out of Phone internal memory. LogMan helps by removing the use of log files and you never run out of disk space because messages can be stored directly to PC. On simulator, the messages are also sent to RDebug (%TEMP%\epocwind.out). Surprisingly I have not seen a Python module, which would enable use of RDebug. Even though it is very easy to implement. With RDebug, there is no need to open a serial port on simulator for reading the logging messages. Of course, I tried to use RDebug on device first, but I never got it working. I also tried REcmt, which is supported on S60 only and the service just kept on crashing on my phone. This is why I decided to write LogMan. Plus both are closed software, which effectively prevented me from fixing the problems. Just wondering what kind of benefit Symbian or Nokia gets from keeping development tools such as these closed? What is there so secret about them? It didn’t take me very long to write the first working version. *sigh* The use of LogMan is similar to RDebug. There are static class methods, which are a bit slower but easier, and instance methods. Check the project’s homepage for more examples.
I recently added a new feature for LogMan, which allows you to log stack and heap usage of the current thread with one function. When you are unsure about your heap or stack usage, these might come handy. Of course there is some memory used when calling these so take that into consideration. Python can access MemoryInfo only, which logs both stack and heap (Well, I got a bit lazy at that point). There are equivalent macros for these, so check them out from “logmanutils.h”.
logman.HeapInfo(); The output from MemoryInfo is something like this: Browser for PyS60I have been trying, unsuccessfully, to get Browser Control working on PyS60. In a nutshell, CBrCtlInterface wrapper for Python. I have developed it against PyS60Community version in Launchpad. See /src/appui/appuifw/. I have used LogMan extensively to debug the extension so if you want a real example, check out “browsercontrol.cpp”. Browser Control would allow one to embed a browser into his PyS60 application, which would be quite cool. No need to do user interfaces with “appuifw”, which is not very portable. With Browser Control, one could create his user interface with html and javascript, which are a lot more portable indeed. Less work leads to more time. And what is time?… it’s money. Or so I have heard. And being able to handle events with Python instead of C++ is another bonus. Unfortunately, the API is not very stable as you can see by searching for “CBrCtlInterface” at Forum Nokia. The browser worked fine on the simulator with small pages such as “www.google.com”, but it crashed miserably with larger pages. The crash happens in browsercontrol.dll when calling e32.Ao_sleep() in Python. On device it was unable to open any page and crashed instantly when trying to load one. With LogMan, I was able to verify that the crash happened in e32.Ao_sleep() on device also. This reminds me to test on device all the time, which I didn’t do for the first versions. But this is why I added the memory logging feature to LogMan, but it only revealed that I was not out of stack or heap. At least not before the browser started to mess around. I tried to compile the WebKit myself to see what is going on, but the build instructions didn’t work and the build scripts are written in Perl(my eyes started to hurt). What a mess. I don’t wonder anymore if there are bugs in browsercontrol.dll. I finally gave up because my idea pool dried up. Any help getting the wrapper working would be very much appreciated. Plans for LogManI’m planning to add remote shell interface so that you could control your phone from PC. I want access to the file system first. Transfer files, list folders and such. If you have TCB rights ( or hacked phone ) you can speed up development remarkably by simply replacing your binaries in \sys\bin or your Python files with new versions. No need to install sis files and fiddle with certificates and play with memory cards. It would be so nice… I can use 1 day in a week for a personal project so this may happen in near future Building commercial grade mobile applications with Python for Series 60Posted on February 21, 2008 by Mikko OhtamaaFiled Under mobile, pys60
This essay is a post-mortem for creating a commercial grade PyS60 application. By “commercial” I mean something which is publicly available for end users and meets the criteria of Symbian Signed certification. I hope this essay will shed light on the real world challenges, thus aiding Nokia, mobile application vendors and PyS60 community to know when to use PyS60, to strengthen PyS60 as a platform and to guide people make most of it. Description of the project The application is divided in two parts: user interface process and background process. The background process is running all the time, monitoring the phone state and posting changes to the server. The UI process can be launched from the phone menu and it commands the background process. Two processes communicate over localhost TCP/IP sockets. The application communicates with the Internet server by HTTP multipart posts. The background daemon had four threads running: main thread, server communication thread, interprocess communication thread and GPS thread. The core modules of the application were written portability in mind. In fact, one can run the core as standalone Python application on a normal PC. This makes unit testing and debugging much easier. Some supporting libraries and files were ported from the normal Python 2.2.2 distritbution to PyS60. Python for Series 60 and the alternative platforms When we started the project, I was not aware of commercial applications built on PyS60, besides Nokia’s own Mobile Web Server. I knew it was used as in-house in few large software companies. Python has traditionally had strong foothold in the academic world. So it does not come with a surprise that Finnish universities use PyS60 on mobile application software development courses due to Python’s low learning curve. Python itself has been around longer than the most popular programming language, Java. Python’s suitability for the complex development jobs was largely untested until the rise of modern web. The dynamic, agile, nature of Python does not come for free. So called abstraction penalty (interpreting) slows down Python execution speed. It was not until the 00s when processor were fast enough to make this irrelevant matter: software development costs outrun hardware costs and more agile development tools had to be found. When building S60 applications, you need to chose the progamming language between three candidates: Good old fashioned C++, Java Micro Edition (Java ME) and Python for Series 60. Currently, most of Series 60, and Symbian, platform specific applications are written in C++, with a twist. One cannot consider this as “normal” C++ code, but the bastard child of this language. In their quest for the ultimate embedded efficiency (read: the smallest memory footprint) Symbian architectures placed an enormous mental burden on the poor Symbian developers. For example, Symbian C++ has custom pointer schemes, manual error handling, wicked threading and a weird build process. Or to put it bluntly: writing C++ for Symbian is far more difficult than writing C++ for platform X. You cannot really use any standard libraries on Symbian (this is changing, however). Thus, building S60 applications based on C++ technologies is costly, and painful. I have not yet met a happy Symbian C++ developer and I have met hundreds of them. Then, we have Java ME. Sun made a good move by creating a unified mobile application run-time environment. Java ME is the only way to get your mobile application running regardless of the mobile phone vendor. But Java’s infamous slogan “write once, run everywhere” doesn’t hold a candle.Java ME implementations have internal differences and they are buggy as two weeks dead squirrel (seen one squirrel once and followed Sun’s Java ME mailing list long enough). Java ME gets the new features last due to standardization process. Java ME applications are heavily sandboxed and they cannot access half of the phone features. And as the last option is Python for Series 60. Started as a Nokia’s research effort, Nokia brings this agile environment to mobile software development. Python for Series 60 has also a sister project, PyUIQ, which targets Sony-Ericsson phones. PyS60 has had the reputation to be “prototype only” technology and it’s use has been counter-argumeted by “serious development needs C++”. This is slowly changing. Nokia has shown serious commitment to Python. With the launch of Mobile Web Server, Nokia proved that Python is feasible in commercial grade applications. This is how my brain worked when I chose PyS60: Symbian C++ is notorious difficult and no sane person wants to ever touch it. Java ME might have not enough access to the phone features to fulfill the application requirements My client already used Python on embedded Linux and web servers. Why not give PyS60 a shot? Arguments for Python for Series 60 Python is agile
PyS60 is open source
Python is portable
Python is extendable
PyS60 community is vivid
You love it
Arguments against Python for Series 60 PyS60 has not been around for long
Python is for high-end phones only
Symbian is dreadful platform
Lack of entrepreneurship spirit
Conclusion Nokia has done a great deed and effort by creating PyS60 and then giving it out for us to play. Now its own turn to act as a community to give back something we can. Building commercial grade PyS60 applications might not be straightforward yet, but it is possible. Working with PyS60 is definitely more productive than using complex Symbian C++. It is not always straightforward as Java ME, but eventually it will be. Not only that, but PyS60 can surpass Java ME development with all the Python’s agility. If you consider PyS60 for your commercial grade product, it all boils down to the feature set of the application, your existing team skill set and the hacker spirit. And now there exists at least one commercial vendor, besides mighty Nokia, who advertises commercial PyS60 services (hint: our company), so you won’t be alone. We could look things in a larger perspective. In the mobile world, PyS60 is the pack leader of Python-in-your-pocket solutions. Windows CE has Python support available, but it is not officially backed up by Microsoft. This may change quickly when mobile .NET virtual machine gets DLR support making it possible to run IronPython. On Linux platforms, Python is a natural part of the platforms (OLPC, OpenMoko). Google Android and Apple iPhone will roll out their own software development platforms soon. In the dream world, we could have a common mobile Python platform akin to Java ME of today. Java is not the only way to provide vendor neutral mobile applications: there already exists effort to get mobile cross-platform .NET implementation (Mono on embedded Linux, Red Five Labs .NET on Symbian). The future popularity of Linux phones, iPhone and Android might decide the faith of mobile Python. Could we have Python as a batteries-included feature in the future phones? |
PythonLatest Top Shouts | Want to get mobile? |
