Logging with LogMan

This is my first post on our company blog and I thought I’d tell you something about LogMan, which is developer’s utility for getting logging messages from Symbian device over a USB cable. It is written by me and mostly on my own time. I started the project because I had to do a Symbian excercise for university course and I thought I’d do something useful instead of quickly tinkering something small and easy.

LogMan supports both C++ and Python. With LogMan, you can send data to the same location from Python and C++ in real-time instead of using log file(s). Browsing through multiple log files can be tedious and you can’t see the debug output while using your application, because on Symbian you must read the file after the debugging session – Symbian cannot share opened files between applications. It is also possible that you create too much log and you run out of Phone internal memory. LogMan helps by removing the use of log files and you never run out of disk space because messages can be stored directly to PC. On simulator, the messages are also sent to RDebug (%TEMP%\epocwind.out). Surprisingly I have not seen a Python module, which would enable use of RDebug. Even though it is very easy to implement. With RDebug, there is no need to open a serial port on simulator for reading the logging messages.

Of course, I tried to use RDebug on device first, but I never got it working. I also tried REcmt, which is supported on S60 only and the service just kept on crashing on my phone. This is why I decided to write LogMan. Plus both are closed software, which effectively prevented me from fixing the problems.

Just wondering what kind of benefit Symbian or Nokia gets from keeping development tools such as these closed? What is there so secret about them? It didn’t take me very long to write the first working version. *sigh*

The use of LogMan is similar to RDebug. There are static class methods, which are a bit slower but easier, and instance methods. Check the project’s homepage for more examples.


#include "LogMan.h" //RLogMan
RLogMan::Log( _L("Hello world ") );


I recently added a new feature for LogMan, which allows you to log stack and heap usage of the current thread with one function. When you are unsure about your heap or stack usage, these might come handy. Of course there is some memory used when calling these so take that into consideration. Python can access MemoryInfo only, which logs both stack and heap (Well, I got a bit lazy at that point). There are equivalent macros for these, so check them out from “logmanutils.h”.


// Store this as a member of your class, for example
RLogMan logman;

logman.HeapInfo();
logman.StackInfo();
logman.MemoryInfo(); // Both stack and heap
logman.Close();

The output from MemoryInfo is something like this:

StackInfo
Free:1039000, Used:9576, Size:1048576
HeapInfo
Free:25856, Used:101004, Size:126860


Browser for PyS60

I have been trying, unsuccessfully, to get Browser Control working on PyS60. In a nutshell, CBrCtlInterface wrapper for Python. I have developed it against PyS60Community version in Launchpad. See /src/appui/appuifw/. I have used LogMan extensively to debug the extension so if you want a real example, check out “browsercontrol.cpp”.

Browser Control would allow one to embed a browser into his PyS60 application, which would be quite cool. No need to do user interfaces with “appuifw”, which is not very portable. With Browser Control, one could create his user interface with html and javascript, which are a lot more portable indeed. Less work leads to more time. And what is time?… it’s money. Or so I have heard. And being able to handle events with Python instead of C++ is another bonus.

Unfortunately, the API is not very stable as you can see by searching for “CBrCtlInterface” at Forum Nokia. The browser worked fine on the simulator with small pages such as “www.google.com”, but it crashed miserably with larger pages. The crash happens in browsercontrol.dll when calling e32.Ao_sleep() in Python. On device it was unable to open any page and crashed instantly when trying to load one. With LogMan, I was able to verify that the crash happened in e32.Ao_sleep() on device also. This reminds me to test on device all the time, which I didn’t do for the first versions. But this is why I added the memory logging feature to LogMan, but it only revealed that I was not out of stack or heap. At least not before the browser started to mess around.

I tried to compile the WebKit myself to see what is going on, but the build instructions didn’t work and the build scripts are written in Perl(my eyes started to hurt). What a mess. I don’t wonder anymore if there are bugs in browsercontrol.dll. I finally gave up because my idea pool dried up. Any help getting the wrapper working would be very much appreciated.

Plans for LogMan

I’m planning to add remote shell interface so that you could control your phone from PC. I want access to the file system first. Transfer files, list folders and such. If you have TCB rights ( or hacked phone ) you can speed up development remarkably by simply replacing your binaries in \sys\bin or your Python files with new versions. No need to install sis files and fiddle with certificates and play with memory cards. It would be so nice… I can use 1 day in a week for a personal project so this may happen in near future

Building commercial grade mobile applications with Python for Series 60

Python programming language, created in 1991 by Guido van Rossum, is the sixth most popular programming language and “the programming language of year 2007“. Nokia Series 60 is a Symbian-based mobile phone platform for high end smart phones which is the most widespread smart phone platform. Python for Series 60 is Python ported to S60. PyS60 is open source and there already exists a notable community around it – a sign of a successful software platform.

This essay is a post-mortem for creating a commercial grade PyS60 application. By “commercial” I mean something which is publicly available for end users and meets the criteria of Symbian Signed certification. I hope this essay will shed light on the real world challenges, thus aiding Nokia, mobile application vendors and PyS60 community to know when to use PyS60, to strengthen PyS60 as a platform and to guide people make most of it.

Description of the project
I was given a task to create a mobile phone client for a web service. The client runs on the the background, posting information from the phone (files, location) to the web site. Two developers were assigned for the job. In the end of the project, almost 100 Python modules and classes had been written (just for the mobile side of the things). I believe this is the most complex PyS60 development effort this far.

The application is divided in two parts: user interface process and background process. The background process is running all the time, monitoring the phone state and posting changes to the server. The UI process can be launched from the phone menu and it commands the background process. Two processes communicate over localhost TCP/IP sockets. The application communicates with the Internet server by HTTP multipart posts. The background daemon had four threads running: main thread, server communication thread, interprocess communication thread and GPS thread.

The core modules of the application were written portability in mind. In fact, one can run the core as standalone Python application on a normal PC. This makes unit testing and debugging much easier. Some supporting libraries and files were ported from the normal Python 2.2.2 distritbution to PyS60.

Python for Series 60 and the alternative platforms

When we started the project, I was not aware of commercial applications built on PyS60, besides Nokia’s own Mobile Web Server. I knew it was used as in-house in few large software companies. Python has traditionally had strong foothold in the academic world. So it does not come with a surprise that Finnish universities use PyS60 on mobile application software development courses due to Python’s low learning curve.

Python itself has been around longer than the most popular programming language, Java. Python’s suitability for the complex development jobs was largely untested until the rise of modern web. The dynamic, agile, nature of Python does not come for free. So called abstraction penalty (interpreting) slows down Python execution speed. It was not until the 00s when processor were fast enough to make this irrelevant matter: software development costs outrun hardware costs and more agile development tools had to be found.

When building S60 applications, you need to chose the progamming language between three candidates: Good old fashioned C++, Java Micro Edition (Java ME) and Python for Series 60.

Currently, most of Series 60, and Symbian, platform specific applications are written in C++, with a twist. One cannot consider this as “normal” C++ code, but the bastard child of this language. In their quest for the ultimate embedded efficiency (read: the smallest memory footprint) Symbian architectures placed an enormous mental burden on the poor Symbian developers. For example, Symbian C++ has custom pointer schemes, manual error handling, wicked threading and a weird build process. Or to put it bluntly: writing C++ for Symbian is far more difficult than writing C++ for platform X. You cannot really use any standard libraries on Symbian (this is changing, however). Thus, building S60 applications based on C++ technologies is costly, and painful. I have not yet met a happy Symbian C++ developer and I have met hundreds of them.

Then, we have Java ME. Sun made a good move by creating a unified mobile application run-time environment. Java ME is the only way to get your mobile application running regardless of the mobile phone vendor. But Java’s infamous slogan “write once, run everywhere” doesn’t hold a candle.Java ME implementations have internal differences and they are buggy as two weeks dead squirrel (seen one squirrel once and followed Sun’s Java ME mailing list long enough). Java ME gets the new features last due to standardization process. Java ME applications are heavily sandboxed and they cannot access half of the phone features.

And as the last option is Python for Series 60. Started as a Nokia’s research effort, Nokia brings this agile environment to mobile software development. Python for Series 60 has also a sister project, PyUIQ, which targets Sony-Ericsson phones.

PyS60 has had the reputation to be “prototype only” technology and it’s use has been counter-argumeted by “serious development needs C++”. This is slowly changing. Nokia has shown serious commitment to Python. With the launch of Mobile Web Server, Nokia proved that Python is feasible in commercial grade applications.

This is how my brain worked when I chose PyS60: Symbian C++ is notorious difficult and no sane person wants to ever touch it. Java ME might have not enough access to the phone features to fulfill the application requirements My client already used Python on embedded Linux and web servers. Why not give PyS60 a shot?

Arguments for Python for Series 60

Python is agile

This is maybe the most important argument to pick PyS60. Python tends to reduce development time compared to C++ or Java. The language itself is very easy to learn. Syntax is expressive and developers can achieve much more functionality in less lines of codes: You can focus on getting the task itself done. With higher level concepts and less hassle with lower level things, changes to introduce bugs radically decreases.

Python development does not have traditional compiling phase, though byte code files can be built for the final release to obfuscate the code and optimize loading times. Changes made into Python code are immediately (in seconds) visible in the running application. Even if the application runs on the target device, since code changes can be automatically synchronized over a Bluetooth connection. No wonder Python enjoys legendary fast prototyping language fame – the developers have less time to surf in web when they don’t need to wait compilers to bake yet another test build.

PyS60 is open source

I assume you are familiar with the benefits of open source already: vendor freedom, no licensing cost, not being mercy on others and so on. This all holds true for PyS60 as well.

PyS60 license (Python license) does not place restrictions for creating closed source software. You can have very good development tools for free: Series 60 SDK, Eclipse IDE with PyDev plug-in and PUTools bluetooth synchronization kit for free. This makes the barrier to enter PyS60 development very low.

Since PyS60 is fully available in the source code form, you can customize and fix it for your needs. The traditional horror story from Java ME development is that your application won’t run in the mobile phone model X, since there is a bug in the vendor’s Java Virtual Machine implementation. Unless you are A Very Big Player, don’t ever think about going to talk with the vendor and ask it to fix its bug – hitting one of these issues might prevent you to release product at all. With PyS60 you can control all aspects of the software execution up to the closed parts of the device. No more being mercy on others!

Being open is pop now. Embracing open source gives much for your company: good citizen reputation, hiring pool and free marketing.

Python is portable

Python runs on Linux, Windows mobile and Symbian – Python runs on almost every smart phone known to man. And not just the phones, it runs on every computer as well: Windows, OSX and Linux server and desktops. Moving your code between these environments is easy. Maybe there isn’t yet “cross-platform Python SDK for mobile phones”, but it might be just a matter of time when something leveraging the idea comes out.

In our project, core files, independent from graphical user interface, were built and unit tested on Linux desktop environment. No emulator was needed. It was not until the late development stages when they had to be moved to Symbian environment. Since you can develop and debug your code in such “easy” environment, you can work more effectively.

Because the full Python is available in the phone and it can run the exactly same modules as your servers, sharing code between these two environments opens possibility for making “smart” clients. Think about the situation where your phone is offline. It can have a subset of the full database loaded on a memory card. Since moving Python code from the server to the mobile is easy, the mobile application can have the same database functions and application logic modules loaded with it. Without the extra duplication of the logic code, one can operate on the small subset of the data and later synchronize it with the main database.

Python is extendable

You can mix C++ and Python. When you run out of ready components or you need unleash all available CPU cycles for CPU sensitive calculation tasks, you can write your own native extensions (DLL files) for PyS60 in Symbian C++. They enjoy all the same privileges as any native built code or application. Actually standalone PyS60 applications are just normal Symbian executables which just happen to have Python interpreter started inside them.

If you are working with phone features which are not yet very common and lack working Java ME implementations, say acceleration sensors, instead of creating a native C++ application you can create a Python application. You need to mix-in low level C++ only for the the critical part of the applications which operate with the native platform API.

PyS60 community is vivid

Surf to forum.nokia.com Python discussion boards and you can witness the activity around PyS60. A lot of folks are answering the beginners’ questions and guiding them. If your case is not totally unique, you can expect help to be found. As a matter of fact, you, reading this text, is a sole proof of powerful community. I was not directly paid to write this and I am not affiliated with Nokia any way – this writing flows from the bottom of my heart to promote PyS60 and open some window for the (hireable) talents of our company.

You love it

For some things, we just feel impulsive attachment. They are beautiful, aesthetic, sexy. When you see it, you need to have it. PyS60 is one of these things.

Arguments against Python for Series 60

PyS60 has not been around for long

PyS60 is a young project. It still has some maturity issues. If you don’t do preparations well and test that all needed modules work as you want them to work, you might need to do some extra work to get around the issues.

Especially PyS60 user interface lacks some crucial components and polish which may force you to create UI components yourself.

By “getting around” the issues I mean leaping to the native Symbian code world and writing embedded C++. Since this leap is all but a little step, in the terms of development skills and complexity, I recommend preparing for the worst before the roof is falling on your head. I have seen too much “plz reply me asap urgent my boss kills me this afternoon” PyS60 discussion board messages from persons who haven’t done their homework. For example, you could make sure that you have some hired Symbian guns available in the case your own effort was not enough.

Python is for high-end phones only

PyS60 and PyUIQ are available for the high end smart phones only. Nokia’s Series 40 devices, and other devices built around in-house operating systems, cannot run Python. If you are targetting to e.g. consumers markets and Java ME has all the required capabilities to implement the application, it is the only real way to go.

Symbian is dreadful platform

Symbian, as a software development platform, is very challenging. Due to its history, the feeling of developing for Symbian differs greatly from other mobile, desktop and server platforms. Things simple elsewhere, like installing an application, might become a development nightmare on Symbian where one has to fight his/her way through capability limitations, artificial device restrictions and signing process just to get the application to the phone.

PyS60 naturally inherits all these challenges. Java ME goes around the issue by having its own custom installation, signing, etc. schemes. On Symbian, even accessing some of the device features – a device you own and hold in your hand – requires you to throw in few hundred bucks cash to get the publisher id – to get the developer certificate – to sign the application – to have all required capabilities – to access the specific features (read: GPS). More cash and third party testing is needed to publish the application. This is the price of virus free DRM’ed platform.

Lack of entrepreneurship spirit

If your company states “we are a Java house” or “we are a C++ house” and you miss courage to look for alternatives, then maybe you shouldn’t. Sailing into the brave new world takes more resources than staying near your home harbor. On the other hand, if you are a technology company, it’s quite important follow technologies outside your mainstream, unless you want to suffer the extinction when the new kids on the block gain power (remember the day when COBOL died).

Conclusion

Nokia has done a great deed and effort by creating PyS60 and then giving it out for us to play. Now its own turn to act as a community to give back something we can.

Building commercial grade PyS60 applications might not be straightforward yet, but it is possible. Working with PyS60 is definitely more productive than using complex Symbian C++. It is not always straightforward as Java ME, but eventually it will be. Not only that, but PyS60 can surpass Java ME development with all the Python’s agility.

If you consider PyS60 for your commercial grade product, it all boils down to the feature set of the application, your existing team skill set and the hacker spirit. And now there exists at least one commercial vendor, besides mighty Nokia, who advertises commercial PyS60 services (hint: our company), so you won’t be alone.

We could look things in a larger perspective. In the mobile world, PyS60 is the pack leader of Python-in-your-pocket solutions. Windows CE has Python support available, but it is not officially backed up by Microsoft. This may change quickly when mobile .NET virtual machine gets DLR support making it possible to run IronPython. On Linux platforms, Python is a natural part of the platforms (OLPC, OpenMoko). Google Android and Apple iPhone will roll out their own software development platforms soon.

In the dream world, we could have a common mobile Python platform akin to Java ME of today. Java is not the only way to provide vendor neutral mobile applications: there already exists effort to get mobile cross-platform .NET implementation (Mono on embedded Linux, Red Five Labs .NET on Symbian). The future popularity of Linux phones, iPhone and Android might decide the faith of mobile Python. Could we have Python as a batteries-included feature in the future phones?