About

mFabrik Blog is about mobile and web software development, open source and Linux. We tell exciting tales where business, technology, web and mobile convergence.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License.

Mirroring App Engine production data to development server using appcfg.py

Posted on March 14, 2011

Google App Engine provides some remote API functionality out of the box. One of the remote API features  is to download data from the development server. After downloading, then you can upload the downloaded data to your development server, effectively mirroring the content of the production server to your local development server. This is very useful if you are working CMS, sites, etc. where you want to test new layout or views locally against the old data before putting them to the production.

First enable remote API in app.yaml:

- url: /remote_api
  script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
  login: admin

Note: Using builtins app.yaml directive didn’t work for me some reason, so I had to specify remote API URI manually.

After this you should be able to download data. Here I am using appcfg.py global installation on OSX. Below is the command and sample output.

appcfg.py -e yourgoogleaccount@gmail.com download_data --url=http://yourappid.appspot.com/remote_api --filename=data.sqlite3
...
Downloading data records.
[INFO    ] Logging to bulkloader-log-20110313.222523
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
...
[INFO    ] Have 1803 entities, 0 previously transferred
[INFO    ] 1803 entities (972883 bytes) transferred in 91.0 seconds

data.sqlite3 is your production database dump in SQLite 3 binary format (used internally by the development server).

If you have sqlite command line tool installed you can explore around the data dump there:

sqlite3 data.sqlite
SQLite version 3.7.5
Enter ".help" for instructions
Enter SQL statements terminated with a ";"

sqlite> .tables
Apps                                   your-app!Model1!Entities
IdSeq                                  your-app!Model1!EntitiesByProperty
Namespaces                             your-app!Model2!Entities
bulkloader_database_signature          your-app!Model2!EntitiesByProperty
your-app!!Entities                     result
your-app!!EntitiesByProperty

Now you can upload data.

Note: Even though there exists option –use_sqlite for dev_appserver.py looks like it cannot directly use the database file produced by download_data. You cannot just swap database files, you need upload the downloaded data to the development server.

Start your development server:

dev_appserver.py .

In another terminal, go to downloaded data.sqlite folder and give the command:

appcfg.py upload_data --url http://localhost:8080/remote_api --file=data.sqlite --application=yourappid

It will ask you for credentials, but it seems that any username and password is accepted for the local development server.

Now you can login to your local development server to explore the data:

http://localhost:8080/_ah/admin

Ensure your data got copied over using Data Viewer:

http://localhost:8080/_ah/admin

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Cross-platform mobile application development and payments

Posted on September 30, 2009

We have been piloting multi-platform mobile application development and payments in few client projects. Target platforms usually include iPhone, Android, Blackberry and Nokia Series 60. Also there are two notable usual cases which need to be specially handled

  • Image uploads
  • Payments for subscribed content

Sounds easy, right? Well it isn’t… Below are some notes for our due diligence work which you fellow developers might find interesting.

SDKs

Mobile phone vendors are jealously and don’t want to co-operate with each other. Building application which works in all handsets is major headache.

We found some reasonable candidates for cross-platform mobile development doing HTML and Javascript. HTML and Javascript pages are converted to native application using a wrapper technology (a.k.a. appaccelerator). Doing Flash Lite or Java ME can be pretty much forgotten nowadays as they won’t run on the most hyped platform, iPhone. Flash Lite has poor support for anything except content authoring due to primitive and limited APIs. Java ME provides horrible user experience.

(X)HTML is the only common language spoken by mobile phones. Thus, there has been a rise of “appaccelerators”, technologies which allow to create mobile applications with HTML(5) and Javascript.

  • Phonegap: iPhone, Android, Blackberry and possibly S60 in the future. Pluses: BSD license, very active community. Minuses: bad documentation, difficult deployment process.
  • Titanium: iPhone, Android. Pluses: Professional, Apache license. Minuses: Too tightly coupled with Appacclerator Inc. company.
  • Rhomobile: iPhone, Android, Blackberry, S60, Windows Mobile. Pluses: Professional, tries to build open source community, the widest platform support. Minuses: Dual licensing and tightly coupled with Rhomobile Inc.
  • Nokia Web-runtime: Nokia S60 and some other Symbian based phones. Pluses: Professional, good documentation. Minuses: Not open source, impossible to extend, Nokia has little interest to make this cross-platform, Nokia doesn’t like updating old models and web-runtime is useable only in the latest S60 5th edition models.
  • Palm Pre supports web applications natively. However Palm Pre application business is still taking a shape.

All these wrap the browser component (WebKit) and provide some extra Javascript APIs when your web pages as executed under the application mode.

  • Locationing
  • Contacts
  • SMS
  • Client-side database
  • and so on…

Rhomobile has little different use cases  from the rest of the bunch as it provides client-side programming using Ruby and less focuses on Javascript/web applications.

Payments and in-application purchases

There are four major way to do mobile payments “inside” the application for bought content and subscriptions. The price tag on the application itself is left out on this discussion as the application stores themselves take care of it.

  • Credit card
  • SMS
  • App Store payment (thus far Apple only)
  • Direct operator payments – you have a service provider (Bango) which can directly charge items to the operator phone bill based on handset identification.

App Store payment is the most attractive as it provides the best end user experience.  It allows you to use App Store payment mechanism inside the application. It is safe and no need to hassle with external payment providers. However, App Store payment can be used only for content consumed directly inside the application. You cannot use it e.g. for ordering a pizza. I think this might be related to recent EU legislation forbidding SMS payments for services not consumed in the phone itself.

SMS payment is ok for little payments. Operators take big cut of the revenue, generally 30% – 70% depending on the country. Short code fees usually start from 500€ set-up fee + 500€ / month. SMS cannot be often send as a background, but the user is presented the normal SMS editor which reduces the user experience somehow.

For credit card payments there exists several providers. Credit card has the cheapest entry fees, but the downside is that the user needs to have the credit card. This excludes teenager audience.

Direct operator payments are not very well supported yet globally. Most western operators support them. The operator also takes a big share and the fixed fee is pretty high.

My favorite payment provider thus far is Bango which provides credit card payment starting 9€ / mo. and scales up to worldwide SMS payments which cost few grannies per month.

In most cases, the payment experience will not be smooth. You need to open the phone main browser on the payment provider page to do the payment. This usually will close your own application. Rarely you can do the payment inside the application and support multiple platforms. After doing the payment most platforms allow you to close the browse and reopen your application using a special URL handler.

Wikipedia mobile payments page is also useful.

Image upload

<input type=”file”> won’t work on iPhone and some other platforms as those don’t have user browsable file system. Also the file dialog usually doens’t have image preview making it useless.

Phonegap has a branch which supports images picking using iPhone’s own gallery browser.

In any case, there is not yet cross-platform solution for this.

Future prospects

In some time-frame we will get rid of the need to wrap HTML applications natively as the web browser applications will support all HTML5 features without extensions and probably have some proprietary extensions for mobile specific features like SMS. We already have had some taste for this: