About

mFabrik Blog is about mobile and web software development, open source and Linux. We tell exciting tales where business, technology, web and mobile convergence.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License.

Mirroring App Engine production data to development server using appcfg.py

Posted on March 14, 2011

Google App Engine provides some remote API functionality out of the box. One of the remote API features  is to download data from the development server. After downloading, then you can upload the downloaded data to your development server, effectively mirroring the content of the production server to your local development server. This is very useful if you are working CMS, sites, etc. where you want to test new layout or views locally against the old data before putting them to the production.

First enable remote API in app.yaml:

- url: /remote_api
  script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
  login: admin

Note: Using builtins app.yaml directive didn’t work for me some reason, so I had to specify remote API URI manually.

After this you should be able to download data. Here I am using appcfg.py global installation on OSX. Below is the command and sample output.

appcfg.py -e yourgoogleaccount@gmail.com download_data --url=http://yourappid.appspot.com/remote_api --filename=data.sqlite3
...
Downloading data records.
[INFO    ] Logging to bulkloader-log-20110313.222523
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
...
[INFO    ] Have 1803 entities, 0 previously transferred
[INFO    ] 1803 entities (972883 bytes) transferred in 91.0 seconds

data.sqlite3 is your production database dump in SQLite 3 binary format (used internally by the development server).

If you have sqlite command line tool installed you can explore around the data dump there:

sqlite3 data.sqlite
SQLite version 3.7.5
Enter ".help" for instructions
Enter SQL statements terminated with a ";"

sqlite> .tables
Apps                                   your-app!Model1!Entities
IdSeq                                  your-app!Model1!EntitiesByProperty
Namespaces                             your-app!Model2!Entities
bulkloader_database_signature          your-app!Model2!EntitiesByProperty
your-app!!Entities                     result
your-app!!EntitiesByProperty

Now you can upload data.

Note: Even though there exists option –use_sqlite for dev_appserver.py looks like it cannot directly use the database file produced by download_data. You cannot just swap database files, you need upload the downloaded data to the development server.

Start your development server:

dev_appserver.py .

In another terminal, go to downloaded data.sqlite folder and give the command:

appcfg.py upload_data --url http://localhost:8080/remote_api --file=data.sqlite --application=yourappid

It will ask you for credentials, but it seems that any username and password is accepted for the local development server.

Now you can login to your local development server to explore the data:

http://localhost:8080/_ah/admin

Ensure your data got copied over using Data Viewer:

http://localhost:8080/_ah/admin

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Flex 3 SQLite and the mystery of “text value to numeric value”

Posted on March 14, 2008

Learning new development environments and languages can be quite fun, but sometimes it turns out just plain frustrating. Especially so if you are trying to learn something that is still in its beta stage. This time it was Adobe AIR and Flex 3.

The program in question required storing date and time values in an sqlite3 database. Now, as the title of this entry says, I should’ve just read the AIR sqlite handling manual 1 more time (or 2-10 more times) and I could’ve learnt that AIR SQL handling accepts “Date” as column type and handles inserting and getting Date objects from the database transparently. I did not notice this while studying and used “TIMESTAMP” as column type instead. This is where things went wrong… so wrong…

Using “TIMESTAMP” column type and inserting datetime strings worked flawlessly in the betas along with using sqlite date functions. But AIR 1.0 release came and suddenly my application started producing the following SQL errors everywhere:

SQLError: 'Error #3115: SQL Error.', details:'could not convert text value to numeric value.', operation:'execute'

Naturally for a few days I went through my SQL code and tables trying different things on columns that had types like “INTEGER”, “REAL”, “NUMERIC” and so on. After trying and trying I finally decided to read the AIR documentation again. The essential part turned out to be how AIR handles the column affinities in the release version: everything not recognized by AIR is given “NUMERIC” column affinity. This and the fact that AIR is now stricter about types is what changed between the betas and release version and broke the columns with “TIMESTAMP” as type since I was trying to insert text.

So, again, I should’ve just rtfm…