The AndroidHttpClient module offers a way to achieve this, when you’re sending JSON requests you can gzip the content of the request params and add some necessary HTTP headers, which will provide a hint to your server that it has to decode the content before it is usable. Needless to say that this technique is not just limited to sending JSON, you can of course use it if you’re sending CSV or text file over the network to server.

I’ll show a way how this can be done using a Rails backend (that’s what I use), but I presume this should not be too difficult to implement in case you’re not using Rails. I’ll be using JSON as an example here:

To encode or decode the JSON content you can use the following utility module in Android:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

public class Util {

    /**
     * Converts an InputStream to String
     * 
     * @param is
     * @return
     * @throws IOException
     */
    public static String streamToString(InputStream content) throws IOException {
        byte[] buffer = new byte[1024];
        int numRead = 0;
        ByteArrayOutputStream baos = new ByteArrayOutputStream();

        while ((numRead = content.read(buffer)) != -1) {
            baos.write(buffer, 0, numRead);
        }

        content.close();

        return new String(baos.toByteArray());
    }

    /**
     * Compresses the content of the request parameters (as a string). Sets
     * appropriate HTTP headers also so that the server can decode it properly.
     * 
     * @param context Context
     * @param content The string request params, ideally JSON string
     * @param postReq The HttpPost request object
     * 
     */
    public static void setCompressedEntity(Context context, String content, HttpPost postReq) {
        try {
            byte[] data = content.getBytes("UTF-8");

            // if the length of the data exceeds the minimum gzip size then only
            // gzip it else it's not required at all
            if (content.length() > AndroidHttpClient
                  .getMinGzipSize(context.getContentResolver())) {
                // set necessary headers
                postReq.setHeader("Content-Encoding", "gzip");
            }

            // Compressed entity itself checks for minimum gzip size
            // and if the content is shorter than that size then it
            // just returns a ByteArrayEntity
            postReq.setEntity(AndroidHttpClient.getCompressedEntity(data, context.getContentResolver()));

        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    /**
     * Extracts the response content. If the server response is compressed, then
     * it transparently decompresses the content. In order to indicate to server
     * that you can consume JSON response, use the following code to add the "Accept"
     * header:
     *
     * AndroidHttpClient.modifyRequestToAcceptGzipResponse(HttpRequest request)
     * 
     * @param response
     *                   HttpResponse Object
     * @return String content of the HttpResponse
     */
    public static String getIfCompressed(HttpResponse response) {
        if (response == null)
            return null;

        try {
            InputStream is = AndroidHttpClient.getUngzippedContent(response.getEntity());
            return streamToString(is);
        } catch (IOException e) {
            e.printStackTrace();
        }

        return null;
    }
}

The HTTP header, which indicates to server that the request content is gzipped, is Content-Encoding: gzip. On the Rails side you can use the following Rack middleware to decode JSON requests (you can set your webserver for example nginx to do the encoding on JSON responses):

A Rack middleware to decode the gzipped JSON, thanks to this gist by relistan:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

class CompressedRequests
  def initialize(app)
    @app = app
  end

  def method_handled?(env)
    !!(env['REQUEST_METHOD'] =~ /(POST|PUT)/)
  end

  def encoding_handled?(env)
    ['gzip', 'deflate'].include? env['HTTP_CONTENT_ENCODING']
  end

  def call(env)
    if method_handled?(env) && encoding_handled?(env)
      extracted = decode(env['rack.input'], env['HTTP_CONTENT_ENCODING'])

      env.delete('HTTP_CONTENT_ENCODING')
      env['CONTENT_LENGTH'] = extracted.length
      env['rack.input'] = StringIO.new(extracted)
    end

    status, headers, response = @app.call(env)
    return [status, headers, response]
  end

  def decode(input, content_encoding)
    case content_encoding
      when 'gzip' then Zlib::GzipReader.new(input).read
      when 'deflate' then Zlib::Inflate.inflate(input.read)
    end
  end
end

Now put this in somewhere in your Rails web application directory for example lib/middleware, just make sure that the file is loaded when Rails boots. To include the file in Rails’ autoload path, add it like this:

1

config.autoload_paths += %W(#{config.root}/lib #{config.root}/lib/middleware)

Once that is done, then you need to add this tiny little Rack app as a middleware, and the important trick is to add it before the ActionDispatch::ParamsParser middleware in Rails 3:

1
2
3

# Handle Compressed Requests, this middleware makes gzip content 
# handling transparent to the Rails stack
config.middleware.insert_before ActionDispatch::ParamsParser, "CompressedRequests"

That’s all you need both on client and server to send and receive Gzipped content. If you’re sending JSON from client and you wan’t rails to interpret it as JSON, then don’t forget to add Content-Type: application/json on your HTTP Request header.

Happy coding! If you have anything to say feel free to contact me on Twitter @arnabc

The term remotely managed means that the app will be automatically updated over the air, there will be a provisioning server where the admins can publish/upload a new build, and that build will automatically be installed in the devices. Other than that the device can also be remotely restarted or shut down. The content that users can access or play using the app is stored in the device for faster access, and that content is synced (i.e. updated) periodically from a remote content syncing server. All of these are happening over the internet and the devices are equipped with a 3G chip, which makes it easier to download large amount of data with a decent network speed, else the content syncing would be a real pain. Below is the list of rough requirements that we had:

• The main app will be the only app which users can access.
• Automatic update over the air using a provisioning server.
• Sync content from a remote content syncing server.
• Report Crashes and device health statuses to the server.

If you look at the requirements above, this ain’t an easy task and sure it’s not, we’re developing the app since last few months and are close to completion, the app is currently being field tested and hopefully it’ll soon be live out there in the wild.

So the big question is what are the challenges that we faced while we pulled off this effort, the purpose of this blog post is to share some of the findings/knowledge that we’ve gained over the period of time while developing the app. The ways that we’ve adopted are not the ones which can be adopted if you’re going to have your app be distributed through the Android market, our app isn’t or will not be distributed from the Android market, this is a custom application developed for a very specific/limited set of devices in mind. I’ll cover each of the requirements mentioned above and will share some of the rough design choices that we made in order to meet the requirements.

The main app will be the only app which users can access

There’s only one way that you can have this kind of application in Android, make your app a homescreen app, so that whenever users press home/back button they’ll land up right there in your app, and they can’t get out of it (provided your app is the default one). In order to prevent users from exiting your app, you may need to handle some other stuff like options menu, disable recent items, and may be some extra buttons (some Chinese devices come with extra buttons on 7″ Android tablets) etc. These extra keys can be handled/prevented using Android key handling code.

Automatic update over the air using a provisioning server

This is an interesting part, you may be aware of that in Android the automatic update of an app can only happen through Marketplace, there’s no other way to update an app. But our app is not going to be distributed through Marketplace, so how do we update then? Well there’s no easy way to do it, and the process is complicated.

First of all, in Android in order to silently install an app, you need special permission called INSTALL_PACKAGES which is a system permission and IS NOT available to apps not signed with system/firmware certificate. This is an issue that we need to get past in order to make it happen, after few days of searching and testing several options, I narrowed it down to the following options:

1. Sign the app with the system certificate - This was not possible because we couldn’t procure the certificate from the hardware vendor of our choice.
2. Install our own Android custom ROM with the app bundled - This was possible, but it requires more engineering effort as well as taking care of device drivers that manufacturers ship with the device. Other than that there’s a maintenance effort involved in future where we have to ensure that our custom Android ROM works with at least two to three different devices. So, this too was not a viable solution for us.
3. Root the device and install the package through code - This was the last option that I tried and IT WORKED!, after couple of iterations on rooting procedure, I managed to root the device properly.

At first I tried the rageagainstthecage exploit to root, but it were only able to temporarily root the device and it wouldn’t survive a reboot. Then I used psneuter to root the device and installed busybox and the modified su binary provided by the excellent guys from SuperOneClick installer. Along with that I also installed the Superuser app to manage the su permissions. Once the rooting procedure is in place it’s time to move on to code and write an Android service which can automatically install our application.

Updater Service

This service is the crux of this application, it acts as a “remote control” to manage the device as well as our homescreen application. We decided to build it as a separate app not part of the homescreen application, because this app is not going to have any GUI. So how does this work? Below is the rough diagram of it’s architecture:

In the above diagram, our Updater app is started whenever the device is rebooted (as per our installation process, we reboot the device after installing the update service app), it sets up a repeating alarm using a PendingIntent, which is invoked after a certain period of time, in the mean time it also makes a quick request to the provisioning server for any available update. If there’s an update then the UpdateService downloads the APK (link provided in the update check response) and once it’s downloaded (stored in the SDcard) it then fires up an intent to start the PackageInstaller and gives the APK file path as part of the intent payload. The PackageInstaller then runs the pm install -r <apkPath> command with su privilege. In order to execute the commands as root I used a modified version of the excellent piece of code available from here. Once the installation process is complete we notify the backend server that the device has been updated with the latest app (by the way it may already be clear to you that the provisioning server has the list of devices running in the wild and knows which device is running what version of the app). Anyway, this cycle of checking for updates continues as every time the alarm fires or the device is rebooted.

Although the entire procedure went smoothly, it wasn’t without a hitch, as I mentioned that our app is a homescreen application and in Android when you have multiple homescreen apps, every time you click on the home button you’re presented with a list of homescreen apps to choose from. So in our case we have two homescreen apps one is the default Android Launcher and the other is our homescreen app, as part of the installation procedure we set that app up to be the default one. But after update that settings was getting reset and pressing on the home button again was displaying that list of homescreen apps to choose from. We tried to solve this through code by setting our own Activity as the default homescreen app using the Android API addPreferredActivty() but that didn’t work because it has been removed/disabled by Google from Android 2.2 Froyo onwards (it was available till 2.1), the device that we’re using has the Android version 2.2.2. This was almost a dead end and that time we came up with a very interesting hack, and the trick is very simple just rename the launcher APK, ya that’s it just rename it and it’s gone :-). So here’s the steps that we have:

Before Install Steps

1. Mount the /system folder as RW volume using busybox, by default it’s readonly.
2. Restore(rename) the original lancher apk back to “Launcher.apk”
3. Unmount the /system back to it’s original ReadOnly mode.

Example commands:

1
2
3

$ busybox mount -o rw,remount /system
$ mv /system/app/Launcher.tmp.apk /system/app/Launcher.apk
$ busybox mount -o ro,remount /system

After Install Steps

1. Mount the /system folder as RW volume using busybox, by default it’s readonly.
2. Rename the original “Launcher.apk” to “Launcher.tmp.apk”
3. Unmount the /system back to it’s original ReadOnly mode.

Example commands:

1
2
3

$ busybox mount -o rw,remount /system
$ mv /system/app/Launcher.apk /system/app/Launcher.tmp.apk
$ busybox mount -o ro,remount /system

Note: In some devices the Launcher apk may be named as Launcher2.apk.

Sync content from a remote content syncing server

Similarly to the update service above, the Content Syncing part is also a separate app without a GUI, it gets started on RECEIVE_BOOT_COMPLETED intent and checks for updated content once a day. We download a pretty large amount of data which can range anywhere from 100MB-1GB. Below is the sequence diagram of the rough architecture that we have:

Once the download sync is complete, the app sends an Intent notifying the Main homescreen application about the content update so that it can display a friendly message to the user in case the device content is being used/played at that time. The main app stores some of the data in memory or in a SQLite database when it starts, so in order to cleanly rebuild that entire data structure we’re currently rebooting the device. But in future we’re planning to avoid it, so that the content update does not need a reboot.

Conclusion

This project has helped us gain a lot of valuable knowledge and insight into the Android platform and development practices as a whole. It has been a tremendous learning exercise for all of us. Even though the project is nearing completion, there are still some cases where we can optimize a bit, for example if you have gone through the above diagrams then you may have realized that the flows can be simplified even further by introducing Android Push Messaging (C2DM), this we’ll do sometime in later months, and as we already have the provisioning server in place then it’s just a matter of time that we develop the feaures and push it to the wild.

On a closing note, if you want to build an app like this, then you can get in touch with us at biz@quadnode.com, we’d be happy to help you.

Thanks for reading!

PhoneGap is an open-source cross-platform Mobile Application development framework by Nitobi Software (now Adobe) which allows web developers to build mobile apps using HTML, CSS and JavaScript.

The PhoneGap project has been moved to Apache Incubator and now it is known as “Apache Callback”. PhoneGap project is divided into several sub-projects where each one represents a separate platform. In order to develop for iPhone and iPad, you need Mac OSX and XCode installed, similarly for Android you need Google Android SDK, Eclipse ADT Plugin, Ant as well as Eclipse IDE.

The official project detail can be found here Apache Callback. It is currently hosted in GitHub and once the Apache infrastructure is ready it will be moved there.

Supported Platforms

PhoneGap supports several mobile platforms, which enables web developers to write code using their existing HTML/JS skills and deploy to multiple platforms:

• iOS (iPhone/iPad)
• Android
• Symbian
• BlackBerry
• WebOS (most likely dead - go blame HP)
• WP7
• Samsung Bada

The environment setup for each of the above mentioned platforms can be found in “Getting Started”.

Architecture

PhoneGap is not a “native application development” framework, even though it provides access through several device features using an abstraction layer, which otherwise are inaccessible to a normal webpage. In simple words, it provies a “WebView” with extended capabilities. Using PhoneGap one can access the following device features:

• Camera
• Geolocation
• Compass
• Contacts
• Media
• Accelerometer
• Network
• Notification
• Storage
• Filesystem
• etc…

The following diagram explains how does a PhoneGap architecture look like:

In the above dialogram we can see that PhoneGap provides a “WebView” which is nothing but a bare bone web browser without a Chrome UI. The native code in PhoneGap SDK exposes uniform APIs across multiple device platforms which are accessible to JavaScript running inside the WebView.

As a web developer if you are using PhoneGap then it’s less likely that you’ll need to know the internals of the SDK (although it is always recommended to go through the source code in case you want to enlighten yourself as well as get fascinated by the tremendous efforts that the PhoneGap developers have put into it)

How to approach PhoneGap development

Even though you’re using HTML/CSS and JavaScript to develop applications, remember that it’s not a traditional desktop browser environment and the user experience pattern is significantly different. The PhoneGap apps are downloadable from App Store, so user might not know(or they don’t even care) that your app is built using PhoneGap or HTML/CSS, it’s always a good practice to be consistent with the platform UI guidelines so that you don’t confuse your users. A simple example could be if you’re developing a PhoneGap app for Android then be sure to handle the hardware/software(ICS) “back key”, if you don’t do that then your users might get confused.

Another important factor is handling scaling and device orientation, you should always make sure that your app looks proper in both Portrait and Landscape orientations.

The recommended way to start PhoneGap development is below:

• Setup environment for the target platform for example iOS/Android etc, see Getting Started.
• Run the sample app provided with the PhoneGap SDK.
• Go though the API Documentation.
• Try creating sample apps of your own using each of the device features available through the PhoneGap API.
• Once you are familiar with PhoneGap, try creating a sample app which uses multiple device APIs, you can try using jQuery Mobile, Sencha, Jo HTML5 Mobile App or jQTouch) to build the Phone like user interface.

Limitations

PhoneGap as development platform is very attractive and promises to bring uniformity across several mobile platforms. It aims to become a de facto mobile application development framework using HTML5 based technologies. But as said above the capabilities offered by the framework is limited to what a “WebView” can do. The following are a list of things to watch out for:

• In iOS the WebView is not JIT enabled (however the actual iPhone browser is), which means the JS code can’t take advantage of the engine optimizations which are otherwise available to actual browser of the device.
• In Android the CSS rendering in WebView is not as par with iOS, the rounded corners often look jagged.
• The WebView runs with limited amount of memory, be sure to optimize your JS code so that it doesn’t cause trouble.
• If you are developing for multiple platforms, make sure that you’re aware of the capabilities available in that platform, for example - in latest iOS you can use WebSocket but in Android it’s not yet available (not confirmed in ICS though).
• PhoneGap has good support from BlackBerry 6.0+ (with WebWorks SDK), phones with older BlackBerry versions may not work as expected.

Security

As your app runs inside WebView (aka a browser), you should not allow untrusted third-party code to execute in the page, because this can compromize the security, the JS code downloaded from internet will suddenly have access to several device features that they can make substantial damage to user’s privacy and security. Do not load an iFrame in your app, or redirect to some other websites when the app is loaded, as this will likely get your app rejected in Apple app store, beacuse Apple does not like applications opening public websites inside the app as it makes it difficult or almost impossible to enforce policies regarding content.

Resources

The following resouces can be useful for HTML5 based mobile apps development (not all are related to PhoneGap):

• jQuery Mobile
• Jo HTML5 App Framework
• Sencha Touch - Commercial
• jQTouch
• AppMobi
• DHTMLX Touch
• NimbleKit - iOS only Commercial
• The M-Project
• Wink Toolkit
• Zepto.js - JavaScript library for Webkit browsers
• Backbone.JS - JavaScript MVC framework

Conclusion

PhoneGap provides an opportunity to a vast pool of web developers reusing their existing skills to develop applications for mobile phones, which in future is going to be one of the most important medium to access information and content in internet. As the phones become more powerful, the new capabilities will be available in your arsenal. The recent developments in HTML5 based technologies also usher a new change in this segment. Do not miss the bus and keep exploring PhoneGap.