Android Masterclass: Data storage and retrieval


Any non-trivial application will need to store some data locally on the phone for use later. There are also a number of different types of information that needs to be stored for later. For instance we may need to store authentication information have a temporary cache of information store images or allow the users to capture a larger block of information via a form. The Android environment provides a range of options for data persistence allowing developers to cater for these different use cases. Storing data on the phone opens up the issue of data security and in the Android system by default one application’s data cannot be accessed by another one — that is application data can be sand-boxed and protected.

In this master class we will start with an overview of the different methods that Android provides for data storage and briefly go over the security model used to protect application data. We will walk through code snippets from two sample applications to cover the different key approaches for data access and storage. All resources for used in this master class are now also available on GitHub.

A Short Recap

Before getting into the details of working with data here is a recap of the core Android concepts covered in the previous master classes. An Android application is made up of Activities that map to an individual window/screen of functionality in an application. We place UI components (called Views) inside a container called a layout and attach this layout container to an activity. The UI is described in an XML file and paired with a Java class that provides the interactivity. The Android framework mandates conventions which specify a set of folders that every Android project must have. The conventions specify location of resource such as images and data for the development tools to process them and generate references to use these resources from within the XML UI description files and Java code. The framework is designed to help developers construct UI that works on devices with varying screen sizes with minimal pain.

Every activity in an Android application has its own life cycle and activities communicate between each other using short messages known as intents. The Android architecture allows any application to send a message to another other application via intents. This flexibility permits developers to make use of functionality provided by other applications with minimal pain. For instance we can broadcast a message requesting display of a web page and all registered browsers will respond – the user then has the choice of selecting the application that they want to use (we will make specific use of this feature in this master class).

Android applications can make use of Maps via the use of Google APIs that are available as a free add-on from within the Eclipse IDE. We can also build applications that incorporate image galleries video and audio by using the media APIs provided as part of the SDK.

Data Storage Options

Android offers us different options to store data all of which have associated trade-offs. Broadly we can store data on the internal flash storage removable SD card or access/store data on an external server. Android devices tend to have an internal flash storage option though the size of this varies — typically many gigabytes. However the SD card slot is an optional feature of many devices but users may not place a card in their phone. Given this situation developers have to be careful with both how much data they want to store on the device and where they want to store it (i.e. SD card is not a good default option).

From the perspective of actual development the following options are available for data reading and storage:

  • Bundle data in the APK as a resource: We can bundle data as a resource (stored as a file in res/raw folder). This data is treated as a read-only resource so it is a great option to bundle some default information but not useful where the user needs to update/add/remove the data. A limitation of this option is that resources have to be smaller than 1Mb targeting older Android phones (running 2.2 or lower). If they are larger then a rather cryptic error message is generated internally and the application is not able to access the resource. Thankfully later versions of the operating system have addressed this limitation.
  • Use Shared Preferences: We can store preference information via the use of a convenience feature known as Shared Preferences. This method is perfect when we only need to store small and simple blocks of data like user name or information like the preferred ringtone or sound to play or timezone information. Software engineers tend to call this type of information simple key-value pairs. That is we store and read the data via a simple string key.
  • Store/Read data from a custom file: We can store information in files using our own custom format. The files can be stored either in the internal flash memory or external SD card. This option is ideal when we want to create a temporary cache storing logs or when we need to storing data in a custom data structure. However we need to write all of the read and write functions from scratch.
  • Persist data to the internal sqlite database: Android (like iOS) bundles the sqlite database management system as part of the platform. The data itself is stored in a file but we can access the information via the use of SQL. Sqlite DBMS runs in-process and manages the data integrity. This option is ideal when the information is tabular or inherently relational. An interesting factoid — Firefox and Chrome browser use sqlite for their own data management. If you navigate to the directory where Firefox stores its profile data you will see a number of sqlite database files that are used to store things like bookmarks history and even extensions.

Security Model

Storing data on a normal computer (Android devices are pocket computers) has one big security problem — the data created by one application is easily accessible by another application. However given the personal nature of mobile phones this open access is often not acceptable — esp. as the default option. Thankfully the Android platform offers developers the flexibility of storing information either privately or making it open as public data.

On Android devices the default security model enforces applications to store data into a private area sealing it off from all other applications. This model is based on the rationale that users store personal information on their phones with an expectations that it will be secure. Furthermore users are not fully aware of (not care to know) the details of file system security. In effect the safest option is to sandbox all application data ensuring a basic layer of security.

Android achieves this level of security by using a rather simple (old) trick that works very well on UNIX inspired operating systems. On Unix (and Android) every user has a user-id and the permissions to read/write or execute a file are restricted based on this user-id. This mechanism of allocating permissions to users is baked right into the heart of all UNIX based operating systems. Now to the trick — Android allocates every application a unique user-id (UID) and restricts data created to that applications UID (see Figure 1). This trick saves inventing a complex additional security model since we can use the one that is already available. Additionally access to network sensors and other devices is also provided on top of this mechanism. Again all that is done is devices sensors are all treated as files and a block of user-ids are given permissions to them. In case you are wondering if the UID this can be spoofed — the UID for an application is generated at install time based on the signature and package name in the APK making it quite difficult to get around.

Figure 1 – Android Security Model

The security model restricts access to application specific private data. However developers often need to access the files during the construction of software in order to verify that the data is being saved properly and also to monitor the size of the data. The good news is that developers have full access to the data that is stored by the application on the emulator because we have root (administrator) privileges. Unfortunately the bad news is that by default Android hardware devices are locked by the manufacturers. That is users do not get root access to the file system of their Android devices by default. This restriction does not have much of an impact on most normal users — similar to most office computers having restricted (non-admin) access causing minimal issues for typical end-users. However developers often need this additional layer of access and on Android we can gain root access by following a bunch of steps. The actual method to gain root access is outlined in a previous APC article ) available at http://bit.ly/zLtw80.

Gaining root access on an Android phone unlocks a lot of hidden power but there are risks. In particular attempting to root devices locked by carriers/manufacturers may damage the loader causing the operating system to not boot up properly. Some manufacturers also consider rooting to be breaking their terms of agreement. Additionally you may lose data stored on the device during the process. Thankfully you can buy devices targeted at developers like the Google Nexus phone range that come with an unlocked SIM/boot-loader allowing you to gain administrator access at lower risk.

Viewing Files and Pulling them from the Device

Android offers us security on the data and a number of different options to help work with data. However before getting into the code it is good to see how we can access the file system directly. The most convenient way is to use the adb (Android Debug Bridge) tool that is provided as part of the SDK. This tool allows us to gain shell access to the phone from the command prompt (or Terminal). As strange as it may sound we can do this on an Android device since it is running a customised version of Linux deep down.

The abd tool is located on the tools directory of the Android SDK so we will have to navigate to that folder first on the command prompt if it is not in your path. Once you have navigated to the location of adb connect your phone or fire up the emulator and run one of the following commands.

If you want to connect to you phone/tablet run the following command:

	
		adb -d shell
	

To gain shell level access to the emulator run the following command:

	
		adb -e shell
	

Once you have gained access to the shell you can run the ‘ls’ command to see the list of files. All application private data is stored in the ‘data’ directory. You can go into this directory using the command ‘cd data’. However if you try to see the list of files in this directory using the ‘ls’ command on a locked phone you will get an error message since you do not have root/admin access to the phone. On the emulator you have root access and hence you will be able to navigate into all directories and see the contents. The application private data is stored under ‘data/data/PACKAGE_NAME’. For instance ‘data/data/apcmag.examples’ will contain data for the applications that we will build and you will be able to see it in the emulator directly.

Once in the shell of your device (or emulator) you can see the contents stored on the SD card as well. In order to do this go to the root directory of your device and then type ‘cd sdcard’. Here you can list the files using the ‘ls’ command. This area is not private even on a locked phone so you will be able to see the contents and navigate around. Even if your phone does not have an SD card this directory is created (virtually) for you by the operating system and hence you will see this directory.

The adb command also allows us to pull files from the device and push them up if needed. On my phone I have a weather application that stores (caches) background images for the weather radar on the SD card. The following command illustrates how to download images stored by this app to your computer but this applies to any file stored on your phone.

	
	$ adb -d pull /sdcard/AussieWeatherRadar ./AWR
	pull: building file list...
	pull: /sdcard/AussieWeatherRadar/IDR023range_img -> ./AWR/IDR023range_img
	pull: /sdcard/AussieWeatherRadar/IDR023topo_img -> ./AWR/IDR023topo_img
	pull: /sdcard/AussieWeatherRadar/IDR021range_img -> ./AWR/IDR021range_img
	

So far we looked at how the Android platform offers us different ways to work with data and covered the security model that offers the ability to keep data private to the application or make it public. In the sections that follow we will look at the API available for working with data and illustrate it with code snippets.

Basic Input/Output in Java

Before getting into the Android API for dealing with data it is helpful to briefly cover how Java platform deals with I/O. Feel free to skip this section if you are already familiar with I/O in Java.

Java uses streaming for I/O as it greatly simplifies I/O operations. In simple terms an IO stream represents either a source of input data or an output destination for the data. The real strength of this concept is that a stream can map to many different kinds of sources and destinations including files data streams from a sensor memory or even other programs. In software engineering terms a stream abstracts away the actual source/destination of the data — and this idea is considered to be one of the seminal contributions of Unix.

In more practical terms we read data from an Input Stream which is often connected to a file (the data source). Similarly we write data to an Output Stream that is connected to a destination. A stream as its name suggests literally carries bytes of data from a source to our program or the other way around.

The Java platform allows us to work with raw bytes in a stream or wrap the raw stream around a buffer to help us work with slightly larger blocks of data. For example when we read from a file it may be easier to read a line of text or an entire record rather than one bit at a time. The code presented in the following sections will make use of streams to read/write data to files.

Reading Raw Resources

The Android platform allows us to bundle read-only resources like images audio and video (we place them under the res folder of the project as expected by the convention). The platform also allows us to bundle either raw binary or text files.

Common scenarios in mobile applications require us to display an end user agreement release notes to let users know about some specific new features in the application or we may be inspired (or forced) to provide a user manual. In these scenarios we can bundle the data as part of the application and show it to the users when the application is started or if the user requests this information via an About dialog.

The following code snippet shows how to connect a stream to the data bundled in the res/raw folder of the project. We also wrap the input stream in a buffer to allow us to read one line of text at a time.


InputStream in = getResources().openRawResource(R.raw.android_dev_agreement);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String line;
while ((line = br.readLine()) != null) {
	// do something with each line
}

As can be seen from the code snippet working with resources is simple. However older versions of Android (2.2 and under) have an unfortunate limitation — resources have to be less than 1Mb uncompressed. If they are larger then you will get an error message in the log. This will not be an issue moving forward but approximately 40% of devices are still running 2.2 or older releases of Android — so developers have to work around this issue for now.

A sample project that reads the Android development agreement file and displays all the words in a list is provided as an on-line resource. This project provides a detailed example of how to read raw resource files and also displaying data in lists. You can download this project from GitHub at https://github.com/downloads/rvasa/medroid/WordList.zip.

Storing Data in Files

Data persistence (storing) is a common need in mobile applications. Developers need to store simple blocks of data like user names preferences as well as more complex structures like images or custom data. The platform offers us a simple set of functions to read as well as write data via the Activity super-class. The data itself is stored on the device in a specific pre-defined location (/data/data/[package-name]/files) as discussed in earlier sections. We can create files here that are private or open to all the other applications on the device. The API also allows us to append data to existing files or override and start from scratch.

The code snippet below illustrates how we can store a block of data using via the use of an output stream. This sample code stores the latitude and longitude location information into a file. The actual method that opens a stream to store the file is provided by the parent Activity class and all we do is make a call to it (see Line 5 in the code below). In the code below we have set a flag to make the file private (MODE_PRIVATE). The API allows us to append to the file if we want to using MODE_APPEND flag. We can also make the file public via two additional flags: MODE_WORLD_READABLE and MODE_WORLD_WRITABLE. These flags are specified as bit masks so we can combine multiple flags. For instance to make create a public read and write file we can specify it as MODE_WORLD_READABLE | MODE_WORLD_WRITABLE.

	
		private void saveGeoLocation() throws Exception
		{
			String FILENAME = "user_geolocations";
			String string = "Melbourne -40.23 143.23";
			FileOutputStream fos = openFileOutput(FILENAME 
								Context.MODE_PRIVATE);
			fos.write(string.getBytes());
			fos.close();	
		}
	

In order to read the data back from the file we use a slightly different method call (openFileInput) which gives us an input stream that we can read the data from. We can also wrap the input stream with a buffer as illustrated in the previous section in order to read the data. Furthermore unlike writing to a file we do not need to worry about private/public modes since all data created by an application is available back to it. The following code snippet illustrates how we can read the data back from a file.

	
		String FILENAME = "user_geolocations";
		FileInputStream fis = openFileInput(FILENAME);
		BufferedReader br = new BufferedReader(fis);

		// We can now process data
	

The code snippets that we presented so far illustrate the key concepts and show how we can start using the API to work with data. Additional details of how to work with data along with complete documentation of the data storage API is available at http://bit.ly/zqy0LE.

Working with sqlite Databases

Storing data in files allows us to work with custom data structures and using read-only raw files is helpful when we want to bundle constant data. However applications often need to store relational data (as in tables of information) and be able to quickly search the data. In these scenarios the built-in sqlite database technology offers us a reliable way to store the data and also permits quick retrieval. We will go over the concepts and API using a simple application that stores quotes from famous people and allows us to add and remove these quotes. You can use the code from the sample application provided to add a quote database and capture interesting phrases from your friends as part of the MeDroid application. The complete source code for the sample application is available on GitHub. It is worth downloading the code as we will walk through only some of the code in snippet form here. We are also not going to cover database concepts or SQL here — if you are bit rusty in these concepts then the free database concepts book from IBM Developer Works should close off that gap.

In case you are wondering what is sqlite really is before going in the code. It is a relational inspired database management system (DMBS) that has tables with rows of data that are accessible via named columns. The database management technology runs in-process stores the data in a single database file locks this file when updating and it does not really offer us any type safety (that is it allow us to store any type of data in the columns). It allows us to run SQL queries and process the result set one row at a time using a Cursor.

The Android data management API directly connects to an sqlite database. We also get support for common tasks like upgrading databases from one release to the next as well as the ability to handle errors. Rather than work with at the lower level the Android API provides us a helper class (like it did for List and Map management) that simplifies data access via the use of conventions. In particular this helper class is designed to simplify creating/opening/reading as well as upgrading a database. It will handle all typical errors that arise during data access and will help us run long queries using via asynchronous background task.

In the Android API we have four key classes that help us work with sqlite databases:

  • SQLiteOpenHelper:This class contains functionality to open/create and upgrade the database.
  • SQLiteDatabase:This class maps directly to the underlying database.
  • SQLiteStatement:This class encapsulates an SQL statement.
  • SQLiteCursor:This type is returned back to us when we run a query and it allows us random read/write access to the result set of the query.

We use the helper class to connect and manage an sqlite database. We can run queries via the use of an statement. When we run a query on the database it returns a set of matching rows that we can access via a cursor. The result set is a copy of the information in the actual database file. Hence the cursor will not reflect any changes made after the query. If the data has changes since the last query we will have to run the query again to get the latest information. The key design implication is that we have to close the cursor as soon as we are done with the information to avoid keeping a copy of a large volume of data in memory.

The sample code that we will walk-through has two classes: QuoteDBActivity and DBHelper. The activity class is the primary UI holder while the helper class inherits from SQLiteOpenHelper and allows us to create/open the database as needed. We create the helper from inside the onCreate method of the primary activity as shown in the code below. This application also makes use of contextual menus for adding and removing quotes.


	public void onCreate(Bundle savedInstanceState)
	{
		super.onCreate(savedInstanceState);
		db = new DBHelper(this); // custom helper class
		showAllPeople();		
		registerForContextMenu(getListView());
	}

The database helper class that we create (Line 4 of the code block above) inherits from the SQLiteOpenHelper class within the Android API. This class itself has another onCreate method that is called internally when a database needs to be created (this will happen when we first launch the application). The code snippet below describes this onCreate method where we create a SQL statement that will create a table called quoteTbl with two columns — name and quote. We also provide an auto incrementing identifier as it is required by convention. Once the statement is constructed we execute that on the database after which we populate the database with some default data.


	// Fields in DBHelper
	private static final String DATABASE_NAME="quotes";
	static final String QUOTE_TBL = "quoteTbl";
	static final String NAME="name";
	static final String QUOTE="quote";

	// Run when database is created
	public void onCreate(SQLiteDatabase db) 
	{
        String createStmt = "CREATE TABLE "+QUOTE_TBL;
        createStmt += " (_id INTEGER PRIMARY KEY AUTOINCREMENT ";
        createStmt += NAME+" TEXT ";
        createStmt += QUOTE+" TEXT);";
        db.execSQL(createStmt);
	    constructDefaultData(db);
	}

In order to populate our database when it is first created with some default data we make use of a convenience class called ContentValues. This class is similar to a Hash table (or a Dictionary) and allows us to store a row of information against column names. Once the content is constructed we insert it into the table (see last line of code in the snippet below). The sample application provided inserts a few more rows of data using the same approach. This information is stored in the database file and will be available to the application when it is restarted.


	public void constructDefaultData(SQLiteDatabase db)
	{
        ContentValues cv = new ContentValues();
        cv.put(NAME "Master Yoda");
        cv.put(QUOTE "Do or do not there is no try");
        db.insert(QUOTE_TBL null cv);
    	//....

Once the database is populated with data we display it to the user by querying it back. The API allows us to run any valid SQL query however in our case we have a query that selects all of the rows and displays that in the list. The query in the code below is stored as a string in the DBHelper class and we make use of it (Line 4). The query returns a Cursor that allows us access the results. In the code below we tie the results directly into an adapter that will display the information in a list (see Figure 2).

List adapters allow us to connect a ListView to a data source. We are using an adapter (SimpleCursorAdapter) that is provided by the API to directly process a database cursor and display the result set. In our application we display the data using a custom formatting for each row — the quote and the associated author are displayed together in a row in slightly different font sizes and styles (see Figure 2). We achieve this formatting using a custom row layout that is defined in row.xml (it is a simple Linear Layout with two TextView’s — you can see the code in the resources provided). The adapter also needs to be given an array to hold the data as well as the resource identifiers to map the data into. The rather long and complex looking line in the code below constructs a list adapter with a custom row formatting style and ties the columns of the database to text view resources.


	private void showAllQuotes()
	{
		if (personCursor != null) personCursor.close(); // close old cursor		
		personCursor = db.getReadableDatabase().rawQuery(DBHelper.SELECT_ALL null);
		ListAdapter adapter = new SimpleCursorAdapter(this R.layout.row
				personCursor new String[] { DBHelper.QUOTE DBHelper.NAME }
				new int[] { R.id.quoteTextView R.id.nameTextView });
		setListAdapter(adapter);
	}

Figure 2 – Show all quotes in a list

In order to add a quote we have to perform an insert operation on the database. But how does the user provide this information? We get the data via the use of a dialog that pops up on top of the application. This is constructed using a layout similar to an Activity and started using a dialog builder that is provided within the API. We call the method shown in the code snippet below when the user enters some data and selects add on the dialog (see Figure 3). To add the actual data we start first by obtaining a writable database (Line 6) and then insert the content. Once the new data has been inserted we repopulate the list with all the quotes. This brute-force re-query approach is fine for small to even medium sized data but it tends to be inefficient for very large data sets. The API does not provide a direct way to efficiently handle large data sets and that will require some careful design and engineering.


	private void processAdd(DialogWrapper wrapper)
	{
		ContentValues values = new ContentValues();
        values.put(DBHelper.NAME wrapper.getName());
        values.put(DBHelper.QUOTE wrapper.getQuote());
		db.getWritableDatabase()
		  .insert(DBHelper.QUOTE_TBL null values);
		showAllQuotes();
	}

Figure 3 – Add new quote dialog

The sample application also allows us to remove a quote. In order to do this we take advantage of the long-press event. If the user presses on a list item for a period of time then they are shown a ‘Delete’ button which will remove the selected item. The process of removing a list item also requires us to get access to a writable database first followed by calling the delete method (see code snippet below). The delete operation requires the row that has been selected for removal and it identifies this specific row via the automatically generated ID column.


	private void processDelete(long rowId)
	{
		String[] args = { String.valueOf(rowId) };
		db.getWritableDatabase()
		  .delete(DBHelper.QUOTE_TBL "_ID=?" args);
		showAllQuotes();
	}

In this application we make use of a contextual menu to add a quote and to clear all quotes in the database (see Figure 4). When the user clicks on the menu button the Android framework calls the onCreateOptionsMenu method on the current activity. In this method we provide the menu construction and handling code as outline in the code snippet below. The API allows us to specify a title an icon as well as a short-cut.

Figure 4 – Contextual menu to add quote and clear database


	public boolean onCreateOptionsMenu(Menu menu)
	{
		menu.add(Menu.NONE ADD_ID Menu.NONE "Add").setIcon(R.drawable.add)
				.setAlphabeticShortcut('a');

        menu.add(Menu.NONE CLEAR_ID Menu.NONE "Clear").setIcon(R.drawable.clear)
        .setAlphabeticShortcut('c');
        
		return (super.onCreateOptionsMenu(menu));
	}

Once the user selects a specific menu item from the contextual menu displayed (for example if they select Add) then the framework makes another call back to the activity — this time it calls the onOptionsItemSelected method. We provide the logic for how we want to deal with the menu item in this particular method as shown in the code snippet below. The add method in the code below constructs a dialog to display for getting a new quote (see Figure 3). The clear menu item requests a confirmation via a standard alert dialog and then based on the user response clears the database of all quotes (see Line 158 of QuoteDBActivity class source code in the supporting resources provided).


	public boolean onOptionsItemSelected(MenuItem item)
	{
		switch (item.getItemId())
		{  case ADD_ID:
			 add(); return true;
           case CLEAR_ID:
             clear_all(); return true;
		}
		return (super.onOptionsItemSelected(item));
	}

In this master class we covered how to persist data directly into a file as well as how to use the built-in sqlite database management system. The specific approach will depend on the needs of the application and often both of these approaches are employed. Some applications also provide the default data as a raw resource file and load that on initial activation. In the next master class we will bring the concepts presented so far into the MeDroid application and will go over the steps required to publish the application to the Android market place.

PREVIOUS: Working with maps and lists | NEXT: How to take your app to Android market

Home: Main masterclass index

  • S Wong

    I want to know more about data management on android:

    1. May I ask if files are deleted from android, where will the files go to? I heard that data still exist on device but there are some pointer reference to the deleted space. Is that true?

    2. Also, if I deleted photos folder accidentally. Will new data immediately overwrite the “deleted” blocks immediately instead of writing on other free blocks even if the phone is quite new and not much data has been written?

    3. Will unroot process in flashing rom clear all the deleted photos memory?