The last few weeks, I was focused on algorithms and data structures. I have been working my way through “Cracking The Coding Interview” and created a PHP library based on this book. I splitted interesting topics, such as dependency resolving using topological sorting, out into separate projects and released them under the MIT lincese on GitHub since I consider them as highly interesting and useful.
In this blog post, I want to try something new. For a long time, I have been wondering how services like contacts/calender/tasks/etc. from Google/Apple/etc. work. Sure, you have a backend server, some API’s and protocols and a handy Ul. But what does it look like under the hood in detail?
Custom Third-Party Contacts Client
I would like to take a deeper look at this topic. My primary focus is on how data is exchanged and synchronized between client and server as well as a good user experience. Since Google/Apple Contacts API require a lot of overhead for authentication, API design, etc. I decided to use a simple and custom data exchange protocol in order to keep the focus on creating the client.
I have a simple mobile-like client in my mind, like this:
So, the client is roughly separated in two parts: a “list view” containing all contacts in an address book and a “detail view” containing all details belonging to a contact.
Data Exchange
There are different ways to exchange data between a client and the server. I want to discuss some of them and share my thoughts about advantages and disadvantages. To keep things simple, I will use REST and JSON as the exchange protocol.
A first and naive approach would be to create two REST endpoints. The first endpoint returns all contact data belonging to an user and the second returns details about a given contact. This would be Ok for smaller apps but in general, it is not a good idea since it entails that every click on a user’s detail creates network traffic. This affects the user experience and in worst case, there is no data to show to the user (more about synchronisation later).
A better approach is to request all data, including details, at the initial app load (when the app starts the first time), and update data whenever needed. Beforing diving into details, there are a few other things to consider. Contact information can contain larger amounts of data, such as profile pictures. Therefore, it could be necessary to split out the biggest parts and download them seperately.
While users would accept to see profile pictures a few seconds later, it would be annoying if they see a spinner for a longer time. Therefore, it is a reasonable approach to have a little bit more effort to download and display all necessary data for better user experience and feeling.
Notice that this is not the only solution. There are other solutions which are good as well and may change for different conditions. The above proposed data exchange solution works well for up to a few hunderd contacts. But what if there are 10/20/30k contacts?
Synchronisation
The biggest advantage of using service to manage your contacts is the synchronisation over all your devices. Therefore, it is important to keep data up to date in both directions (client to server and server to client).
A quick and dirty, naive solution would be to wipe all data on each start (or foreground) and re-request it from the server. However, this would influence user experience bad and should be avoided. While it is reasonable to request all contact data on the initial app launch, it is bad in terms of user experience and data usage if you do it on each foreground.
A good approach is to separate available data from data that have to be synchronized with the server. For example, if you change a user’s phone number, the app creates a new JSON string with the user’s id, phone number and the update date.
The update date is used by the server to determine which data is newer. If there are any newer changes on the server, the client’s changes are discarded and the newer data is send back to the client. Otherwise, the server updates its data with the clients data.
If there are no new data, the client sends an empty JSON string to the server. The server knows then that the client has no changes. If the server has newer data, it sends back them to the client and the client is responsible to update its own data.
Caching
In my opinion, caching is the most important part of a contacts client. Imagine, you are abroad or at any place with poor network connectivity and want to call someone. But your client shows a spinner and tries to request data from the server.
Therefore, it is mission critical to cache all data. There are many ways to cache data locally. A naive solution is file serialization. But be aware, if you consider to use the file system, you usually have to deal with file permissions, file organizations (like subfolders and subfiles) and securing your data from a third party. That means, you have to handle stuff that puts your focus on something else than your contacts client.
Instead, a simple database, like sqlite or NoSQL, could help. It again depends on what you need. A key-value type NoSQL database can be used for storing all contact data. In this case the key would be the unique user identifier and the value the contact data (the JSON string, for example). This approach enables O(1) data look-up as the database serves as a kind of hash map.
If you want to go for “classic” databases, sqlite provides a simple and fast relational SQL standard where you can define one or more contacts table and use raw SQL queries. On the other hand, this would add a boilerplate since you have to create/maintain your schemas/tables. Furthermore, NoSQL type databases are a bit faster.
In the previous section, I discussed my approach to sync data between the client and server. Whenever a client gets new data from the server, the data has to be cached.
Data Spliting
Requesting the whole data set as multiple subsets from the server can be very important for a client with bad network connectivity as well as for user experience. If you take a closer look at the data set, you will see that you can swap out the profile picture requests and handle the contact list and details first. The reason is again: user experience and data usage. While users would accept to see names without profile pictures, they would be stucked if they need to call someone and see the loading spinner.
Therefore, I propose another REST endpoint that returns the pictures as URI’s or Base64 encoded, for example. Your client can then decide how it handles those information. For instance, if you support paging or scrolling, you could implement a lazy loading which processes picture data on page swap/scroll.
Implementation
I want to implement such a client. I am really very excited about this idea since a contacts client is something you use day in day out, but you do not care about the work behind the scenes. Furthermore, I want to try work with a programming language that I do not use very often: Python, since Python is that easy:
I also want to use a NoSQL database and want to understand its foundations. This is going to be fun, stay tuned!