Thu, 22 Dec 2011 22:49:15 -0800

Calypso — CalDAV/CardDAV/WebDAV for Android and Evolution

Ever since I bought my first Palm Pilot in 1997, I've relied upon a pocket-able device to carry a copy of my calendar and contacts, and for that same database to be present on my laptop. I went through a long list of Palm-compatible devices, including both the Palm Treo and Palm Centro telephones. I even wrote a number of my own Palm applications. Years ago, it was pretty obvious that I'd have to find a new phone, but I was stuck looking for something that could provide the same 'hot-sync' functionality.

SyncML on the Series 40

I bought a Series 40 Nokia phone in Shanghai that promised 'SyncML' support. Given that I had seen numerous SyncML implementations for Linux, it seemed like I should be able to get something to work.

SyncML on the Series 40 is a disaster -- the phone couldn't actually store all of my contacts, and it couldn't hold half of the data fields I used. So, that phone landed in the box and back to the Centro I went.

SyncML on the N900

Nokia kindly sent me an N900 at some point, so I gave SyncML another try. Given that the N900 runs evolution-data-server, and that I've had evolution-data-server running on my laptop, it seemed like I should be in business. Well, almost. It took several days of hacking to fix bugs in the evolution SyncML back end, then another several days fussing with opensync.

I ended up abandoning direct synchronization as unworkable -- opensync would sit in an infinite loop, or worse, trash my database completely. I finally found 'syncml-ds-tool', which is a debugging tool that comes with opensync. This tool simply synchronizes a set of disk files, one per contact or calendar entry, with the phone. That worked for well over a year. And then, a few months ago, Bluetooth on the laptop stopped connecting with the phone's SyncML server. I'd get 'ECONNREFUSED' every time I tried to use it. So much for the N900. DUN still worked, mostly, although it too would get ECONNREFUSED at times, but retrying seemed to make it work.

However, While the N900 SyncML solution worked, I discovered another thing I wanted—contacts and calendar entries stored in individual files and revision controlled with git. This makes it reasonable to delete stale calendar entries and know that they're never really gone, just left behind in an older version of the calendar. And, if you mess up, you can recover by poking at the database with git directly.

I switched back to the venerable Palm Centro; it turns out that calendar and contacts are more important to me than being able to surf the web on my phone. Alas, my Centro went 'swimming' in August and has passed on to the great electronics recycling house in the sky. I pulled the SIM out and switched back to the N900.

I got my contacts imported on the N900 by copying files over the net work; not a long-term strategy, but at least I had phone numbers again. There was no hope for my calendar. I started looking for a solution in earnest.

How about Android?

At this point in my story, I'm sure you're asking why I didn't just use one of the numerous Android phones that came through my hands. The answer is simple — my calendar and contacts are probably some of my most personal data, and I'm not willing to store them outside of my direct control, for reasons similar to those which are driving the development of the FreedomBox.

When Android first came out, it could only talk to Google services, which didn't meet my hard requirement for personal data storage.

One of my co-workers had his Google account suspended for violating the terms-of-service; he asked what he had done, but they wouldn't say. He asked if he could get his data back, and they said “no”. They invited him to create a new account, but it would not ever get any of the old data. A few days later, he got a nice apologetic email letting him know that they'd made a mistake and that he hadn't, in fact, violated any of the terms-of-service, and that his old account was restored with all of his data intact. Wasn't that nice of Google?

WebOS?

I got a WebOS phone over the summer and discovered that while it had multiple contact/calendar back ends, none of them used standard protocols and so you only had the choice between multiple corporate data centers, which isn't actually a choice at all.

Furthermore, the WebOS phone refused route PAN packets over the phone network, even though I have a data plan which allows this. It's not that it couldn't support it, it's that it refused.

A couple of weeks after the WebOS phone arrived, HP canceled all of their WebOS hardware products, which made me less interested in trying to solve this problem.

Android recovers

About the same time the WebOS phone arrived, I discovered that Google had published enough information about the calendar/contacts internals for Marten Gajda to write CalDAV-Sync and CardDAV-Sync. And then, Andrew McMillan wrote aCal, which is a complete replacement for the built-in calendar and contacts applications and supports CalDAV and CardDAV.

With two different standards-compliant solutions available, it seemed like it might be time to try Android again.

I'd love for CardDAV-Sync and CalDAV-Sync to become free software like aCal is. Andrew makes money from aCal by offering it for sale via the Android Market, while still publishing the sources for those who want to build their own copy.

CalDAV/CardDAV on Linux

I think the most widely known CalDAV server for Linux is probably DAViCal, a huge pile of PHP and SQL sitting on top of Apache. I'm sure it's suitable for running on a server and being accessed over the internet, but I'm not interested in that, nor am I interested in having my laptop run Apache and PostgreSQL.

I found a tiny little CalDAV server, Radicale, which seemed like a lot better fit. It's written in Python and uses the usual Python HTTP server infrastructure, which provides SSL and authentication support along with some fairly convenient APIs for parsing and generating HTML.

Before long, I discovered that Radicale was actually too simple for my needs. It stores the whole calendar in a single file, re-parsing it whenever a request is made, so a calendar with just hundreds of entries caused the server to slow down enough that evolution would time-out when talking to it.

Also, Radicale doesn't actually parse the calendar entries completely, it has some ad-hoc code that finds various pieces of data, but without dealing with the whole syntax.

I started hacking at Radicale to see how far I could get. I changed the storage code to store one event per file, then added hooks to use git for change management. Then, I found a full vcalendar/vcard parsing library in python, vobject, which I used to replace the ad-hoc parsing code. Finally, I added support for VCARD entries as well, allowing the system to store both calendar and contact information.

Introducing Calypso, a CardDAV/CalDAV server

With this much divergence from the original project, I've figured I'd best rename things to avoid confusion, so I decided to call it 'calypso', after a brief trip through the dictionary looking for names starting with 'ca'.

Calypso works with evolution, iceowl and the Android CalDAV/CardDAV plugins. It does not yet work with aCal; for some reason aCal cannot find any calendars on the server.

Calypso also supports importing calendar changes from the command line, allowing you to integrate support into a text-based email application like notmuch or mutt.

Calypso is available via git from git://keithp.com/git/calypso and is distributed under the GPL (v3 or later). I still consider it a work derived from Radicale, and so the code retains all of the Radicale copyrights along with my own.

Using Calypso

Initial setup

Calypso runs as a regular user, all data are stored in ~/.config/calypso. To initialize calypso:

$ mkdir ~/.config/calypso ~/.config/calypso/calendars
$ cat > ~/.config/calypso/config << EOF
[server]
ssl=true
certificate=/home/keithp/.config/calypso/ssl/server.crt
key=/home/keithp/.config/calypso/ssl/server.key

[acl]
;type=htpasswd
type=fake
filename=/home/keithp/.config/calypso/passwd

Running calypso

Then run calypso:

$ python ./calypso.py

No, I haven't figured out how to install it...

Creating new calendars

To add a new database:

$ mkdir -p ~/.config/calypso/calendars/private/my_calendar
$ cd ~/.config/calypso/calendars/private/my_calendar
$ git init
$ git commit --allow-empty -m'initialize new calendar'

The new calendar should now be visible as https://localhost:5233/private/my_calendar

You can add files to the directory at any time; calypso will check the directory mtime at each operation and update its internal state from that on disk automatically when the directory changes.

Importing files

Given a set of files with VCALENDAR or VCARD entries, you can import them with:

$ calypso --import private/my_calendar <filenames...>

This will update any changed entries and add any new ones.

ToDo list for calypso

1. Document the config file contents.

2. Make it installable

3. Figure out what aCal wants

4. Support calendar creation

More Android info

Synker. This provides a desktop widget which starts the sync process running on a list of accounts. This makes it easy to manually synchronize with the laptop when you are connected

Contact Editor. This lets you fully edit contacts synchronized over CardDAV. The built-in contact editor doesn't let you change anything other than the name for some reason.

I'm running cyanogenmod on my Nexus S as that provides PAN support. With PAN, I can create a network link between laptop and phone which doesn't depend on any local WiFi infrastructure and which gives both phone and laptop static IP addresses, allowing me to configure the sync URLs statically on the phone. I'd use mDNS, but Android doesn't bother to support that.