Go to the first, previous, next, last section, table of contents.

Tutorial

This chapter shows, by example, how to write ColdSync conduits. The examples are written in Perl, simply because I happen to like it. However, you can use any language you like to write conduits.

The example conduits in this chapter use the ColdSync.pm module that's part of the ColdSync distribution, and also the p5-Palm module from
http://www.coldsync.org/.

Conduit Workings: A Quick Overview

A conduit is simply a program, one that follows the ColdSync conduit protocol (see section Specification).

In a nutshell, ColdSync runs a conduit with two command-line arguments: the string conduit, and another that indicates the conduit flavor, either fetch or dump.

ColdSync then writes a set of header lines to the conduit's standard input, e.g., 

Daemon: coldsync
Version: 2.3.0
InputDB: /homes/arensb/.palm/backup/ToDoDB.pdb
Phase-of-the-Moon: lunar eclipse

followed by a blank line.

The conduit reports its status back to ColdSync by writing to standard output, e.g.: 

202 Success.

The three-digit code indicates whether this is an error message, a warning, or an informational message. See section Status Codes. The rest of the line is a text message to go with the status code. It is not parsed by ColdSync; it is intended for human readers.

A conduit should print such a message before exiting, to indicate whether it was successful or not.

todo-dump

Let's write a Dump conduit that writes the current To Do list to a file. This is a single-flavor conduit, so we'll use the following template: 

#!/usr/bin/perl
use Palm::ToDo;
use ColdSync;

# Declarations and such go here.

StartConduit("dump");

# Actual conduit code goes here

EndConduit;

The Palm::ToDo module is a parser for ToDo databases; it adds hooks so that when the conduit reads the ToDo database, its records will be parsed into structures that can easily be manipulated by a Perl program (see Palm::ToDo(1)).

The ColdSync module provides a framework for writing conduits, and defines the StartConduit and EndConduit functions.

StartConduit takes one option indicating the conduit flavor (Dump, in this case). It checks the command-line options and makes sure the conduit was invoked with the proper flavor. It reads the headers from standard input and puts them in %HEADERS. If the conduit was given a InputDB header, StartConduit loads the database into $PDB.

EndConduit takes care of cleaning up when the conduit finishes. For Fetch conduits, it writes $PDB to the file given by $HEADERS{OutputDB}.

Starting with this template, all we need to do now is to insert the actual code: 

#!/usr/bin/perl
use Palm::ToDo;
use ColdSync;

$OUTFILE = "$ENV{HOME}/TODO";           # Where to write the output

format TODO =
@ @ @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
$marker, $priority, $description
      ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ~~
        $note
.

StartConduit("dump");

open OUTFILE, "> $OUTFILE" or die("401 Can't open $OUTFILE: $!\n");
select OUTFILE;
$~ = TODO;                      # Set the output format

foreach $record (@{$PDB->{"records"}})
{
        $marker         = ($record->{"completed"} ? "x" : "-");
        $priority       = $record->{"priority"};
        $description    = $record->{"description"};
        $note           = $record->{"note"};

        write;
}

close OUTFILE;

EndConduit;

The ColdSync.pm module provides wrappers for Perl's die and warn functions, so that their messages will be passed back to ColdSync. The rest of the code should be self-explanatory.

pine-aliases

Now that we've seen a trivial conduit, let's take a look at a slightly more complicated one: a conduit to synchronize addresses in the Palm Address Book database with those in Pine's address book.

Note that this is still just a tutorial conduit: we'll be making some simplifying assumptions that will make this conduit unsuitable for use in the real world.

Having said this, let's take a look at the conduit: 

#!/usr/bin/perl
use Palm::Address;
use ColdSync;

$PINE_ALIASES = "$ENV{HOME}/.addressbook";

ConduitMain(
        "fetch" => \&DoFetch,
        "dump"  => \&DoDump,
        );

Unlike todo-dump (see section todo-dump), pine-aliases is a multi-flavor conduit: it can be used either as a Fetch conduit or as a Dump conduit. For this reason, we use ConduitMain rather than StartConduit/EndConduit.

There are several reasons why one might want to write a multi-flavor conduit like this one. The first is that the Fetch and Dump functions really just implement the two halves of a single conduit that performs two-way synchronization between the Palm and Pine.

Secondly, we'll be writing some convenience functions that will be used by both &DoFetch and &DoDump, so it makes sense to keep them together.

Finally, in many cases, the two things that one is synchronizing (in this case the Palm Address Book and Pine's addressbook file) don't contain the same information, or represent it in such a way that it's difficult to convert one to the other, and the conduit writer must resort to a number of tricks to perform the sync correctly.

For instance, the Fetch conduit for kab tries to save each person's fax number in the Palm database. If there is no fax field, it will append "(212) 123-4567 (fax)" to the "Other" field. Therefore, the kab Dump conduit must look for the fax number in the "Other" field as well as the "Fax" field. Keeping the two conduits together in the same file makes it easier to keep track of these sorts of tricks and make sure that the two conduits work properly.

ConduitMain takes as its arguments a table that tells which function to call for each flavor. When the conduit is run, ConduitMain parses and checks the command-line arguments, reads the headers from standard input and stores them in the hash %HEADERS, and calls the appropriate function. If an InputDB header was specified, that file will be read into the variable $PDB. Then it calls the flavor-specific function (in this case, either &DoFetch or &DoDump) to do the actual work of the conduit, and finally cleans up: for Fetch conduits, it writes the contents of $PDB to the file specified by the OutputDB header.

&DoFetch

The &DoFetch function reads Pine's alias file. For each address that it finds there, it updates the email address in the appropriate record in the Palm database. 

sub DoFetch
{
        my %aliases = ();

        if (!defined($PDB))
        {
                $PDB = new Palm::PDB;
                $PDB->Load($HEADERS{"OutputDB"}) or
                        die "502 No input database\n";
        }

        open ALIASES, "< $PINE_ALIASES" or
                die "Can't open $PINE_ALIASES: $!\n";

        while (<ALIASES>)
        {
                my $alias;
                my $addr;
                my $fullname;
                my @rest;

                chomp;
                ($alias, $fullname, $addr, @rest) = split /\t/;
                $aliases{$fullname} = $addr;
        }

        my $fullname;
        my $address;

        while (($fullname, $address) = each %aliases)
        {
                my $record = &find_person($PDB, $fullname);

                next if !defined($record);      # No entry in PDB

                my $pdb_addr = &get_address($record);

                next if $pdb_addr eq $address;
                                # It already matches. Ignore it.

                print STDOUT "101 Setting $fullname -> $address\n";
                &set_address($record, $address);
        }
        close ALIASES;

        return 1;               # Success
}

The InputDB header is optional for Fetch conduits, so $PDB may not have been initialized. But pine-aliases does not create a new database from scratch; it only modifies an existing one. If no InputDB database was specified, we load the database specified by OutputDB.

The body of &DoFetch is divided into two phases: in the first phase, it reads the Pine alias file and builds a hash, %aliases, that maps each full name to its email address. The second phase goes through this map and updates each record in $PDB. This two-phase approach may seem overly complex; the reasons for it are discussd in section Limitations of pine-aliases.

Each line in the Pine address book contains a set of tab-separated fields: the person's alias, full name, email address, and a few others that we don't use.

We'll need some way of figuring out which Pine alias goes with which Palm Address Book record. Since the Pine alias file does not list Palm record IDs and Palm records don't list mail aliases, we'll settle on the full name as the next best way of uniquely identifying a person.

The second phase of &DoFetch uses a number of helper functions: &find_person takes a person's full name and returns a reference to the corresponding record in $PDB; &get_address extracts the email address from that record; and &set_address sets the email address in the record.

One important thing to note is that &set_address marks the record as dirty. During a normal sync, ColdSync only considers those records that have changed in some way. When we update the address, we need to make sure that the record is marked as dirty; otherwise it will not be uploaded to the Palm.

When &DoFetch returns, ConduitMain writes $PDB to the file given by $HEADERS{"OutputDB"} and exits. Then, during the main sync, ColdSync will upload to the Palm any records pine-aliases has modified.

&DoDump

The &DoDump function implements the Dump conduit: 

sub DoDump
{
        open ALIASES, "< $PINE_ALIASES" or
                die "502 Can't read $PINE_ALIASES: $!\n";
        open ALIASES_NEW, "> $PINE_ALIASES.new" or
                die "502 Can't write $PINE_ALIASES.new: $!\n";

        while (<ALIASES>)
        {
                chomp;

                my $alias;
                my $addr;
                my $fullname;
                my @rest;
                my $record;

                ($alias, $fullname, $addr, @rest) = split /\t/;

                $record = &find_person($PDB, $fullname);
                if (!defined($record))
                {
                        # This name doesn't appear in $PDB.
                        print ALIASES_NEW $_, "\n";
                        next;
                }

                # This person appears in both the alias file and in
                # the PDB.
                my $pdb_addr = &get_address($record);

                if (defined($pdb_addr))
                {
                        # Found an address
                        print STDOUT "101 $fullname -> $pdb_addr\n"
                                if $pdb_addr ne $addr;
                        print ALIASES_NEW
                                join("\t", $alias, $fullname,
                                        $pdb_addr, @rest),
                                "\n";
                        next;
                }

                # The PDB record doesn't have an email address. Mark it
                # as deleted.
                my $year;
                my $month;
                my $day;

                ($year, $month, $day) = (localtime)[5,4,3];
                $year %= 100;
                $month++;

                $alias = sprintf "#DELETED-%02d/%02d/%02d#%s",
                                $year, $month, $day, $alias;

                print ALIASES_NEW
                        join("\t", $alias, $fullname, $addr, @rest),
                        "\n";
        }

        close ALIASES_NEW;
        close ALIASES;
        rename "$PINE_ALIASES.new", $PINE_ALIASES or
                die "Can't rename $PINE_ALIASES.new: $!\n";

        return 1;               # Success
}

In &DoDump, we read each line of `~/.addressbook' in turn and write a possibly-update version to `~/.addressbook.new'. The reasons for using two files is twofold: first of all, the length of a line might change, so we can't just update the file in place. Secondly, if anything goes wrong during the sync, we can simply abort before moving the new file into place, and leave the old alias file untouched, rather than risk corrupting it.

Again, we use &find_person to look up the Palm record corresponding to a person's full name, and &get_address to extract the email address from the record. There are three cases we need to consider:

Helper functions

These are the helper functions used in pine-aliases.

&find_person takes a reference to a Palm::Address and a full name, and returns a reference to the record corresponding to that name: 

sub find_person
{
        my $PDB = shift;
        my $fullname = shift;
        my $record;

        foreach $record (@{$PDB->{"records"}})
        {
                next unless ($record->{"fields"}{"firstName"} . " " .
                        $record->{"fields"}{"name"}) eq $fullname;
                return $record;
        }
        return undef;           # Failure
}

Since Palm Address Book records don't contain a full name field, we construct one from the first and last names, and see if it matches.

Note that a better version of this function would also consider other fields: an entry such as "Ooblick Technical Support" might be listed on the Palm with no first or last name, but with the company field set to "Ooblick" and the title field set to "Technical Support".

&get_address takes a reference to a Palm record, and extracts the email address, if any: 

sub get_address
{
        my $record = shift;
        my $field;

        # Look through all of the "phone*" fields
        foreach $field ( qw( phone1 phone2 phone3 phone4 phone5 ) )
        {
                next unless $record->{"phoneLabel"}{$field} == 4;

                # Found the (or an) email field
                my $addr = $record->{"fields"}{$field};

                $addr =~ s/\n.*//;      # Keep only first line

                # Remove parenthesized expressions
                $addr =~ s/\([^\)]*\)//;
                $addr =~ s/^\s+//;      # Remove leading whitespace
                $addr =~ s/\s+$//;      # and trailing whitespace

                return $addr;
        }

        return undef;           # Couldn't find anything
}

This was made into a separate function for clarity: the Palm Address Book record format does not contain a separate field for the email address. Rather, it has five fields named phone1 through phone5, each of which can be a home phone, work phone, fax number, email address, etc. See Palm::Address(1) for details.

&get_address looks at each phone field in turn until it finds one whose phoneLabel is 4, meaning "Email". It extracts the useful part of the address and returns it.

Note that this function is very simplistic: all it does is remove the parentheses from addresses of the form 

JDoe@ooblick.com (John Doe)

The general case is much more complex.

&set_address is the converse of &get_address: it stores an email address in a record: 

sub set_address
{
        my $record = shift;
        my $addr = shift;
        my $field;

        # Find the Email phone field
        foreach $field ( qw( phone1 phone2 phone3 phone4 phone5 ) )
        {
                next unless $record->{"phoneLabel"}{$field} == 4;

                # Found it.
                $record->{"fields"}{$field} = $addr;
                $record->{"attributes"}{"dirty"} = 1;
                return;
        }

        # No Email field found.
        foreach $field ( qw( phone1 phone2 phone3 phone4 phone5 ) )
        {
                next if $record->{"phoneLabel"}{$field} =~ /\S/;

                # Found an empty field
                $record->{"phoneLabel"}{$field} = 4;
                $record->{"labels"}{$field} = $addr;
                $record->{"attributes"}{"dirty"} = 1;
                return;
        }

        # No Email fields, and no empty fields. Fail silently.
        return;
}

Again, due to the format of Palm Address Book records, this function is more complicated than it seems that it ought to be.

In the simplest case, we look at all of the phone fields, find one marked "Email", and update it.

If there is no email field, &set_address tries to find an empty field and turn it into an email field, then writes the address to that field.

If there are no empty fields, we'll give up, since this is just a tutorial. A real conduit ought to keep trying: it might consider adding the email address to the "Other" phone field, if there is one. As a last resort, it might add the email address to the note. Of course, &get_address also needs to know about all of the places where an email address might lurk.

In any case, &set_address marks the record as being dirty, so that it will be uploaded to the Palm at the next sync.

Limitations of pine-aliases

The conduit we've just seen is just a tutorial. For the sake of simplicity, we've ignored several real-world considerations that would have made the code even harder to read.

The first simplifying assumption we've made is that there is only one email address per person. In the real world, people often have a home address and a work address. To deal with this, &DoFetch should collect an array of addresses for each person, then make sure that each address in the array exists in the Palm record (this is why &DoFetch is split up into two phases).

One issue that complicates matters is that a Palm Address Book record might contain multiple phone fields marked "Email". &get_address ought to handle this case. The other side of the issue is that &set_address shouldn't just dump all of the email addresses into the first "Email" phone record that it finds, otherwise the second and subsequent addresses will be duplicated.

Secondly, we've assumed that each full name uniquely identifies a single person. This obviously fails if the user knows two people named John Smith. In the case of pine-aliases, we can get away with documenting this limitation and requiring the user to list one of them as "John Allan Smith" and the other as "John Paul Smith". We might also consider setting up a separate file that maps Pine mail aliases to Palm record IDs, since those are unique identifiers in their respective domains.

Finally, &set_address shouldn't fail so easily: if it fails to add an email address to the record, then at the next sync, the corresponding Pine alias will be commented out. If a record is so full that there are no empty phone records, then obviously it's very important, and the user would be rather upset at losing this email address.

Style and Warnings: things to watch out for

The conduit presented above is very simple, and does not address many problems you will run into when writing "real" conduits.

Go to the first, previous, next, last section, table of contents.