Chapter 5. QSF - QOF Serialization Format.

5.1. What is QSF?

After some discussion on the GnuCash lists, an XML serialization format has been created - i.e. it lays out a QOF object in a series of XML tags. The format will be QSF - QOF Serialization Format and will consist of two component formats:

  1. qof-qsf for the QSF object data and

  2. qsf-map to map sets of QSF objects between QOF applications.

QSF object files will contain user data and are to be exported from QOF applications under user control or they can be hand-edited. QSF maps contain application data and can be created by application developers from application source code. Tools may be created later to generate maps interactively but maps require application support as well as an understanding of QOF objects in the import and output applications and are intended to remain within the scope of application developers rather than users.

A QSF file written by one QOF application will need an appropriate QSF map before the data can be accessed by a different application using QOF. Any QSF objects that are not defined in the map will be ignored. QSF files written and accessed by the same application can use maps if necessary or can simply import the QSF data as a whole.

I've created a Schema that covers all QSF files - it's generic and uses tags based only on the QOF data types, so until new types are defined, it should cope with all QOF XML needs.

5.1.1. Features of QSF

  • QSF itself is now being built into the QOF library for use with pilot-link to allow Palm objects to be described in QOF, written to XML as QSF and imported directly into GnuCash and other QOF-compliant applications. As a generic format, it does not depend on any pre-defined objects - as the current GnuCash XML format depends on AccountGroup. Instead, QSF is a simple container for all QOF objects.

  • QSF grew from the qof_book_merge code base and uses the qof_book_merge code that is now part of QOF. Any QofBook generated by QSF still needs to be merged into the existing application data using qof_book_merge.

  • QSF can be used as an export or offline storage format for QOF applications (although long term storage may be best performed using separate (non-XML) methods, depending on the application).

  • QSF is designed to cope with partial QofBooks at the QofObject level. There is no requirement for specific objects to always be defined, as long as each QOF object specified is fully defined, no orphan or missing parameters are allowed.

  • Work is continuing on supporting QSF in GnuCash and QOF. QSF is a very open format - the majority of the work will be in standardising object types and creating maps that convert between objects. Applications that read QSF should ignore any objects that do not match the available maps and warn the user about missing data. The example file outlines three objects defined in pilot-link and in the pilot-qsf-GnuCashInvoice.xml QSF map to retrieve the data from pilot-link to create a GnuCash invoice. The range of QSF objects and QSF maps will then be increased from that base.

  • Anyone is free to create their own QSF objects, subject to the GNU GPL. It is intended that QSF can be used as the flexible, open and free format for QOF data - providing all that is lacking from a typical CSV export with all the validation benefits of XML and the complex data handling of QOF. The QSF object and map formats remain under the GNU GPL licence and QSF is free software.

5.1.2. Requirements of QSF

  • The data is described as genuine QOF objects,

  • References are written as the GUID of the parent or child

  • The type attribute of the object tag must match an existing QOF object e_type string so that the object can be matched, either against the same QOF object or against a QSF map.

  • The full description of the object, as defined in the source code for that QOF object, should be listed as a string parameter within the object, using the same type as the object itself. This may seem like duplication, but it provides a valuable check on object identity.

  • Object parameters are written out in a defined sequence:

    1. All string parameters, any order

    2. All GUID parameters, including references to parent or child objects, in any order.

    3. all boolean parameters, any order

    4. all numeric parameters, any order

    5. all date parameters, any order

    6. all int32 parameters, any order

    7. all int64 parameters, any order

    8. all double parameters, any order

    9. all char parameters, any order

    10. The KVP frame will be supported soon.

    Note that KVP is not included in QSF at this stage, but support is being organised.

    The example XML file demonstrates this pattern.

  • The XML itself is valid, according to the QSF schema. Keep any comments to between the xml version and the first qof-qsf tag or after the closing qof-qsf tag.

  • QSF uses xsd:dateTime formatting for times. This imposes strict limits on the formatting of time strings in the QSF XML. xsd:dateTime follows the ISO 8601 standard in the Coordinated Universal Time (UTC) syntax to be timezone independent. This generates timestamps of the form: 2004-11-29T19:15:34Z - you can reproduce the same timestamps with the GNC C Library formatting string %Y-%m-%dT%H:%M:%SZ - remember to use gmtime() NOT localtime()!. From the command line, use the -u switch with the date command: date -u +%Y-%m-%dT%H:%M:%SZ

  • Every QSF file must contain at least one book and at least one QOF object but can contain more - including QOF objects from more than one QOF application.

  • A QSF file written by one QOF application will need an appropriate QSF map before the data can be accessed by a different application using QOF. Any QSF objects that are not defined in the map will be ignored. QSF files written and accessed by the same application can use maps if necessary or can simply import the QSF data as a whole.

5.1.3. Validation of QSF.

All QSF files must be valid XML according to the schema above, but XML validation will NOT be the only restriction on QSF data. XML files that validate against the schema will not necessarily be importable into any QOF application because the application needs a qsf-map to understand the objects defined in the QSF. The map is selected according to the object type, expressed in the type attribute of the object tag in the XML.

The example file therefore contains pilot-expenses, pilot-datebook and pilot-address objects. If the receiving application only had a map defined for pilot-expenses, the other two objects would be ignored and the user warned. Note that any one QSF file can contain data from different objects. If a suitable map exists, there is nothing to prevent one QSF file containing objects from different QOF programs.

5.1.4. QSF examples

  • Example XML file using pilot-link objects. More objects will be added as support improves.

  • Second Example using GnuCash objects. Note that the range of GnuCash objects is limited initially and some parameters are not yet fully supported, notably KVP as well as objects that are currently unsupported by QOF itself, like commodity and pricedb. This QSF example was hand edited (which is why it is small and consists solely of account objects!) and final QSF files written by QOF/GnuCash are likely to contain more parameters for each supported object. e.g. in a qof_book_merge, typical GnuCash Account objects have eight parameters. The actual strings used for the type attributes can be adjusted in the GnuCash source. It may be possible to have an XSLT stylesheet to do more transformations. Note also how references are described as obj:guid with type="reference", type=guid" being reserved for the guid of the object itself. Overall, QSF may add a small overhead to the existing GnuCash XML format but this is not anticipated as a problem - QSF is intended as an intermediary format between applications, not long term storage.

  • The GnuCash XML v2 file for the second example QSF above. Note how the QSF file uses the one object tag and obj namespace for all supported objects. The obj namespace tags are standardised to the underlying QOF type for that parameter and the type takes on the descriptive / human readable name for the QOF parameter. Note also the strict sequence in a QSF file - string, GUID, boolean, numeric, date, int: (SGBNDI). No neat acronym but still important!

  • qsf-object.xsd.xml - The QSF Object Schema.

    qsf-map.xsd.xml - The QSF Map Schema.

A simple parser written in C is also available, based on a tutorial for libxml. This has been adapted to remove all dependencies on Gnome or Gtk and enhanced to cope with all QOF data types currently supported by QSF.

A reader utility in C using the Sixtp parser used by GnuCash was tested but proved inadequate for the QSF format - it would appear not to be possible for a single tag to be both parsed and contain children. The test file that was produced contained a large amount of unnecessary code and the parser itself added unnecessary bulk to the routines. QSF - as implemented in QOF and GnuCash - will therefore use only direct calls to libxml and QOF itself.

To compile qsf-xml.c, use the command:

gcc -g -Wall `xml2-config --cflags --libs` -o qsf-xml qsf-xml.c

The default looks for the qof-qsf.xml file in the current directory. It is also capable of reading the qsf-gnc.xml file above. Schema validation has also been added to the test routine - it looks for qsd.xsd in the current directory. When finished, it will look in install_dir/share/qsf/.