1.1. Terms and definitions.

1.2. Other versions of this documentation.

This documentation created using DocBook http://www.docbook.org/tdg/en/html/docbook.html. This allows other versions to be created from the one source.

The permanent URL for the HTML version of this document is: http://code.neil.williamsleesmill.me.uk/.

So far, there is a PDF version: http://code.neil.williamsleesmill.me.uk/qof_book_merge.pdf.

Also a single HTML file .

On request, TXT, TEXI, LaTeX, DVI, RTF and PostScript versions can be generated, or a HTML version with all pages in one file. A tarball of this DocBook and associated XML files is also available:

http://code.neil.williamsleesmill.me.uk/qof_book_merge.tar.gz

3.1. Doxygen Documentation.

3.2. Other general documentation

3.3. qof_book_merge Source Code

GnuCash CVS: http://cvs.gnucash.org/cgi-bin/cvsweb.cgi/gnucash/src/engine/

• qof_book_merge.c

• qof_book_merge.h

• Test and example programs. This test file shows an implementation of the qof_book_merge API:

  test-book-merge.c

  This example files shows an implementation using GnuCash objects:

  example-gncBookMerge.c

  • The example now uses external XML files as data sources:

    druid-business.gnc

    druid-simple.gnc

    These are simple GnuCash files, created using the New File druid.

• A GUI example is also under development - to allow the New Account Hierarchy druid to be run over an existing file.

  • druid-merge.c - small changes required to main-window.c to allow for the menu item to call the druid.

    merge.glade - the Glade code to describe the druid resource.

3.4. QSF Source code

qsf-backend.c

qsf-dir.h.in

qsf-xml.c

qsf-xml.h

qsf-xml-map.c

4.1. Beginnings

This is where this entire saga began. I needed to collate data from my Palm PDA (using Palm OS 5) and create an invoice in GnuCash with the minimum of extra typing. The PDA databases contain details of the hours to be billed (Calendar), incidental expenses incurred during the invoiced work (Expenses) and contact details for the client and workplace (Contacts). By adding a user-configurable set of defaults, the other essential elements of an invoice are ready, item type (hours, material etc.), accounts to use for each type and the account to use for posting the invoice, if the user chooses to let the program do the post. (Not advisable until the user is fully familiar with the program!) This area is now under development.

The mechanism was not considered at that stage. I didn't really care how the data would be passed from PDA to GnuCash and considered XML/XSLT, CSV and QIF before agreeing to do the hard bit of writing the generic merge routine. The benefits are huge but the work has been long and a steep learning curve. The main reason for writing this documentation is to see if it can help others join the GnuCash team. Since starting qof_book_merge, the Doxygen source code documentation has been rendered directly from the CVS code on a nightly basis. These pages are therefore designed to cover the other questions; where did I start, how can new developers get under the skin of GnuCash quickly and without hassling other developers as I did!

• Initial discussions about Invoices from a PDA.

• Discussions about a merge function for GNCBook*, a synonym for QofBook.

• My first mistakes.

• Learning QOF.

• Function pointer problems.

• QofSetterFunc changes to aid qof_book_merge.

4.2. Getting into GnuCash / QOF

This section is meant to be helpful and useful - nothing here is intended as a rule or to be interpreted as the only way to do things. It is not meant as patronising either, I made mistakes when starting this mini-project because this kind of project was so unfamiliar to me.

The critical areas for me are highlighted above. The up to date source documentation provided by Doxygen is always the first place to look for answers. General overviews and design texts do exist but you need to use and understand the program before most of this will make any sense. All documentation must choose a starting level and mine will mimic my own start:

• Experience of C++ more than C - able to write and compile console programs, design C++ classes and implement inheritance. Includes some gaps in C knowledge, specifically function pointers.

• A real need, not just curiosity. GnuCash is a large program and trying to learn it all is more than most people need. To pick up GnuCash, QOF and the vagaries of using only C, I can only recommend a clear objective that relates to an existing GnuCash objective - you can't expect a lot of assistance from GnuCash developers if you are not working on a project that is seen to be useful to the rest of GnuCash.

• Experience of GnuCash itself, preferably in situations where the figures are real and the calculations matter. A tax return focuses the mind very well.

• Time - always more than you had originally planned.

• Patience. If you don't have much patience, you probably wouldn't still be reading!

• http://www.gnucash.org/. Read the "Developer Information" section and join the gnucash-devel mailing list.

• Make sure what you want GnuCash / QOF to do coincides with what the rest of the developers consider useful. Explain your ideas and like my experience above, be prepared to:

  1. Alter your own plans substantially.

  2. Provide good reasons for not altering the bits you want to keep.

  3. Acknowledge your strengths AND weaknesses and ask for help.

  4. Keep to what you know you can achieve - don't be tempted into taking on a larger project than you can manage.

  5. Keep it simple, concise and informative - other developers are busy too.

• Download the CVS HEAD and build it.

• Spend lots of time understanding the current Doxygen output.

There are lots of sites that have helped me understand QOF, GnuCash, automake, doxygen, glib, libxml and Glade. In the hope that some might be useful . . .

• http://qof.sourceforge.net/

• http://www.gnu.org/manual/manual.html GNU Manuals Online, autoconf, autogen and automake.

• http://developer.gnome.org/doc/API/2.0/glib/index.html GLib Reference Manual - GLib is a general-purpose utility library, which provides many useful data types, macros, type conversions, string utilities, file utilities, a main loop abstraction, and so on. It works on many UNIX-like platforms, Windows, OS/2 and BeOS. GLib is released under the GNU Library General Public License (GNU LGPL).

• QOF does NOT require Gtk or any GUI elements. These are provided to aid GnuCash development.

  • http://developer.gnome.org/doc/API/2.0/gtk/index.html GTK+ Reference Manual

  • http://glade.gnome.org/ Glade is a free user interface builder for GTK+ and GNOME. It is released under the GNU General Public License (GPL).

• http://www.yolinux.com/TUTORIALS/GnomeLibXml2.html YoLinux Tutorial - XML and Gnome libXML2

• http://www.w3.org/TR/xmlschema-0/primer.html#Intro XML Schema Part 0: Primer Second Edition, Introduction.

• http://www.xmlsoft.org/ The XML C parser and toolkit of Gnome.

• http://www.eskimo.com/~scs/cclass/int/sx10a.html Declaring, Assigning, and Using Function Pointers.

4.3. Building QOF onto pilot-link

• qof-expenses.h

• qof-expenses.c

• qof-address.h

• qof-address.c

• qof-datebook.c

• qof-datebook.h

The principle is to wrap the existing struct inside a new QOF-compatible struct. The essential addition is QofInstance inst; then define a get and a set routine for each parameter to be supported by QOF. In each routine, cast the address of the wrapped struct to a suitable pointer and use that to store or retrieve the real data via QOF.

	Localtime is determined by the process running QOF.
	There are also issues of dates and times between QOF and pilot-link. Pilot-link usually works on localtime - the Palm only sets the timezone in Preferences, it doesn't use it internally - so the link to QOF will have to make this crucial assumption. When converting Palm localtimes into QOF universal (UTC) times, the time will be converted by QOF using the localtime of the QOF process.

In real terms, QOF will be running as a framework around pilot-link so the offline storage, (QSF XML), will be written by pilot-link. Pilot-link itself does not directly support users who cross timezones with their Palm - it may be wise that users are reminded that if timezone issues are important to their DateBook or Expenses data, the user should take the sensible precaution of changing the Palm Preferences to use the same timezone as the computer running pilot-link. Once in QOF, all dates and times are UTC so the QSF XML files can be freely exchanged across timezones without complication.

QOF has now been configured to build as a framework around pilot-link when compiled from source. Using the --enable-qof=<path> option to ./autogen.sh, the QOF code is built into libpilotqof.so and a pilot_qof executable will be installed. Full documentation for pilot_qof is included in this documentation as manpages for pilot_qof.

The ability to import QSF data directly into the Palm will follow at a later date.

4.4. Pilot-QOF

Pilot-QOF is an application to use QOF with pilot-link which provides access to Palm application data. Pilot-QOF currently provides access to Expenses, ToDo (Tasks), Contacts (Addressbook) and Appointment (Datebook/Calendar). It can be extended to provide access to data in ANY Palm application. Naturally, it is considerably easier to add open source Palm applications like FreeCoins or PCash. Proprietary Palm applications can also be supported, depending on cooperation from the proprietary developers to provide details of the relevant database structure for the user data created by the application.

5.1. What is QSF?

After some discussion on the GnuCash lists, an XML serialization format has been created - i.e. it lays out a QOF object in a series of XML tags. The format will be QSF - QOF Serialization Format and will consist of two component formats:

1. qof-qsf for the QSF object data and

2. qsf-map to map sets of QSF objects between QOF applications.

QSF object files will contain user data and are to be exported from QOF applications under user control or they can be hand-edited. QSF maps contain application data and can be created by application developers from application source code. Tools may be created later to generate maps interactively but maps require application support as well as an understanding of QOF objects in the import and output applications and are intended to remain within the scope of application developers rather than users.

A QSF file written by one QOF application will need an appropriate QSF map before the data can be accessed by a different application using QOF. Any QSF objects that are not defined in the map will be ignored. QSF files written and accessed by the same application can use maps if necessary or can simply import the QSF data as a whole.

I've created a Schema that covers all QSF files - it's generic and uses tags based only on the QOF data types, so until new types are defined, it should cope with all QOF XML needs.

5.2. Mapping QSF objects between QOF applications

1:  <definition qof_version="3">
2:    <define e_type="qof-expenses">Pilot-link QOF expenses</define>
3:    <define e_type="qof_datebook">Pilot-link QOF datebook</define>
4:    <define e_type="gncInvoice">Invoice</define>
5:    <define e_type="Trans">Transaction</define>
6:    <define e_type="gncEntry">Order/Invoice/Bill Entry</define>
7:    <default name="mileage_rate" type="numeric" value="28/100"/>
8:  </definition>
9:  <object type="gncEntry">
10:   <calculate type="string" value="action">
11:   <if type="qof-expenses">
12:     <set>Material</set>
13:   </if>
14:   <else type="qof-datebook">
15:     <set>Hours</set>
16:   </else>
17:   </calculate>
18:   <calculate type="numeric" value="iprice">
19:     <if type="string" value="expense_type">
20:       <equals type="string" value="etMileage">
21:         <set>mileage_rate</set>
22:       </equals>
23:     </if>
24:   </calculate>
25:   <calculate type="string" value="desc">
26:     <if boolean="use_weekday_descriptor">
27:       <set format="%A">qsf_time_string</set>
28:    </if>
29:    <else>
30:      <set object="qof-expenses">expense_vendor</set>
31:    </else>
32:   </calculate>
33: </object>

5.3. Writing new QSF maps

Until tools can be developed to design the map, each QSF map will have to be edited by hand. The process requires detailed knowledge of the QOF objects of both applications as well as a general understanding the source code for each program. This section contains the notes I used to create the pilot-qsf-GnuCashInvoice QSF map.

The expenses QOF object in pilot-link can be described as follows. This text is copied directly from the QofObject and QofClass parameter definitions in the source code.

struct tm date	 		EXP_DATE,	QOF_TYPE_DATE,
enum ExpenseType type		EXP_TYPE,	QOF_TYPE_INT32,
enum ExpensePayment payment	EXP_PAYMENT,	QOF_TYPE_INT32,
int currency			EXP_CURRENCY,	QOF_TYPE_INT32,
char *amount			EXP_AMOUNT,	QOF_TYPE_STRING,
char *vendor			EXP_VENDOR,	QOF_TYPE_STRING,
char *city			EXP_CITY,	QOF_TYPE_STRING,
char *attendees			EXP_ATTENDEES,	QOF_TYPE_STRING,
char *note			EXP_NOTE,	QOF_TYPE_STRING,

enum ExpenseType {
	etAirfare, etBreakfast, etBus, etBusinessMeals,
	etCarRental, etDinner,
	etEntertainment, etFax, etGas, etGifts, etHotel,
	etIncidentals,
	etLaundry,
	etLimo, etLodging, etLunch, etMileage, etOther, etParking,
	etPostage,
	etSnack, etSubway, etSupplies, etTaxi, etTelephone, etTips,
	etTolls,
	etTrain
};

enum ExpensePayment {
	epAmEx, epCash, epCheck, epCreditCard, epMasterCard,
	epPrepaid, epVISA,
	epUnfiled
};

Note that although a struct tm in the code, date is considered as a standard QOF_TYPE_DATE - a Timespec. The use of an integer for currency and a string for the amount could also cause problems in GnuCash.

• Date is used in the query to obtain the expenses for the days covered by the invoice. Date maps to gncInvoice::INVOICE_OPENED

• Type is used to determine any mileage rate, to calculate other reimbursement parameters and to complete the description of this transaction within the invoice. Type(as string) maps to Transaction::TRANS_DESCRIPTION. Type (as an enum) is used to retrieve the rate that maps to gncEntry::ENTRY_IPRICE.

• Payment indicates the method of payment. This can be incorporated into the description. In other circumstances, it would be used to set the GnuCash account to be debited but that will not be used for my test case involving invoices. Other maps can use this field as the user sees fit.

• currency needs experimentation to see how it is used by pilot-link and how it can be translated into GnuCash. With gncCommodity still being a problem within QOF, this may simply be left to the GnuCash default.

• Amount - interesting that it is used as a string, it will need careful conversion to a gnc_numeric. This will be used as the Amount of the expenses transaction within the invoice. Currency problems could reappear here too. Amount maps to gncEntry::ENTRY_QTY.

• Vendor - this may appear optional but could be an important check that the correct expense is being allocated to the correct invoice by comparing with the gncJob or gncCustomer. Not directly mapped.

• City - again, can be used to check that the correct expense has been identified by date.

• Attendees - not used in this test map, may be useful to others.

• Note - not used in this test map, may be useful to others.

Datebook is defined as follows (items not used in the map omitted):

int event; /* Is this a timeless event?  */
	DATEBOOK_EVENT, QOF_TYPE_INT32,
struct tm begin; /* When does this appointment start? */
	DATEBOOK_BEGIN, QOF_TYPE_DATE,
struct tm end; /* When does this appointment end? */
	DATEBOOK_END, QOF_TYPE_DATE,
enum repeatTypes repeatType;
/* How should this appointment be repeated, if at all?*/
	DATEBOOK_REPEAT_TYPE, QOF_TYPE_INT32,
int repeatForever;
/* Do the repetitions end at some date or carry on forever? */
	DATEBOOK_REPEAT_FOREVER, QOF_TYPE_INT32,
	(this may be changed to QOF_TYPE_BOOLEAN).
struct tm repeatEnd;	/* What date do they end on? */
	DATEBOOK_REPEAT_END, QOF_TYPE_DATE,
int repeatFrequency;
/* Should I skip an interval for each repetition? */
	DATEBOOK_REPEAT_FREQUENCY, QOF_TYPE_INT32,
enum DayOfMonthType repeatDay;	/* for repeatMonthlyByDay  */
	DATEBOOK_REPEAT_DAY, QOF_TYPE_INT32,
int repeatWeekstart;
/* What day did the user decide starts the week? */
	DATEBOOK_REPEAT_WEEK_START, QOF_TYPE_INT32,
int exceptions;
/* How many repetitions are there to be ignored? */
	DATEBOOK_EXCEPTIONS, QOF_TYPE_INT32,
struct tm *exception;	/* What are they? */
	DATEBOOK_EXCEPTION, QOF_TYPE_DATE,
char *description;
/* What is the description of this appointment? */
	DATEBOOK_DESCRIPTION, QOF_TYPE_STRING,
char *note;	/* Is there a note to go along with it?    */
	DATEBOOK_NOTE, QOF_TYPE_STRING,

Timeless events are anniversaries, birthdays or other marker events that are set in the PDA as 'No time'. If event is set, the datebook object needs to be ignored for the purposes of an invoice.

begin and end will be used to calculate the number of hours to charge on the invoice. end - begin maps to gncEntry::ENTRY_QTY.

Repeating events may need care to identify the actual date of the event. Check the repeatType: repeatNone, repeatDaily, repeatWeekly, repeatMonthlyByDay, repeatMonthlyByDate or repeatYearly. Check the repeatForever value and the repeatEnd Timespec. Check the repeatFrequency, repeatDay and repeatWeekStart only if appropriate. If still within the repeatEnd, check if any dates are to be ignored and find out which ones. Finally, if the event survives all checks, map to gncInvoice::INVOICE_OPENED and gncEntry::ENTRY_DATE, gncEntry::ENTRY_DATE_ENTERED. (Other maps are free to map to just one or two of those.)

Description should match with the gncJob but this is a simple check.

Some items are not read by pilot-link as of 0.11.8. There is a category setting and a Location setting. User intervention will be required to confirm that the correct event has been selected, in the absence of either of these settings, if there is any mis-match in the Description.

Finally for this map, the AddressBook object is defined as a list of all strings:

(concat(ADDR_FIRST_NAME,ADDR_LAST_NAME)	<==> ADDRESS_NAME ||
ADDR_COMPANY,				<==> ADDRESS_NAME)
ADDR_PHONE_ONE,		<==> 	ADDRESS_PHONE	(if regexp matches)
ADDR_PHONE_TWO,		<==>	ADDRESS_FAX	(if regexp matches)
ADDR_PHONE_THREE,	<==>	ADDRESS_EMAIL	(if regexp matches)
ADDR_PHONE_FOUR,
ADDR_PHONE_FIVE,
ADDR_ADDRESS,		<==>	ADDRESS_ONE
ADDR_CITY,		<==>	ADDRESS_TWO
ADDR_STATE,		<==>	ADDRESS_THREE
ADDR_ZIP,		<==>	ADDRESS_FOUR	(if regexp matches)
ADDR_COUNTRY,
ADDR_TITLE,
ADDR_CUSTOM_ONE,
ADDR_CUSTOM_TWO,
ADDR_CUSTOM_THREE,
ADDR_CUSTOM_FOUR,
ADDR_NOTE,
ADDR_CATEGORY,

Note how this map is not fixed. In the last summary, I've tried to summarise the process using familiar symbols. The map will be intelligent and will always check the incoming content against a regular expression of what the mapped field normally contains. In most cases, an address is only created once. More commonly, the details of the address will be used to check against the existing addresses to locate the correct customer. As invoices will be almost always new items, a GUID check is not practical. It will also be possible to map the custom fields on a per-user basis, perhaps to contain the rates to charge etc.

6.1. Preparing the rule set

The rule set for qof_book_merge needs to be open and extendable. The difficulty with this method is handling errors. If the rule set is wrong, errors at compile and debug time could be difficult to trace because qof_book_merge needs to rely on the rule set to decide which variables can be assigned to which structs. These errors will appear the same as errors arising from invalid values in existing structures. In order to allow qof_book_merge to run smoothly in the final program, the rule set needs to be definitive and robust. Each object declares its own parameters in the qofclass object definitions.

These parameters include:

• The QOF object type. A unique name for the object that identifies all other objects of the same type in any QofBook.

• A full list of all useful parameters held in the object. Note that this list is crucial to qof_book_merge but is actually defined by the object developer. qof_book_merge can only work with objects that describe themselves to QOF clearly and with all necessary parameters sensibly defined. This is the price of a generic merge.

• The QOF type of each parameter. The type is essential if qof_book_merge is to know how to compare and merge the parameter data into the target QofBook.

• Templates for functions. To be used to get and set suitable parameters. This was the hardest part of the merge operation for me to understand initially. I was unused to typed function pointers and with GnuCash using C, not C++, casting a function pointer to retrieve a parameter value from an object without knowing in advance what object is being queried, seemed like a black art. I missed the virtual keyword like never before!

The merge code needs to determine the type of object, the relative importance of the data and to set a sequence for the query that precedes the match. In any merge, some parameter data is inconsequential to the final result of the merge. Values that relate only to the import environment and values that are calculated from other parts of the import environment need to be recalculated in the final merge; the actual value of the calculation in the import is irrelevant to the final calculation after the merge. What matters is that all relevant components of the calculation are retained and merged into any existing components in the final book. Merging books necessarily causes the balance of merged accounts to be different from the balance of the import or the pre-merge book. The final value should be consistent with the pre-merge values (by addition and/or subtraction), but cannot be set explicitly, it must be determined from the real data as it exists after the merge.

A simple premise is therefore used to determine the relative importance of parameters within an object:

	If a parameter cannot be set, it is ignored by the merge.
	(The object is still compared using other parameters).

This is simply because a value that does not have a set function declared in the object definition should be defined as a calculated value or similar that should never be set. e.g. An account holds many transactions, each with a value. Although the value of a transaction clearly does need to be set, the balance of the account is a calculated value, dependent on the transactions contained within the account. Setting an account balance artificially would generate invalid data. So although the balance of an account can be retrieved (it has a get function), the lack of a set function determines that the balance parameter will not be compared as part of the merge. The comparison is based on the other parameters for that object. Two identical accounts are still identical objects, even if they contain different transactions and therefore a different balance. The transactions differ, the account does not. To consider a bank account or credit card account, each month the transactions and the balance change but the account is still the same account.

This clearly illustrates the need for objects to be described sensibly and logically! It also allows qof_book_merge to be used with other kinds of objects in the wider QOF design - the parameters that determine the details of the merge are set by the implementation of the code, rather than the code itself, subject to the implementation remaining a valid QOF environment.

In all rules, the prescence or lack of a GUID must be of primary importance in any merge because an exact match in GUID allows qof_book_merge to import the attached data, overwriting that section of the existing QofBook.

qof_book_merge only accepts input data as a QofBook* structure and therefore all import objects will be assigned a new GUID if one does not exist already when the data is inserted into the import QofBook. If the GUID in the imported data does not match the target QofBook*, qof_book_merge must use the rule set to proceed to match the contained data. A sequence of equality checks are required against existing data. In allowing fast processing of these checks, the rule set must balance the desire to not generate thousands of collision queries for the user to resolve against the need to limit the amount of manual editing of the final file.

6.2. Draft of a rule set framework.

The rule set itself is built from the QOF registration definitions of each component of the book. GnuCash uses QOF to provide all internal query functions. QoF in turn uses definitions within the code of each object to determine the kind of data contained in the object and how that data can be addressed and modified.

static QofObject commodity_table_object_def =
{
  interface_version: QOF_OBJECT_VERSION,
  e_type:            GNC_ID_COMMODITY_TABLE,
  type_label:        "CommodityTable",
  book_begin:        commodity_table_book_begin,
  book_end:          commodity_table_book_end,
  is_dirty:          NULL,
  mark_clean:        NULL,
  foreach:           NULL,
  printable:         NULL,
};

gboolean
gnc_commodity_table_register (void)
{
  gnc_quote_source_init_tables();
  return qof_object_register (&commodity_table_object_def);
}
	

A more complex definition will help to show how the rule sets from the first draft will be applied into the existing QoF structures:

static QofObject gncInvoiceDesc =
{
  interface_version:  QOF_OBJECT_VERSION,
  e_type:             _GNC_MOD_NAME,
  type_label:         "Invoice",
  book_begin:         NULL,
  book_end:           NULL,
  is_dirty:           qof_collection_is_dirty,
  mark_clean:         qof_collection_mark_clean,
  foreach:            qof_collection_foreach,
  printable:          _gncInvoicePrintable,
};
gboolean gncInvoiceRegister (void)
{
  static QofParam params[] = {
    { INVOICE_ID, QOF_TYPE_STRING, (QofAccessFunc)gncInvoiceGetID, NULL },
    { INVOICE_OWNER, GNC_ID_OWNER, (QofAccessFunc)gncInvoiceGetOwner, NULL },
    { INVOICE_OPENED, QOF_TYPE_DATE, (QofAccessFunc)gncInvoiceGetDateOpened, NULL },
    { INVOICE_DUE, QOF_TYPE_DATE, (QofAccessFunc)gncInvoiceGetDateDue, NULL },
    { INVOICE_POSTED, QOF_TYPE_DATE, (QofAccessFunc)gncInvoiceGetDatePosted, NULL },
    { INVOICE_IS_POSTED, QOF_TYPE_BOOLEAN, (QofAccessFunc)gncInvoiceIsPosted, NULL },
    { INVOICE_IS_PAID, QOF_TYPE_BOOLEAN, (QofAccessFunc)gncInvoiceIsPaid, NULL},
    { INVOICE_BILLINGID, QOF_TYPE_STRING, (QofAccessFunc)gncInvoiceGetBillingID, NULL },
    { INVOICE_NOTES, QOF_TYPE_STRING, (QofAccessFunc)gncInvoiceGetNotes, NULL },
    { INVOICE_ACC, GNC_ID_ACCOUNT, (QofAccessFunc)gncInvoiceGetPostedAcc, NULL },
    { INVOICE_POST_TXN, GNC_ID_TRANS, (QofAccessFunc)gncInvoiceGetPostedTxn, NULL },
    { INVOICE_POST_LOT, GNC_ID_LOT, (QofAccessFunc)gncInvoiceGetPostedLot, NULL },
    { INVOICE_TYPE, QOF_TYPE_STRING, (QofAccessFunc)gncInvoiceGetType, NULL },
    { INVOICE_TERMS, GNC_ID_BILLTERM, (QofAccessFunc)gncInvoiceGetTerms, NULL },
    { INVOICE_BILLTO, GNC_ID_OWNER, (QofAccessFunc)gncInvoiceGetBillTo, NULL },
    { QOF_QUERY_PARAM_ACTIVE, QOF_TYPE_BOOLEAN, (QofAccessFunc)gncInvoiceGetActive, NULL },
    { QOF_QUERY_PARAM_BOOK, QOF_ID_BOOK, (QofAccessFunc)qof_instance_get_book, NULL },
    { QOF_QUERY_PARAM_GUID, QOF_TYPE_GUID, (QofAccessFunc)qof_instance_get_guid, NULL },
    { NULL },
  };

  qof_class_register (_GNC_MOD_NAME, (QofSortFunc)gncInvoiceCompare, params);
  reg_lot ();
  reg_txn ();

  return qof_object_register (&gncInvoiceDesc);
}
	

Note that the initial config of Account had a NULL where the QofSetterFunc could have been declared.

A problem arises in Transaction where a transaction is voided. There are three QofAccessFunc routines for each part of the void but only one function that could be used as a QofSetterFunc which sets all three values, as well as a second set function that undoes the void operation.

Invoice: unable to set the due date - no set_fcn?

Lots are also potential problems - this will need to be monitored.

In Jobs, gncJobGetActive is defined in two separate parameters?

The files in use with QofSetterFunc are:

• engine/Account.c

• engine/Transaction.c

• gnc-pricedb.c

• business/business-core/gncInvoice.c

• business/business-core/gncAddress.c

• business/business-core/gncBillTerm.c

• business/business-core/gncCustomer.c

• business/business-core/gncEntry.c

• business/business-core/gncJob.c

• business/business-core/gncOrder.c

• business/business-core/gncOwner.c

• business/business-core/gncTaxTable.c

• business/business-core/gncVendor.c

6.3. Design of the merge

The qof_book_merge code is already documented.

A sample test program would use the three qof_book_merge API routines:

• qof_book_merge *mergeData;
  mergeData = qof_merge_bookInit(QofBook *import, QofBook *target);

  Make sure you check that mergeData is not NULL before proceeding with the user intervention loop. A NULL indicates a failure and your code MUST abort.

• Write your loop to present the output for the user and take the results, ready to pass to qof_book_mergeUpdateResult(). Offer the user options to abort the merge at any/all stages of this loop - nothing in the target book is changed until the code proceeds to the qof_book_mergeCommit stage.

• Only when all REPORT items have been re-assigned to NEW, DUPLICATE or UPDATE should you call qof_book_mergeCommit(mergeData).

  Commit cannot be stopped or undone!

  Remember that the commit is a ONE WAY operation - there is NO going back or "unmerging" the books. Your user involvement routine must make it clear to the user that the user must take responsibility for either making a backup or unpicking the merge manually if they have chosen the wrong import file etc. Commit cannot be aborted - forcing a shutdown carries significant risks of corrupting the target book.

This is the pseudo-code for the Init function where most of the work is done.

Processing starts with Init
	Init calls ForeachType (once for each registered type of object)
		ForeachType calls ForeachParam
			parameter added to list of parameters for the object
		Return to ForeachType
	ForeachType calls Foreach (each entity in turn)
		Foreach creates a new rule and looks up an absolute match in the target book
			using Compare
		If no absolute match, calls ForeachTypeTarget
			Checks that the entity type matches and calls ForeachTarget
				ForeachTarget calls orphan_check to prioritise the match
			returns to ForeachTypeTarget
		returns to Foreach
	Sets target(s) or sets MERGE_NEW
	returns to ForeachType
returns to Init

mergeData = qof_book_mergeInit( QofBook *importBook, QofBook *targetBook)
	qof_object_foreach_type(qof_book_mergeForeachType, mergeData);
		qof_class_param_foreach(merge_obj->e_type, qof_book_mergeForeachParam , mergeData);
	qof_object_foreach(merge_obj->e_type, mergeData->mergeBook, qof_book_mergeForeach, mergeData);
		mergeRule = g_new(qof_book_mergeRule,1);
		targetEnt = qof_collection_lookup_entity (
			qof_book_get_collection (mergeData->targetBook, mergeEnt->e_type), g);
		g_return_if_fail(qof_book_mergeCompare(mergeData) != -1);
		qof_object_foreach_type(qof_book_mergeForeachTypeTarget, mergeData);
			if(safe_strcmp(merge_obj->e_type, currentRule->importEnt->e_type) == 0) {
			qof_object_foreach(currentRule->importEnt->e_type, mergeData->targetBook,
						qof_book_mergeForeachTarget, mergeData);
				qof_book_merge_orphan_check(difference, mergeRule, mergeData);
			}
		}
	mergeData->mergeList = g_slist_prepend(mergeData->mergeList,mergeRule);

Note that qof_book_mergeForeach is the first time real user data is available.

6.4. Using qof_book_merge with new and existing QOF objects

The test-book-merge.c program demonstrates a full and complete qof_class parameter and function description:

static QofObject obj_object_def = {
  interface_version:     QOF_OBJECT_VERSION,
  e_type:                TEST_MODULE_NAME,
  type_label:            TEST_MODULE_DESC,
  create:                (gpointer)obj_create,
  book_begin:            NULL,
  book_end:              NULL,
  is_dirty:              NULL,
  mark_clean:            NULL,
  foreach:               qof_collection_foreach,
  printable:             NULL,
  version_cmp:           (int (*)(gpointer,gpointer)) qof_instance_version_cmp,
};

gboolean myobjRegister (void)
{
  static QofParam params[] = {
{ OBJ_NAME,	QOF_TYPE_STRING,	(QofAccessFunc)obj_getName,	(QofSetterFunc)obj_setName	},
{ OBJ_AMOUNT,   QOF_TYPE_NUMERIC,	(QofAccessFunc)obj_getAmount,   (QofSetterFunc)obj_setAmount	},
{ OBJ_GUID,	QOF_TYPE_GUID,		(QofAccessFunc)obj_getGUID,	(QofSetterFunc)obj_setGUID	},
{ OBJ_DATE,	QOF_TYPE_DATE,		(QofAccessFunc)obj_getDate,	(QofSetterFunc)obj_setDate	},
{ OBJ_DISCOUNT, QOF_TYPE_DOUBLE,	(QofAccessFunc)obj_getDiscount, (QofSetterFunc)obj_setDiscount  },
{ OBJ_ACTIVE,   QOF_TYPE_BOOLEAN,	(QofAccessFunc)obj_getActive,   (QofSetterFunc)obj_setActive	},
{ OBJ_VERSION,  QOF_TYPE_INT32,		(QofAccessFunc)obj_getVersion,  (QofSetterFunc)obj_setVersion   },
{ OBJ_MINOR,	QOF_TYPE_INT64,		(QofAccessFunc)obj_getMinor,	(QofSetterFunc)obj_setMinor	},
{ QOF_PARAM_BOOK, QOF_ID_BOOK,		(QofAccessFunc)qof_instance_get_book, NULL },
{ QOF_PARAM_GUID, QOF_TYPE_GUID,	(QofAccessFunc)qof_instance_get_guid, NULL },
{ NULL },
};

  qof_class_register (TEST_MODULE_NAME, NULL, params);

  return qof_object_register (&obj_object_def);
}
	

The prototypes for each get and set function illustrate the standard mechanisms for use by the merge operation:

/* getter functions */
void obj_setName(myobj*,	char*);
void obj_setGUID(myobj*,	const GUID*);
void obj_setAmount(myobj*,	gnc_numeric);
void obj_setDate(myobj*,	Timespec h);
void obj_setDiscount(myobj*,	double);
void obj_setActive(myobj*,	gboolean);
void obj_setVersion(myobj*,	gint32);
void obj_setMinor(myobj*,	gint64);

/* setter functions */
gnc_numeric	obj_getAmount(myobj*);
char*		obj_getName(myobj*);
const GUID*	obj_getGUID(myobj*);
Timespec   	obj_getDate(myobj*);
double		obj_getDiscount(myobj*);
gboolean	obj_getActive(myobj*);
gint32		obj_getVersion(myobj*);
gint64		obj_getMinor(myobj*);
	

Please note the return types and parameter types. To use the merge operation, your object should use the same prototype for each relevant QOF_TYPE.

In particular, please keep the same pattern for pointer parameters - strings should be char (or gchar) pointers, GUID should be constant pointers and all other types are passed and returned without using pointers.

Declaring functions that use additional parameters, unexpected pointers or other discrepancies is likely to cause a segmentation fault in the merge.

Any parameter that does NOT have a get and a corresponding set function will be ignored by the merge operation - this rule should be used to discriminate between parameters that should only ever be calculated (e.g. account balances) and parameters that must always be set by the backend, user or merge (e.g. account names).

Note that all objects require a foreach AND a create function in the object definition. If any object is not fully QOF compliant, ensure that either the foreach or create functions are NULL or you will get errors when qof_book_merge runs, even if no objects of that type exist in either book.

6.5. Known problems

As a result, qof_book_merge does not currently handle commodities.

A problem arises in Transaction where a transaction is voided. There are three QofAccessFunc routines for each part of the void but only one function that could be used as a QofSetterFunc which sets all three values, as well as a second set function that undoes the void operation.

Invoice: unable to set the due date - no set_fcn?

Lots are also potential problems - this will need to be monitored.

In the following table, when the specified object is merged, data in the specified struct member variables will not be read, compared or set. Existing values in the target QofBook will not be changed. Where the object is QOF compatible, if the object is new (MERGE_NEW) the listed variable will be set to the default value given in the table.

In the current design, a session may hold multiple books. For now, exactly what this means is somewhat vague, and code in various places makes some implicit assumptions: first, only one book is 'current' and open for editing. Next, its assumed that all of the books in a session are related in some way. i.e. that they are all earlier accounting periods of the currently open book. In particular, the backends probably make that assumption, in order to store the different accounting periods in a clump so that one can be found, given another.

6.6. Design improvements

qof_book_merge needs a method to add references to entire QOF objects into other QOF objects as variables. e.g. Account and GncAddress objects are often referenced by other objects. qof_class definitions exist under GNC_ID types and merge will be extended to get and retrieve such object references.

Each GNC_ID is defined in QOF anyway - propose a check on the mergeType before the comparison.

Use the unknown type to reproduce the links that already exist in the import book for objects that contain the unknown type. i.e. Set get_fcn and set_fcn for object parameters that take or return objects that are already registered with QofObject/QofClass. Then when that parameter is compared, the GNC_ID... will fail to match any of the QOF_TYPE... fundamental data types - this is the unknown type. Store the QofEntity of that type within the rule for the 'parent' object.

Later, run a secondary loop to match up the child QofEntity with the qof_book_mergeResult of the actual object, use that to modulate the qof_book_mergeResult of the parent object.

e.g.

{ INVOICE_ACC, 	GNC_ID_ACCOUNT, 	(QofAccessFunc)gncInvoiceGetPostedAcc, 	NULL },
	

This parameter could use (QofSetterFunc)gncInvoiceSetPostedAcc - which takes an Account object - not a fundamental QOF_TYPE.

In the rule that is dealing with the Invoice QofEntity, use the QofObject and QofClass definitions to obtain the parameter of the INVOICE_ACC account. qof_class_is_registered, QofInstance -> QofEntity.

7.1. Data mining within QOF

7.2. Data Freedom within QOF