Getting The Most Out Of FpML - Part 1

FpML is an open source industry standard for sharing information on, and dealing in, financial derivatives and structured products. By leveraging FpML you gain from decades worth of financial modelling expertise and production use. Whether you are creating a new model from scratch or maintaining an existing model, you will benefit from FpML’s insights into the problem domain.

However, despite the richness of the definitions, there remains a number of technical barriers that impede adoption and prevents many projects from getting the most out of FpML.

In this two part blog we will explore how a solution combining the extensive logical model [1] of FpML with the integrations provided by Spear can address a number of adoption concerns including making it available to a larger set of language formats, reporting challenges and support for augmenting FpML models with your own attributes in a way that is compatible with the standard.

A World Beyond XML/XSD

The technical landscape at the dawn of FpML in the late 90s and early 2000s was rather different from what we would consider the norm today. A primarily XML based approach to every problem reigned supreme and many of the technologies that are now pervasive (JSON, Python, Big Data Reporting - the list goes on) were not available at the time, or at least very limited in terms of adoption.

This reliance on XML/XSD (a very comprehensive modelling tool in its own right) has lead to many projects unfairly dismissing FpML as 'dated' or 'difficult to integrate with modern frameworks'. The power of FpML however, lies in the broad logical model it defines, not the technical implementation it just so happens to be expressed in.

Spear's ability to consume existing data models and expand its use to a larger range of technologies have been explored before as part of a previous blog. Let us apply the same principle to FpML.

FpML Expressed In Spear

Spear has built-in support for interpreting models defined in a wide range of definitions languages - including XSD which the latest recommended FpML standard is made available. As per our TeraHelix FpML reference project on GitHub - https://github.com/TeraHelix/apps-FpML - this simply involves the invocation of these two Maven plugins:

Spear Maven Plugins - pom.xml

As alluded to in the code snippet above, JAXB is used as a transitional step in generating the Spear code from the supplied XSDs.

The generated JAXB/Java code that is produced however, is not rich enough to fully capture the data modelling constructs which FpML attempts to convey. For instance, one example where the JAXB/Java representation of the FpML model falls short in is how multiple occurrences of xsd:choice elements are handled:

Table 1. JAXB/Java does not give you the full data model picture
XSD Java/JAXB
XSD Loan Definition Extract
JAXB/Java type information lost

The Java/JAXB code in the snippet above misrepresents the borrowerOrBorrowerReference attribute as simply being of type List<Object>, which is far too broad a definition and likely to cause confusion to downstream integrations and developers. Sure, there are a number of advisory @XmlElement descriptions here; however, as these are annotations only and not a core part of the compiler life cycle, they tend to offer only slightly more value than a Javadoc.

Spear by contrast, being a data modelling platform, is able to express this construct as originally intended - a unionstruct [2] which is in effect a choice of either a LegalEntity or LegalEntityReference.

Spear Defined Loan

And the associated UnionStruct:

Spear UnionStruct

FpML Accessibility For All

With a Spear description of FpML, a whole new world of possibilities beyond XML/XSD opens up. For instance, taking an example loan trade object provided as part of the FpML specification - loan_trade_ex001.xml - can easily be consumed and converted to other formats (loan_trade_ex001_example.java)

Type Conversions

In addition, Python is generated as standard and you have the option to also generate ProtoBuf [3] definitions as required:

Table 2. More Automated Integrations
Python ProtoBuf
Python Extract
ProtoBuf Extract

FpML Reporting Challenges

FpML data structures are defined as highly nested structures adhering to well known object orientated design techniques such as polymorphism, encapsulation and inheritance. For instance, in even a simple definition such as an Org.FpML.Confirmation.Loan, you can already see values being nesting up to 3 levels deep;

Org.FpML.Confirmation.Loan - Nesting already 3 levels deep

For the quantitative and/or developer use-cases, it is perfectly appropriate to deal with the structures in this way. It is the natural paradigm if you program in a language such as Python, Java or C++.

However, when doing reporting, the most appropriate shape for the data is 'rectangular' or 'relational' - these extra levels of nesting simply gets in the way! Report developers typically just want to interact with their data using SQL. So sometimes the nested structure of definitions like these may even drive developers to maintain manual mappings to convert between the object world and the relational world.

Beware - this a trap ! Manual mappings introduces a huge maintenance burden and opens the door to the 👤 shadow model 👤 [4] and all the undesired consequences this entails.

Instead, Spear has a relation operator that automatically flattens any data structure down a rectangular table which can then be consumed by a large number of reporting tools. Including Apache Spark SQL - the default implementation TeraHelix uses when manipulating relational data sets.

Spear makes the richness of the FpML model accessible in either nested object or relational/reporting form as the use case requires.

FpML Trades - Auto converted to relational data

Conclusion

In this blog we explored how modellers can leverage the richness of the FpML’s model in platforms beyond XML/XSD by combining it with the data modelling capabilities of Spear.

In the next blog in this series we will examine how FpML can be customised to meet the specific organisational requirements. For instance, narrowing of coverage of products definitions only to those the business trades in or to augment the base FpML model institution specific data fields.

Any thoughts, queries and comments are, as always, welcome. Feel free to let us know at https://www.terahelix.io/contact/index.html

Reference Project

The reference project upon which this blog is based can be found here: https://github.com/TeraHelix/apps-FpML/


1. FpML provides a description of the data structures associated with financial derivatives and structured products. This can be thought of as the 'local model' for the industry domain. Being a mature standard, FpML covers a significant portion of this domain.
2. For more information on Spear unionstruct - please refer to the examples and feature guide.
4. Shadow data models are created when developers respond to an inadequate data model by adding 'small tweaks' at various points within the system to make up for the features that are missing or ambiguous. The subtle accumulation of technical debt happens quietly and unnoticed in the background, until it eventually becomes a serious issue. You can refer to the full section on the shadow data model here: https://www.terahelix.io/blog/Why-Protobuf-Isnt-Enough/index.html#beware-of-the-shadow-data-model