Skip to content
Big data

Open data and harnessing the whole data spectrum

Dr. Andrew Fletcher  Director, Thomson Reuters Labs™ – London

Dr. Andrew Fletcher  Director, Thomson Reuters Labs™ – London

The evolution of open data

Open data is not new, but our understanding of how businesses are harnessing it has changed dramatically, even within the last couple of years. In the financial world, open data (“data or content … anyone is free to use, reuse, and redistribute”) may seem at odds with how any organization can drive competitive advantage. If you can access the same data as everyone else, at zero cost, how do you differentiate yourself and drive value for you and your customers?

The modern focus on open data came when the Obama administration established the portal, closely followed in the UK by Both shared the same principle: If data is made available, then people will do more innovative things with it – and this was a welcome source of opportunity after the financial crisis. Open data has been used in many different businesses, ranging from crop insurance for farmers to more realistic terrain in computer games and live train times in your transport app. Both governments also wanted to enable transparency and scrutiny, and this drive has continued to gather pace,
as shown by the anti-corruption summit hosted by former Prime Minister David Cameron in London in May 2016.

The UK government went further by funding the Open Data Institute (ODI), based in London but with a network of 28 nodes globally. It has been driving forward the understanding and use of open data since its formation in 2012. The UK was placed at the top of the global ranking calculated by the Open Knowledge Foundation in 2013 and 2014 and was only knocked into second place by Taiwan in 2015.

So what does this mean for finance? Well, one driver is the continued push for openness and transparency across the sector, again prompted by the financial crisis. Barclays and the ODI were asked to be joint chairs of an open banking working group, which published a recommendation for the Open Banking Standard in February 2016 as a drive to implement the EU regulations on banking data. The standard includes a recommendation that “bank data, including information about banks’ products and services, should be made available as open data” and “open APIs should be created to enable services to be built using bank and customer data.” Whilst these recommendations are focused on consumer banking, the trend towards open data is clear across the industry.

The whole data spectrum

With a lot of the recent focus on open data having come from government data, it is easy and dangerous to view open data as the end goal for all data. Even with government data there must be exceptions for data, for example where there are privacy concerns such as health data. A significant maturing of the discussion around open data is seeing it not as the ultimate goal for all data, but as an important component of the whole data spectrum. Ranging from Closed to Shared and then Open Data, we would expect private, commercial and government data to exist in different parts of the spectrum and with different terms and access rights.

Figure 1. The data spectrum

Chart from the Open Data Institute detailing the full data spectrum.

So how does any person or any organization begin to get value from open data? In 2014 we wrote a paper jointly with the ODI called “Creating Value with Identifiers in an Open Data World,” which introduced the importance of using an open identifier scheme in being able to create linked data. We went on to launch, a Web portal providing access to Thomson Reuters Permanent Identifier (PermID), metadata and tools available under an open licence and under an ODI open data certificate. PermID currently covers companies, instruments and quotes. Why should you care about using an open identifier? Well, an identifier acts as a bar code for information, enabling you to link together everything you know about a company across all the different locations where you might hold or have access to that data. And by using an identifier under an open licence, you can act with the confidence that the same bar code can continue to be used, even commercially.

Figure 2. Knowledge graph

A relationship graph showing how PermID works.

Why would a company like Thomson Reuters make its data open? It’s all about building and maintaining trust. It makes sense for our customers and it also provides value back to the open community that they can reuse, including commercially.

Shareable by default

For data to be effectively shared or opened it must first be shareable. The power of open data is really in the other data that you can use alongside it. Open innovation requires data that is shareable. Invisible frameworks that govern the structure and management of data must be interoperable to facilitate the value that can be extracted from it. Data must be treated as infrastructure, with special attention given to the features of data and the surrounding mechanisms that make data shareable, regardless of whether that data is closed, shared or open.

Together with the ODI we describe this in a checklist to make data “Shareable by Default.”

  • Communicate and document meaning to ensure that data can be understood by others
  • Describe data provenance to determine the context of its origin and fitness for use
  • Describe access and usage rights to ensure that the right people gain access to the data and that all aspects of privacy are respected
  • Publish a description of the data in a data catalogue to enable search and discovery
  • Provide efficient ways for users to access data on demand, using a variety of tools
  • Use common standards strategically to ensure interoperability and take advantage of network effects
  • Design data for everyone so that its potential use is not restricted to the current community and application
  • Cultivate an open, collaborative culture to encourage the creation of data that is shareable by default and the reuse of data

“Data helps us navigate to a decision,” but it is vital that the answers we seek are powered by the whole data spectrum. The future of open data is bright, being driven by a global movement. We’ll see open data to come from governments and, increasingly, companies. But open data will never be the whole picture. That whole picture can only come from a world where data is shareable by default.

Learn more

Discover PermID

Image of Exchange magazine pages

Read more from Exchange Magazine in the Know 360 app