Data infrastructure consists of more than just datasets. It includes other data assets such as identifiers and registers, the standards, technologies and policies that manage and govern access to data, and the organisations and communities interacting with data.
ODI, Refinitiv and Microsoft have been leaders in this space for years, with Refinitiv’s PermID becoming a critical part of global finance sector data infrastructure, and the Microsoft Planetary Computer open data platform supporting the Industry Data for Society Partnership, a hallmark in open collaboration and innovation.
Together, as part of our mutual partnerships, we hosted a roundtable to discuss the importance of open data and open infrastructure that can help build markets, deliver value to individuals and organisations and improve society and the environment. We convened important private sector companies across multiple sectors, including finance, tech and energy, as well as nonprofits all involved in building and maintaining open data and digital infrastructure The subsequent discussion yielded insights around how further open data collaboration and innovation can help us all achieve our missions.
This blog post highlights the key elements that were discussed under Chatham House Rule by the organisations involved, outlining the opportunities, challenges and ways forward for open data collaboration to drive real impact in society.
The opportunities
There is a huge opportunity to leverage private sector data sharing to tackle problems across numerous sectors, geographies and topic areas. Open data and data sharing currently exist, but are limited in their scope and impact, and more is needed especially amongst competitors and across industries, to address issues and opportunities.
Companies hold lots of valuable data and are very capable at building important nodes of a data ecosystem, such as platforms and portals to provide access to data and information. However they lack the independence to build system-level data infrastructure such as standards for data, APIs, policies for data access and protection, or manage institutions in the long run that can convene whole ecosystems and industries.
We can look at the banking, energy and water industries to see some clear examples:
- Banks make open data APIs available for ATM and branch locations, service metrics and product information, but it is the open banking ecosystem and standards that has had a transformative impact on the sector via its secure facilitation of customer data sharing.
- Energy networks have open data portals of varying scope and quality that publish valuable data about the energy network infrastructure and usage. Initiatives like Open Net Zero are facilitating greater searchability to unlock the further benefits of sensitive data sharing and open data aggregation.
- Water companies have begun their open data journeys in earnest over the last year, and programmes like Stream are helping to move the entire sector forward through data collaboration that increases value and decreases risk from going alone.
Possibly the biggest problem spaces and opportunities for this kind of collaborative building and maintaining of open data infrastructure is in the world of sustainability, with ESG (environmental, social, and corporate governance) and net zero initiatives relying heavily on data flows to ensure success.
All of the participants, though in different industries and with different roles, were working, often collaboratively, in some part of the ESG and net zero ecosystems. These projects ranged from increasing access to smart meter data in the UK, to connecting small businesses to financial institutions to increase access to green finance, to using open data and open standards to improve the monitoring and evaluation of government and industry ESG commitments.
The challenges
Despite the optimistic outlook, it was clear from the participants that realising these opportunities will not come without facing a number of challenges. Though the flow of data is conducted through technological means, technology was not seen as a major barrier to unlocking the benefits of open data infrastructure. Rather the challenges lie with different elements of processes and people. The participants articulated these challenges in the following three broad categories:
Value and commercial models. The challenges facing the value of data are twofold, both the difficulty with understanding how to value data, and how to use those insights to create a commercial model for financially sustaining open data infrastructure. Data is difficult to ascribe a value to for both economic and contextual reasons. Economically, data does not act like most other assets, and can actually gain value through use, as opposed to the consumption of other goods and services. Equally, understanding who data is valuable to and how, is highly contextualised. Is one dataset or data access model better suited as a commercial service, or should it be openly published within funding made available another way? Understanding how to maximise value for the data providers, users and wider beneficiaries still seems to be nascent.
Data literacy. Data literacy is the ability to think critically about data in different contexts and examine the impact of different approaches when collecting, using and sharing data and information. It goes beyond data science and the manipulation of data to think about what role data has in a society or economy. Currently, there is a data literacy disconnect in the private sector between those actually working with data, and many other decision makers. Data specialists, from practitioners to CDOs often see the value in sharing data and collaborating openly, but other parts of the businesses either worry about the risk of increasing access, or believe there is more value to be had from keeping data closed.
Policy to support data sharing. The absence of supportive policies has been a significant constraint on data sharing. Supportive policy has played a crucial role in facilitating data sharing in fields like astronomy where data is scientifically valuable but lacks commercial value. However, the lack of policy development in areas such as private sector data publishing and intellectual property policy has introduced potential challenges. The former does not compel companies enough to seek the benefits of open data publishing and in the latter the risk emerges that intellectual property rights, including copyright, can obstruct the use of open data.
A cross cutting theme to all of this, and one that has been at the forefront of open data and data sharing for years is around trust. Complex networks and supply chains, unsupportive government policy, and low literacy on the value of increasing access to data all contribute to a low-trust environment where open data is eschewed for more restrictive commercial models. It is possible that an overarching focus on how to improve trust in markets and ecosystems is the critical way forward to improve openness.
The way forward
Expanding the level of open data infrastructure in the private sector demands a comprehensive approach that addresses not only technological aspects but mainly focuses on the process and people challenges. This can only be overcome by a focus on the use case and benefits trying to be achieved, and clearly demonstrating how open data and greater data sharing is the means of doing so, not just another tech-first solution.
To navigate the complexities, tools like data ecosystem mapping can evaluate data value for key actors and promote mutually beneficial data-sharing business models. Besides being a practical planning tool, this can help foster discussion where representatives from different organisations can see how their challenges are not unique. It can also help prioritise interoperability, where the development of consistent open data publishing standards, data formats, licences, and catalogues, can enable a more cohesive data-sharing environment. This is particularly important in ESG and net zero data ecosystems.
Government policies and regulations also play a pivotal role in providing a clear framework for data sharing without compromising security. Strategic regulation not only ensures compliance but also nurtures a culture of innovation, as seen in the advances brought about by GDPR and PSD2, propelling the development of data ecosystems such as open banking and secure data sharing approaches such as privacy-enhancing technologies. Current initiatives focussing on Smart Data and digital twins are providing interesting and exciting new government-backed ways to enable collaborative data innovation.
Promoting data literacy among both practitioners and decision-makers is critical, enabling a comprehensive understanding of the risks associated with data sharing but especially the potential value that can be unlocked through effective data collaboration.
Ultimately this all comes down to trust. Trust that data holders will provide access to data that is correct, timely, and with goals beyond the financial bottom line in mind. Trust that data users will act ethically when accessing data from the private sector. Trust that policymakers will create and enforce just and equitable rules for data ecosystems. By collaborating across the private sector and beyond, businesses and their partners can build these trusted ecosystems that will guarantee greater openness, providing more value from data for all.