Executive summary
The Environment Agency (EA) works to create better places for people and wildlife, and support sustainable development in England.
Underpinning this work is data. A lot of it. EA has 1,700 defined datasets which are valuable not just to EA but to other groups and businesses who want to understand more about the environment. EA used to charge those who wanted to access its data, but in 2010 stopped charging for non-commercial use. Since then, EA has committed to making more of its data accessible to wider groups by releasing it as open data: data that anyone can access, use and share.
When EA first opened valuable datasets on bathing water quality and flood risks, developers and hackers quickly used them to build useful applications and tools, and businesses ingested them to use for themselves. After this, EA committed to releasing all of its commercial datasets by 2018.
This is the story of EA on its journey to becoming open by default: its transition, its challenges and how it has benefitted – from core objectives being achieved and resources saved to improved data quality and public perception.
It also outlines how EA open data is being used in tools, applications and business by external groups, and highlights how the data has benefitted the wider community. It also raises some lessons learned for other organisations looking to release their data, including getting the timing right and publishing in accessible formats.
The ODI has produced this report through extensive interviews with members of the EA team involved in the release of open data, along with key EA data users.
How to cite this report
Open Data Institute (2015) Environment Agency: Going open. London, UK. Available at ea-going-open-summary
About the Environment Agency
The Environment Agency (EA) is a non-departmental public body, sponsored by the UK Government’s Department for Environment, Food and Rural Affairs (Defra). EA works to create better places for people and wildlife, and support sustainable development in England. It provides services and information in a range of related areas, from fisheries and flooding to wildlife and waste.
Established under the Environment Act in 1996, EA is responsible for regulating major industry and waste, treating contaminated land, water quality and resources, fisheries, inland river, estuary and harbour navigations, and conservation and ecology. It is also responsible for managing flood risks and issuing flood warnings to the public.
All this activity is underpinned by data, which informs EA on the decisions it makes. There are currently around EA 1,700 datasets that are known and defined. It works with a wide range of partners – from government and local authorities to businesses and communities – for whom it prioritises making its data available and accessible.
Since its inception, EA has been approached by companies looking to use its data in their products and services. Where once companies used to each photocopy EA’s records and digitise them themselves, EA has long since shared its data and information from a central platform, DataShare, and recovered some revenue from users. Through DataShare, EA has made significant datasets available for reuse under commercial (and other) licences.
One of these, the Risk of Flooding from Rivers and Sea (RoFRS) dataset, has now been released with an Open Government Licence, and EA has committed to releasing all of its currently charged-for data as open data by 2018.
How and why EA went open
Timeline
- 2010: EA stopped charging for non-commercial use of their data
- 2011: EA released bathing water quality data as open data
- Dec 2013: A storm surge culminated in severe flooding across Southern England
- Feb 2014: EA temporarily released flood data for a Flood Hack hosted by Tech City UK
- May 2014: EA committed to becoming an open data organisation
- Nov 2014: First Data Advisory Group meeting held to help prioritise data for open release
- Dec 2014: EA released Risk of Flooding from Rivers and Sea (RoFRS) dataset as open data
- Jun 2015: EA agreed to a transition plan to open all its commercial data by 2018
- Sep 2015: EA released valuable Light Detection and Ranging (LIDAR) as open data
- Apr 2018: EA is committed to having released all of its charged-for datasets as open data by April 2018
Data and the Environment Agency
Initially EA charged everyone to access and use its data for their environmental decisions – whether public, private, commercial or noncommercial – in line with the Wider Market Initiative policy. It later stopped charging for the non-commercial use of its data. As the reuse of public sector information regulations came into place in 2005 and the topic moved up the political agenda, EA started to think about making more of its data freely available.
Since [the regulations] came in it's been a natural progression and of increasing interest and increasing importance [to make data publicly available]. There's this ever growing body of evidence suggesting that we should be releasing open data, and a growing interest from politicians and the rest.
– Miles Gabriel, Data, Mapping, Modelling & Information, EA
It began by releasing relatively niche, less commercial datasets as open data. Notably, however, bathing water quality data was released as open data in 2011 and useful products and services were soon built using it.
Over the past few years, EA has further explored the potential of open data, often working with the ODI to help it to make the most of its information.
It makes sense to be more transparent, to use data to help deliver our outcomes and to enable the rest of society to benefit from it in ways we could not imagine. Recently the new Defra Secretary of State has made open data an absolute priority for the department, and so we are well placed to accelerate our open data approach.
— Miranda Kavanagh, Executive Director of Evidence, EA
Weighing up the costs and benefits
Several issues concerned EA about releasing more of its data under open licences. The contractual relationship it had with people paying for its data meant it could ensure it was used appropriately, without risk of misuse or misinterpretation. Opening it up for anyone to use would mean losing that control. EA would also stand to lose revenue gained from charging for access to its data: about £6m per year.
Opening the floodgates
Between December 2013 and May 2014 England suffered a period of extreme weather activity. A storm surge and heavy rain led EA to issue 50 severe flood warnings. About 8,000 properties flooded in that period and 200,000 properties were protected by EA’s flood defence assets. EA closed the Thames barrier over 25 times during the floods to manage the water flow.
In February 2014, EA was asked by the government to open some of its flood data for a Flood Hack to be hosted by Tech City UK at Google’s Campus. At this stage, access to EA’s data was restricted and opening it up for use in such a short space of time would not have been easy. EA enlisted the help of Shoothill, a specialist data and software company. Shoothill already had paid access to the flood warning data and had built a robust API – an application program interface – to enable others to use the data in their own applications and tools. EA agreed to make the API available for the Flood Hack and beyond.
As detailed later, this enabled developers to come up with a range of solutions: from a phone service that connects people with their energy suppliers in the event of a power cut to an app that alerts Twitter users to volunteering opportunities in their local area.
Initially this data was only supposed to be open for three months, but as Open Data Manager at EA’s Evidence Directorate Mike Rose explains: “it was unrealistic ever to think that after three months we’d lock it all back up again […] Though it didn’t seem like it at the time, this fundamentally changed how we thought about our data and information.”
Planning for better quality open data
Seeing the benefits from having better-informed users of its data, and more useful applications to reduce the risks and effects of flooding, EA worked through resourcing and structural shifts in order to open up its RoFRS dataset with an Open Government Licence in December 2014.
EA raised data management as an ‘organisational risk’ in 2011, meaning senior management monitors how well the organisation is doing using a data maturity model developed by EA. It covers governance, ownership, security, sharing, interdependencies, data standards and data quality. This helps track and communicate progress across the organisation, and helps to identify risks and priorities.
EA assesses its open data maturity and journey towards becoming ‘open by default’ using the ODI’s Open Data Pathway.
Engaging the community
With more focus on open data, EA has created a sector engagement plan to understand how different types of people and organisations might use it, thinking about users outside of those it had worked with before and were familiar with. This culminated in the launch of the Environment Agency Data Advisory Group (EADAG).
EADAG, comprising organisations who have an interest in EA data, like Shoothill and ESI International, along with local councils and independent experts such as those from the ODI and Owen Boswarva. The group meets quarterly and advises EA on data it should prioritise for release, best practice and what will be of most use to data users.
How the data is used
Businesses, communities and government groups have used data made open by EA in many different ways.
Showcasing what is possible with open data
When EA released its data, members of the open data community worked quickly to show what was possible in using it. The Flood Hack, held in conjunction with EA’s first flood open data release, saw 18 applications developed. These included Flood Feeder (a visualised aggregated feed of flood and related data) and DownStreams (a social enterprise linking downstream communities that are at risk of flooding with upstream communities that could prevent it).
Shortly after the release of the Risk of Flooding from Rivers and Sea (RoFRS) dataset, Owen Boswarva, an Environment Agency Data Advisory Group (EADAG) member, created a number of web apps to illustrate the data’s potential.
Building flood risk awareness tools
Most applications that use EA open data outside of the public sector use flood risk data or live flood data feeds. Shoothill used the data to build apps like My Flood Risk and FloodAlerts, that can inform the public of their risk of flooding. FloodAlerts received millions of hits during the severe flooding of 2013/2014 and featured on the websites of UK media outlets, such as Sky News, the BBC and the Daily Mail.
Shoothill has also used river level data to create GaugeMap, an ODI Award-winning product and the world’s first live river level map system. Shoothill has assigned a Twitter account for each gauge that is used to collect the river level data – over 2,400 gauges overall – so that users can be alerted to rising river levels that could lead to flooding.
Software consultancy KnowNow has recently combined EA data with other datasets to predict where road accidents caused by flooding would occur. Engineering group CH2M has used the data to release their own Flood Alert app.
ESI International, a specialist geoscience consultancy, has developed environmental models using closed or shared data for about 19 years. It has been quick to use the increasing amounts of open data made available by EA – including river level data linked to ESI International's own mathematical models, to create water flow models for England. The models can be used to produce forecasts and predict flooding.
Solving commercial problems
Business and markets can be greatly affected by environmental change. To avoid disruption and manage future risks, businesses need to plan around environmental factors.
Flood risk data is generally used in two areas of business: for risk selection and pricing in insurance, and in environmental reports for property groups. It is used by UK household insurers and commercial property insurers. Firms like Argyll Environmental and GroundSure use it to produce environmental flood reports for the property industry. Some large insurers supplement EA open data with proprietary commercial models. There is now a new Memorandum of Understanding under which insurers will cede the underwriting of high-risk properties to a government-backed reinsurance scheme, Flood Re. Flood Re will use a proprietary flood risk model, which is being developed by commercial firm JBA, and will use some EA data.
In 2014 the UK’s innovation agency Innovate UK held a competition for companies to ‘solve business problems with environmental data’. Winning companies like ImageCat Ltd used EA flood risk data to help build forecasting tools for infrastructure and insurance sectors. Others, like International Synergies, used EA pollution and waste data to help find low-carbon waste streams as an alternative raw material for major construction projects.
As more of this data is published openly, more businesses can use it in combination with other commercial or societal data to develop valuable new insights.
Campaigning about climate change
As EA flood data is now more accessible to people without specialist technical skills, it is likely to be used more in different industries and across different platforms. Flood risk research based on EA open data by GroundSure was featured in the Evening Standard, paving the way for charities and interest groups to use the data in their own campaigns.
Benefits for EA
The Environment Agency (EA) has found many benefits in opening its data, ranging from increased efficiencies to improved data quality.
Helping EA to achieve its core objective
The release of the Risk of Flooding from Rivers and Sea (RoFRAS) dataset as open data will ultimately increase the public's awareness and knowledge of flooding, according to many EA staff. Applications based on the data have helped raise awareness about environmental issues and flood risks, helping EA to achieve its core, overarching objectives.
We were surprised and happy about the rapid proliferation of products that were built off the back of the RoFRS dataset]. We in the open data team thought these products would develop. I think the rest of the business was more unsure.
— Mike Rose, Open Data Manager, Evidence Directorate, EA
With [the release of] RoFRAS, it enables individuals and communities to be much more aware, resilient and able to manage flood risk a lot more.
— Miranda Kavanagh, Executive Director of Evidence, EA
Saving time and resources
While opening valuable flood data will lead to the loss of its commercial value for EA – around £1m – its impacts could save the agency significant time and resources.
[Opening data] is not a completely altruistic gesture. We’re not just giving it up for the good of the world. Actually, it can have a direct, significant Environment Agency benefit.
— Mike Rose, Open Data Manager, Evidence Directorate, EA
For example, law often requires developers to conduct flood risk assessments before building properties, which EA’s Area operational staff then have to check for model clarity and accuracy. Currently, flood modeling in the UK is regularly conducted using data from sources other than EA. By making its high-quality data open and free, some at EA believe that it will be used more widely for this purpose. This will save some of the significant time and resources currently spent checking flood risk assessments built using other data.
The potential benefits of this type of saving could be vast. A 2011 study found that €2bn could be saved per year by improving the accessibility of information required for mandatory environmental impact assessments in the EU – or 20% of the total costs.
Building external relationships and getting the user voice heard
EA received operational support and advice from external groups while planning and implementing its flood data release in 2014, including Shoothill. Since then it has gathered a strong advisory group, with diverse members – from businesses like Shoothill and Esri UK, to local authorities and universities such as Salford Council and Queen Mary University of London – helping to prioritise its data release and focus on user requirements.
Previously, when we weren't so focused on making our information open, we were quite internally focused, delivering our own data needs [...] and not so focused on our customers’ needs. We have been getting better at that over the last few years, but open data has been [the] real catalyst - [we try to] include their requirements so that we provide something that is going to be used by as many people as possible.
— Hayley Bowman, Flood & Coastal Risk Management, EA
Improving data quality and public perception
External scrutiny of data, whether from advisory groups or data users, can help to improve its quality in two ways: data is often prepared to a higher standard before publication (to avoid potential criticism of its quality), and data can be corrected or improved according to feedback, once it is open. This means that EA can benefit from higher quality data for its own decision making, as well as those external groups who also use it. Being more open about the data EA uses will also help to improve its transparency to the public.
Open data shines a light on how well we manage our data generally. It's a really good test of how well we're doing. In the future it's going to improve how we do things and it could potentially drive up quality [...] If we're more open about what we're doing and the data we're using to make our decisions then we will gain the trust of the public and the businesses we regulate. Whether [the data] is perfect or not, at least we can say this is the best that we've got, this is what we're working with. If users find flaws in the data this could help us to make it better.
— Deborah Yates, Data, Mapping, Modelling & Information, EA
Better understanding and managing risks around data use and publication
Collecting data with openness in mind, or being ‘open by default’, helps EA to improve it according to both its own needs and those of its owners. It also means that governance issues like licensing are taken into account early, so EA is more aware of risks around its use and publication, and can manage them better.
If we're publishing new data it's open by default now. There's a presumption the data will be open; the legal checks we do will see whether there are any reasons why it can't be open.
— Deborah Yates, Data, Mapping, Modelling & Information, EA
Bringing diverse teams together
In opening its flood data, EA brought together lots of its teams to work towards a collective goal. This helped the agency to work cohesively and collaboratively through internal groups.
Working more easily and efficiently with external partners
Working with external partners – from businesses to other agencies and government bodies – can often be made difficult for EA by the restrictive licensing terms of their data. They can limit what and how data is shared with others, slowing down or prohibiting work throughout projects or the co-delivery of services. Publishing data with an open licence makes collaboration much easier.
As Miranda Kavanagh, Executive Director of Evidence, explains:
A strategic imperative [for EA to release] open data with an Open Government Licence is that it enables partnership working. It's much easier to bring people on board, and in some cases to actually get them to lead projects. So I think [the release of open data] is very much related to the trend in public service to co-deliver services in conjunction with stakeholders – it's a big part of it.
Harnessing the power of the tech industry to make useful applications
When data is made open, it becomes accessible to a host of expert developers and businesses. This means it can be used to build diverse, high-quality applications – like Shoothill’s GaugeMap – and may save organisations like EA the need to invest in the technology and training needed to make them themselves.
Through making data freely available you can enable the rest of society to come up with applications and uses for it [which] you might never have thought of in the first place – applications and uses that you also wouldn't necessarily have been able to afford, particularly at a time of declining public expenditure and austerity.
— Miranda Kavanagh, Executive Director of Evidence, EA
We're here to regulate and protect and improve the environment, so we shouldn't really be developing apps; I think [releasing open data] will ultimately save us a lot of time and money. We won't need to be investing in the technology to do that unless it's part of our core activity.
— Deborah Yates, Data, Mapping, Modelling & Information, EA
There is currently some debate surrounding the extent of the role to be played by large government departments, agencies and bodies – such as EA – upon the release of open data. Some argue that the development of applications and other tools, such as those that increase awareness of flooding, should be left solely to external developers and businesses. Others think that these arms of government should also use the data themselves, or in tandem with external actors, to create the applications required by the public.
This debate is set to become more prominent with the release of larger swathes of open data – such as that released by EA between now and 2018 and the 8,000 datasets to be published by Defra this year.
Many different groups have benefited from open data released by the Environment Agency (EA) – from businesses using it to create products and services to communities using it to understand more about flood risks.
Expert, agile businesses were engaged to help release and host the data
In order to respond quickly to a bout of severe flooding, EA worked with Shoothill to release the Risk of Flooding from Rivers and Sea (RoFRS) dataset in 2014. Bringing in a smaller, more agile business enabled EA to work flexibly and respond quickly to external demands – Shoothill also provided the APIs for a number of EA open datasets.
As Rod Plummer, CEO of Shoothill describes:
Prior to [the Flood Hack], EA had licensed the data to companies on a commercial basis [...] Fortunately for EA, Shoothill had already built an API for live flood data during the development of our FloodAlerts product, and so we were in place to deliver the API within the tight timescales required.
The Environment Agency has also worked with Epimorphics, a specialist linked-data solutions consultancy, since 2011, when it decided to release its bathing water quality data as open data. The Epimorphics team has worked with EA to continuously improve that data and now hosts other EA datasets, such as real-time river level monitoring and flood alert open data.
Releasing open data effectively does not just involve a change of licensing.
You need rich access. As well as the raw data, you need applications that mean that [regular users] can go in and use the data, as well as the developers. EA is trying to create a whole ecosystem around its open data, which gives opportunities to people like us to develop those applications and provide the infrastructure.
— Dave Reynolds, CTO, Epimorphics
Working with EA has provided a platform for businesses like Epimorphics and Shoothill to find and attract more customers with similar demands. As Dave Reynolds explains:
[Working with EA] has been a useful source of revenue. It's also given us a reputation for doing a good job of releasing data, which has helped us to find other customers who are doing the same thing.
The release of EA open data saves businesses money
As more EA data is made open, more money can be saved by those who previously had to pay to use it. A national coverage licence for corporate use of the RoFRS dataset costed around £20,000 prior to its release as open data. The £1m (approx.) in revenue generated each year by the data can now be reinvested by the businesses that are no longer charged for their use of it, to develop new products and services that add even more value.
I don't think [users of the data] ever wanted to pay for it anyway […] there were a lot of people thinking ‘we shouldn't have to pay for it’.
— Paul Wyse, Flood & Coastal Risk Management, EA
When data is made open, more people use it
Mike Rose, EA Open Data Manager and member of the Evidence Directorate team, worked extensively on the release of the RoFRS dataset and found that "what seems to have been proved is that by making it open, more people are using it."
Those concerned with flooding are using the RoFRS dataset to understand the risks in their areas. Deborah Yates, of the EA Data, Mapping, Modelling & Information team, described how she had seen one flood forum member use the data to quickly come up with a custom, local view. “He's mapped it with other data that they've got and they can start to see what's coming in from rivers upstream that could affect them, on a totally personal or local level,” she explained.
Opening data also means developers and small business who could not afford to pay for access can now also use it to enhance their work and build new tools. There are nearly 250,000 raw requests each month for river levels and flood alerts data, just one of the flood-related datasets hosted by Epimorphics.
Members of the EA team have found that even the nonprofit organisations that could access the data for free (before it was published as open data) now use the data more. Primarily, this is thought to be a result of the Open Government Licence, which has made the data easier to use than EA’s old non-commercial licensing terms did.
Local communities and councils can access the data more easily
Without licensing restrictions or administrative hoops to jump through, more community groups and councils can access and use data for local initiatives.
The day [the RoFRS data] came out, we worked with Get mapping [who produce] Parish Online for Parish and Town councils. Being able to access and use the whole dataset was really good for them – they were able to get the data first-hand without having to go through [the] long process of getting each Local Authority to supply their bit of RoFRS individually to Getmapping, or Getmapping having to pay for a full commercial licence. [The data] could get to community groups a lot more easily.
— Angharad Stone, Data, Mapping, Modelling & Information, EA
We now have more tools to help us to understand and deal with flooding
When faced with flooding, a number of applications, products and services developed using EA’s open data are now available to help us to understand and deal with it.
Lessons learned
The Environment Agency (EA)'s experience going open has raised some useful lessons that other organisations considering opening their data can learn from.
Consider data licensing and terms from the very beginning
When different organisations are involved in collecting data, it is important to make sure that their licensing accounts for the data being released, whether immediately or later on.
The release of the Risk of Flooding from Rivers and Sea (RoFRS) dataset as open data was not considered during the initial agreements made with all third-party data suppliers, who may have held intellectual property rights or contractual restrictions over data included in the dataset. These agreements had to be checked and addressed in a short space of time before the data was released.
We need to get the data management process right […] so when you come to publish something it's just downloadable – anyone can use it to do what they want because we've checked all the rights are there […] We need to build [this] into how the data management function works as a whole, rather than [do] what we've done in the past, which is pick it all up at the end.
– Mike Rose, Open Data Manager, Evidence Directorate, EA
Publish your data in accessible formats
Consider different format options in order to make sure your open data is easy to access and use. Once EA’s flood data was released, a number of users complained that it was made available in a proprietary format that made it difficult to use.
Angharad Stone, of the Data, Mapping, Modelling & Information team, suggested that:
To begin with, we only made RoFRS available as an Esri geodatabase, which is a proprietary format which can only be used with certain software. So we have had people coming back and saying 'you've produced this as open data but you've put it out there in a format most of us can’t use.'
Shortly after, EA responded to the feedback from its users and made the data available in different formats. Some EA staff also suggested that their teams would benefit from specialist training on the latest technical aspects of open data publishing, such as linked and real-time data.
Create user feedback loops to evaluate your data’s impacts
Once data is released, it helps to have systems in place to monitor, assess and evaluate its impact. Without these, it is impossible to track who is using your data and for what purposes, because open data does not require people to sign-up or register to use it.
With limited resources available, EA did not put processes in place to actively engage its data users, so found it hard to evaluate its data's use and benefits.
On the one hand having no registration means that there's no barrier to entry, but on the other hand you've got no way of either helping people [to] make better use of the data or of working out the difference between someone making genuine heavy use of the APIs and someone with rogue code who doesn't realise the load they are accidentally placing on the service. So, for managing a service, [releasing open data] can be quite a hard balance [to strike].
— Dave Reynolds, CTO, Epimorphics
Tell businesses about your data
Opening data is the first step towards making it useful. Promoting your data and engaging users is key.
A number of the EA team described how they felt they didn’t do enough to ‘sell’ their open data. Guiding users to the data, describing to them what it can be used to understand and helping them to use it were all areas in which EA staff think more could be done. In particular, by making more businesses aware of the data it could enable them to make further substantial cost savings.
As Mike Rose asks:
Is the commercial companies’ [who provide flood modelling] data better or worse than ours? Who knows. But should we be putting effort into selling our free product? Absolutely. We believe it's the best stuff and we are making regulatory decisions based on it. We believe that people out there should be using it to make their decisions. If we're not telling the insurers that our data’s available for nothing, they'll only see the flyers from the companies that are going to charge them £500,000 a year."
EA - Acknowledgements
The ODI conducted extensive interviews with EA team members and others to tell the story of EA ‘going open’. We would like to thank the following people for their contributions:
- Paul Ashton, EA
- Hayley Bowman, EA
- Miles Gabriel, EA
- Claire Hainsworth, EA
- Miranda Kavanagh, EA
- Trevor Linford, EA
- Rod Plummer, Shoothill
- Dave Reynolds, Epimorphics
- Mike Rose, EA
- Angharad Stone, EA
- Adam Tobin, EA
- Paul Wyse, EA
- Deborah Yates, EA
The Environment Agency has committed to releasing its commercial data as open data by 2018. To find out more about EA data release, you can follow @EnvAgency on Twitter and visit the EA blog page.
For more information about open data and how it can benefit you or your business, see our 3-minute video and explore our courses.
The Open Data Institute · Friday Lunchtime Lecture: The Environment Agency's open data journey