Invitation to Tender: Companies to develop open source tools for open data publishing
29 August 2017
Call for tenders by the Open Data Institute
Contact: [email protected]
The objective of this work is to develop new tools or significant incremental improvements to existing tools that are used to publish open data. Two awards will be made. The successful contractors will work in collaboration with the Open Data Institute (ODI) who will provide guidance, review and assistance throughout.
Summary and timeline
50% of the agreed contract price will be paid in December 2017, and the remaining 50% will be paid only upon satisfactory completion of the work, including responses to feedback from the ODI.
Public and private sector organisations want to publish open data on the web so they can create efficiencies, improve trust and encourage innovation. Some organisations find this difficult due to the required skills and toolsets.
The ODI is now starting a project – part of a broad programme of R&D activities – to improve this situation through the development of tools and services. We aim to help public and private sector organisations to publish more open data, and publish it in a way that makes it easy to use.
As part of this initiative, the ODI wishes to commission two packages of development of tools and software to improve ease, speed and efficiency of the processes involved in the supply side of the open data lifecycle, including the creation and gathering of data, curation, cleansing and pre-processing, pre-publishing, uploading or connecting to backend systems, access management, etc.
We are currently running a series of discovery activities, including workshops and interviews, which will result in the publication at the end of September 2017 of a report on the state of the art in open data publishing tools, as well as findings on unmet needs and pain points in the processes, methodologies and usage of tools for open data publishing. These will be expected to feed into the refinement of the scope of the tool development projects covered by this Invitation to Tender.
The three core hypotheses of this discovery phase are:
- Quality - The quality of open data being published today and the quality of how this data is being published are often too low to enable effective discovery or reuse. Our hypothesis is that better tools (for e.g. cleaning data, describing it or simply better integration of the various steps of the publishing process) may improve this situation.
- Speed - Publishing quality open data takes too long. Our hypothesis is that it is possible to improve tools and methodologies for open data publishing that decreases the time it takes someone to publish open data.
- Cost and automation - Publishing quality open data today is a costly process involving a large ratio of manual, tedious processing. Our hypothesis is that there are opportunities to automate the process of publishing open data, increasing speed, quality and reducing cost.
We are particularly keen to see the development of tools that support high quality publication of data that is more complex than spreadsheets. This includes data such as statistics, streaming data, large-scale imagery, geospatial data and so on.
We are also particularly interested in tools and methodologies for pre-processing and anonymising of personal data.
The tools developed as part of these two projects may be entirely new, or may be incremental improvements or extensions to existing tools. We also expect that all deliverables, including the source code and other outputs will be published under an agreed open licence.
For each of the projects awarded through this tender, the successful applicant will create and deliver:
- open source software, including tests
- documentation for the software, both at the technical level (for an audience of potential open source contributors to or reusers of the code) and non-technical (for an audience of potential users of the tool)
- training material such as tutorials and videos, and collaboration with the ODI in integrating those into ODI training courses and curriculum
The software developed through this project may also be offered to the public in the shape of an online service, in which case the successful applicant will be responsible for the hosting and maintenance of the service for the duration of the work, which may be made available through a subscription or freemium service-level agreement as desired. We are more than happy for you to make money hosting a service with the code that you develop through this tender.
The ODI favours working in the open. The successful applicants will be expected to conduct this work in this manner through blogging and speaking about their work. In practice, the successful candidate will be expected to write at least one blog post in the course of the project, as well as give a presentation of the final deliverables at an ODI Lunchtime Lecture or similar event - which may occur after the end of the project.
The tool development will be expected to happen with users at its heart, and the successful applicants will be expected to conduct user research, testing and other forms of engagement with the potential or existing user base for the tool as a matter of course.
The ODI will aim for this work to be done in an agile and collaborative manner.
There is an expectation that the successful applicants will work closely with the ODI team, which includes regular face-to-face meetings and being available remotely (eg skype, email, slack). There will be:
- A face-to-face kick off meeting, held in London
- A wrap-up meeting, held in London
- Fortnightly meetings to share progress on the project. The ODI will provide feedback and comments on draft deliverables during these. Short presentations of findings at regular interval will also be delivered by the successful candidate as part of the fortnightly status meeting with ODI.
The successful applicants will then be expected to conduct its work on the development of the tool in the open as much as possible, with code, issues, roadmaps and decisions being documented in the open by default and exceptions to be approved by the ODI during fortnightly progress meetings.
The ODI will, where possible, provide space to work in our offices in London, when members of the successful company wish to work onsite. There is an expectation that the successful company will work closely with the ODI labs team, which includes regular face-to-face meetings and being available remotely (e.g. Skype, email, Slack).
Form of tender response
Interested parties should submit a costed proposal (in English) to [email protected] which include:
- a short (no more than 3 pages) explanation of your proposed scope and approach, including whether you would aim to create a new tool or improve an existing one. It should also include an explanation of the desired or expected impact and target audience(s) for the tool. Finally, the explanation should include an explanation of how the tool or its new iteration would help validate or realise our three hypotheses around quality, speed and cost.
- a short (no more than 2 pages) explanation on your proposed methodology, including whether you will work from already identified user needs or whether your work will include phases of user research to refine the scope and ambition of the tool
- an estimated breakdown of effort between the core software development activities and the creation and delivery of the other deliverables, including training and communication materials
- a description of the team who will do the work, including bios
- examples of relevant past work
- the costing should be at activity level, but feel free to provide more detail
The ODI reserves the right to make both anonymised questions and answers public or shared with other organisations having stated their interest.
All proposals will be assessed as described in our public procurement policy. In addition, for this procurement we will be looking for:
- evidence of past successful development of open source software for data management and publishing
- evidence of, or suitably advanced reflection on, the target audience, usefulness and potential impact of the tool to be developed on the state of the art of publishing open data
- the scale and importance of the type of open data that the tool supports the publication of (e.g. statistical data, streaming data)
- the number of existing tools and services that already support the proposed features. We believe that there should be a healthy variety of open data publishing tools, especially as it may be hard for a single tool to cover all needs well. We will, however, favour attempts to tackle a new problem or address an existing problem in a novel way rather than merely add another tool to an already healthy market.
Q: Would you welcome the inclusion of a public sector partner as the problem owner and if so if there is any guidance as to what role they could take?
A: The ability to align tool development to the actual needs of existing or new users is very important to us in this project, and we would certainly welcome the inclusion of a user/stakeholder/problem owner as part of your proposal.
Q: Are you looking for (a) a development partner to help you create (or expand) one or more tools which you will define, or (b) proposals for creating (or expanding) tools, which you will help fund?
A: our expectation is for the latter, but we would welcome proposals to develop any of the tools in our ODI Labs toolbox further: https://theodi.org/labs
Q: Are you wedded to any specific technology stacks or is the decision of the technology part of the project?
A: Decision on technology stack will be up to the successful candidate. We do encourage you to briefly mention in your proposal if you are already aware of libraries or other software you would be building upon or using in order to make your development more efficient. We expect proposals to make the case for how the proposed tool or feature will be adopted and lead to better publication of data. As such, integration with existing workflows and widely used tools is likely to be a positive factor in how we consider the proposals. This may also influence the technology stack used.
Q: We understand your report on the state of open-data tools is due to be published towards the end of September, but do you have any early indications of gaps or pain-points that you are able to share at this point?
A: Nothing to share at this point beyond the broad issues mentioned in the invitation to tender.
Q: Other than the mentioned report, what existing research/discovery has been completed and can we get access to the outcomes?
A: Ongoing research is mostly taking the shape of stakeholders/users interview and the organisation of a number of workshops with users (and possibly creators) of data publishing tools, to be held in the next month.
Q: You mention two week check ins, are we correct is assuming you are willing to run this project agile with a phased release?
Q: How many third parties are you expecting to use this portal and what do you know of their needs right now? Or is this decision part of the project?
A: We make no assumptions on the target user base of the tools developed through this programme of work.
Q: Who is the target audience for the training material that is to be produced?
A: We expect this material to be mainly of use to the users of the tool - with some expectations of technical literacy and proficiency. Secondary target audience is a slightly less technical group of decision makers at organisations willing to publish data (or publish it better/faster/etc) who may desire to consult documentation and training to get a sense of the value of the tool.