On Monday 19th October, we convened a cross-party, invitation-only roundtable in the House of Lords, hosted by Lord Clement-Jones and Baroness Owen. The discussion examined a pressing question for public policy: when citizens rely on AI for information about government services, rights and entitlements, who is accountable when the answers are wrong?
Drawing on findings from the ODI’s CitizenQuery research, the roundtable brought evidence of unreliable AI-generated advice about government services and policies into direct discussion with parliamentarians. Participants explored the quality of government information, independent assurance, liability and redress, and the safeguards needed to protect people using AI systems in their day-to-day lives. The discussion identified support for treating trusted, authoritative government data as critical public infrastructure and sharpened the questions that an effective policy response must answer.
Why this matters now
As AI chatbots become an increasingly important route to public information, their answers can influence decisions with serious consequences. A confident but inaccurate response about benefits, health or legal rights can prevent someone from accessing support or lead them to act on false advice. Participants highlighted the particular risks for people seeking help in dangerous circumstances, including women experiencing domestic abuse and benefit claimants needing timely and accurate advice about their entitlements.
These risks are unevenly distributed. People who most need reliable advice may have the fewest resources to verify an answer or challenge its consequences. Although the government does not control many of the interfaces citizens use, it can influence the quality, accessibility and authority of the information on which those systems draw.
What the evidence shows
The ODI’s CitizenQuery work has tested how AI chatbots respond to real questions about government services. It involved asking 22,000 questions across approximately 7,500 GOV.UK topics, checked against the relevant guidance. Ten AI models have been tested so far: five closed and five open. (A model is the underlying AI system that generates a chatbot’s responses, while “open” and “closed” refer to whether the model’s trained parameters are publicly available for others to download and use).
The early findings of this work revealed inconsistent performance. No model performed best across every topic, and smaller open models often outperform larger alternatives. Answers were frequently lengthy; prompting models to be more concise and deliver reduced accuracy. Models also failed to acknowledge uncertainty reliably. This means that models will often give emphatically ‘accurate’ responses, even when there is uncertainty.
Reflecting on these irregularities, participants in the roundtable suggested that in their own experience, citations do not consistently substantiate the claims they accompany, even when they refer to respected institutions such as the London School of Economics and Parliament.
Examples from the Citizen Query research presented at the roundtable included a carer being told that eligibility for an allowance depended on their child having died, and an answer misstating the legal requirements for a birth certificate. In the subsequent discussion, participants emphasised that even a 1% error rate could be unacceptable where people's rights or safety are at stake. Participants also raised concerns about confirmation bias and the confidence with which chatbots present plausible answers, which can compound these risks.
Several participants called for an independently stewarded benchmarking authority to measure the performance of different models over time. The case for independent assurance was clear: providers’ own assessments cannot, by themselves, establish whether systems deserve public trust.
Making government information ready for AI
A recurring question between participants was how to make GOV.UK information easier for AI systems to find, interpret and use accurately, and who should be responsible for the work and its cost. Suggestions included clear metadata identifying authoritative, current information, alongside standards to prove provenance, disambiguate meaning and make limitations explicit. Participants explored how systems could give appropriate weight to government guidance over less reliable material.
There was strong support for treating trusted data as critical public infrastructure, while recognising that the scale of investment required remains poorly understood. Transport for London’s open APIs and the services they enabled, including Citymapper, offered an example of government providing dependable infrastructure on which others can build. Participants considered whether a similar approach could support AI access through tools such as the Model Context Protocol. The implications extend beyond government services to financial advice and public health.
Sovereignty, trust and discrimination
Participants raised concerns about dependence on a small number of large technology companies and overseas providers for access to data and politically sensitive systems. Questions of ownership, control and commercial incentives were central to the discussion. Trust was understood as a matter of institutional conduct and integrity, requiring more than procedural compliance.
The discussion also addressed the risk that AI systems deployed for data matching and predictive profiling can reinforce discrimination. Participants raised concerns about medical data being used to harm individuals, including unequal effects on Black and minority ethnic communities, and the potential impacts on children. These concerns reinforced the need for meaningful safeguards and a fair relationship between citizens, the state and technology providers.
Emerging risks from connected data and AI agents
AI’s ability to connect previously separate datasets could help identify people in need of support, including victims of domestic abuse. The same capability raises difficult questions about privacy, inference and appropriate intervention. Public benefit and the sensitivity of the information must be considered together.
Participants also discussed the potential for AI agents to overwhelm (“flood”, or “swarm”) public services through large volumes of complaints, objections or requests. Planning departments, freedom of information teams, MPs’ offices and schools were among the services identified as exposed. AI-enabled scams and misleading financial advice on social platforms were highlighted as areas meriting further investigation.
Accountability, liability and redress
The discussion exposed unresolved questions about responsibility when AI-generated information causes harm. Participants questioned whether existing transparency mechanisms provide sufficient insight into how systems operate or meaningful routes to redress. The Algorithmic Transparency Recording Standard and proposals for algorithmic impact assessments were discussed as relevant parts of the policy response.
Reliable logging (records of what the AI system is doing, and the answers it gives) was identified as essential to evaluation, incident investigation and accountability. Participants doubted that keeping proper records is technically impossible, and worried that fear of legal liability could make organisations reluctant to keep them. Parliamentarians also sought clearer evidence of who had tested models, which methods had been used, what had been found and what action had followed.
Participants questioned how liability should be allocated when AI provides incorrect information about public services, including how responsibility should be shared between AI providers and public bodies. They explored whether product liability principles could be extended to AI, making providers responsible for harm caused by their systems rather than leaving users to bear the consequences of incorrect advice. One contribution suggested that some forms of product liability are absolute, raising the question of whether certain duties should extend beyond foreseeable harm. Participants also considered responsibility for harm resulting from a failure to act. The discussion explored whether an overarching duty of care could provide a broader framework, drawing on the Law Commission's work and considering the relevance of duties established under the Online Safety Act. These were questions for further policy and legal development, rather than an agreed allocation of responsibility or settled conclusions about the current law.
The susceptibility of models to misinformation and contamination added to the case for safeguards throughout their lifecycle. Monitoring must address both the acceptance of false information and the risk that models reject accurate material.
Children, education and AI literacy
Children’s access to information, learning and development required particular consideration. Participants highlighted the risk that, when a minor is the user, a chatbot’s apparent personality can obscure its limitations and encourage misplaced confidence. The ODI’s CitizenQuery programme found that this matches with verbosity, where long answers are too overwhelming for users to critically assess. AI literacy was seen as an important safeguard: people need to understand how these systems work, question their outputs and consult authoritative sources.
International approaches to education and digital literacy (including AI), including Estonia’s curriculum, were discussed as potential sources of learning. Clear labelling of AI-generated information was another practical suggestion. Participants also favoured a basic principle of honest presentation: AI systems should identify themselves as AI and should never present themselves as human. Effective safeguards were seen as necessary to sustain public confidence and preserve access to the technology’s benefits.
Responsibility across government
Responsibility for AI and data was described as fragmented across government, with some confusion around where responsibility lies. Participants questioned whether any department clearly owns the quality and accessibility of the information on which AI systems rely. The Cabinet Office and the Office for National Statistics were mentioned in that context, alongside the possibility of a dedicated minister for data (as separate to the minister for AI).
Suggestions included stronger central coordination across government to address the fragmentation of AI and data work across departments and public bodies. One participant proposed bringing these separate strands together in a shared institution, drawing on the Francis Crick Institute as an analogy: teams would work alongside one another rather than in silos, helping to connect expertise and move policy forward. This was an exploratory suggestion, rather than a developed proposal for the institution's remit, structure or powers. During the discussion, the practical requirement was consistently in favour of accessibility, sharing and interoperability being built into departmental responsibilities, supported by clear ownership and accountability.
The original proposed ODI recommendation
Where general-purpose AI systems provide information concerning government services, legal rights, public entitlements, health or public safety, providers should take reasonable and proportionate steps to ensure that:
- High-stakes responses are grounded in authoritative information,
- Material uncertainty is communicated to users,
- There is continuous monitoring and auditing by independent parties,
- There is a mechanism to report and analyse incidents and
- Effective mechanisms exist for correction and redress where foreseeable and demonstrable harm occurs.
Refining the ODI recommendation
The ODI presented a draft recommendation for scrutiny. Participants welcomed it as a useful starting point and pressed for greater precision about its scope and implementation. Their questions provide a clear agenda for further development:
- Define what “reasonable and proportionate” requires, who owes the duty and to whom, and how responsibilities should be allocated between providers and public bodies.
- Clarify whether “foreseeable and demonstrable” harm provides sufficient protection, including how harm can be assessed and prevented before it occurs.
- Specify the government action sought, with clear departmental ownership, coordination and measurable indicators of success.
- Address the authority of information presented to citizens, routes to correction and redress, and the contribution of education and AI literacy.
- Explore alignment with relevant Law Commission work on duty of care and product liability.
- The discussion established a basis for further work on the proposal while leaving important questions open.