#92. An Agent That Wouldn’t Stop and a Model that Won’t talk

For this week’s What I Read This Week, I explore a contrast in how AI is evolving. On one side, an OpenAI agent that bypassed restrictions on an Australian government health website raises important questions about autonomy, accountability and cyber resilience in an AI-enabled world. On the other, Jev, a new model designed to operate within tightly defined boundaries, offers a different vision of AI that prioritises predictability and confidence over open-ended generation.

Alongside these developments, I look at a new warning from RCGP Northern Ireland about the sustainability of general practice and what it means for plans to shift more care closer to home.

An OpenAI agent hacked an Australian health service website

On 18 June, an OpenAI agent carrying out what was described as benign research hit the Medicare Statistics Reporting Service run by Services Australia. When the portal blocked its requests, it didn’t stop. It tried alternative routes, got around the access restrictions, viewed both public and non-public files, and wrote files to an internal server. Australian officials believe no personal medical information was accessed and OpenAI’s own review found no evidence patient records were touched.

OpenAI discovered the activity in August while reviewing misaligned model behaviour, but didn’t notify Services Australia until 10 September (84 days after the event) and did so via the agency’s generic “public disclosures” mailbox, rather than through senior officials or the Australian Signals Directorate. The agency read the email the next day, verified it, and escalated to the ASD on 15 September. Ministers were briefed around the 17th; the public found out on the 24th.

Australian officials have called the delay and the method of notification unacceptable and raised Australia’s “extreme concern” directly with Sam Altman. A taskforce led by the Department of the Prime Minister and Cabinet is now investigating -including why government monitoring never detected the intrusion in the first place

In his interview with RTE radio, Puneet Kukreja discussed this situation and how AI does not remove the need for strong cyber security fundamentals, how organisations should increasingly assume breaches will occur and focus on rapid detection and response, and why critical services must identify the minimum level of operation they need to maintain through disruption.

Read more:

Jev: the AI model that refuses to talk

TypeSafe AI came out on 15 September with US$40m in seed funding led by DCVC and a model called Jev. Its founder is a former OpenAI researcher and one of the people behind RLHF - the technique that made ChatGPT behave the way it does.

Jev doesn’t generate text at all. It answers exactly three kinds of bounded question about data you hand it: Choice (pick one from a defined set), Score (place the input on an ordered scale) and Noul (a yes/no question answered as a probability). Every answer returns a probability distribution and a confidence value, and because the possible answers are fixed in advance, the output cannot fall outside your schema.

This article does the work of translating what this might mean for health. The authors candidates: guardrails around clinical LLMs, referral triage, terminology mapping and synthetic data validation. His exclusions are just as useful - anything numerical, anything that needs a rationale, and identifiable data on a US-hosted API.

So this is not a clinical decision tool and shouldn’t be dressed up as one. But the architectural idea is the interesting bit. A model that can only choose from a list you defined, and tells you how confident it is, is a different risk profile to Australia’s experience of an agent that doesn’t accept no for an answer.

RCGP NI: safeguard the future of general practice

RCGP Northern Ireland launched a new policy paper this week on Safeguarding the future of general practice in Northern Ireland, warning that the sustainability of GP services is at risk without urgent action. It sets out six recommendations for government and health leaders.

The numbers in the report are stark. General practice delivers 90% of patient contacts in the system, yet receives just 5.4% of total health spending - well behind the other devolved administrations. Twenty-nine practices have handed back their contracts since July 2022, while the number of registered patients keeps rising.

Dr Ursula Mason, Chair of RCGPNI, framed it as a transformation problem rather than purely a funding one. If we’re serious about shifting care closer to home, general practice has to be at the centre of those plans, and right now its future remains “fragile and uncertain”.

Originally published on LinkedIn.