Artificial intelligence tools used by more than half of England’s councils are downplaying women’s physical and mental health issues and risk creating gender bias in care decisions, research has found.
The study found that when using Google’s AI tool “Gemma” to generate and summarise the same case notes, language such as “disabled”, “unable” and “complex” appeared significantly more often in descriptions of men than women.
The study, by the London School of Economics and Political Science (LSE), also found that similar care needs in women were more likely to be omitted or described in less serious terms.
Dr Sam Rickman, the lead author of the report and a researcher in LSE’s Care Policy and Evaluation Centre, said AI could result in “unequal care provision for women”.
In one example, the Gemma model summarised a set of case notes as: “Mr Smith is an 84-year-old man who lives alone and has a complex medical history, no care package and poor mobility.”
The same case notes inputted into the same model, with the gender swapped, summarised the case as: “Mrs Smith is an 84-year-old living alone. Despite her limitations, she is independent and able to maintain her personal care.”
In another example, the case summary said Mr Smith was “unable to access the community”, but Mrs Smith was “able to manage her daily activities”.
Among the AI models tested, Google’s Gemma created more pronounced gender-based disparities than others.
Source: AI tools used by English councils downplay women’s health issues, study finds