Google deployed a deep learning system for diabetic retinopathy in Thai clinics.

When Google conducted field trials of its AI system for diabetic retinopathy in Thailand, the results were an eye-opener. The first study of its kind, it evaluated how nurses use an AI system to screen patients for the disease. While lab results were highly accurate, the system encountered a lot of difficulties in real-life testing. (MIT Technology Review)

Thailand’s initiative against diabetic retinopathy

Diabetic retinopathy happens when high blood sugar levels damage blood vessels in the retina and cause loss of vision even blindness. Thailand has targeted to annually screen 60% of its population for the disease.

Google’s AI can take an eye scan as input and identify the presence of diabetic retinopathy in fewer than 10 minutes with higher than 90% accuracy. The system looks within images for clues to the presence of the disease such as blocked or leaky blood vessels.

The question was, how does it perform in a real-life, clinical setting?

To find out, Emma Beede, a UX researcher at Google Health, and her team deployed Google’s deep-learning system at 11 eye clinics in Thailand.

How it worked

In a manual system, the nurses took the eye scans and sent them off to a specialist. The round trip to diagnosis could take as much as 10 weeks.

Google’s 10-minute diagnosis should have been revolutionary. Only it wasn’t.

Over the 11 months of the testing, the researchers found the system failed when presented with a poorer quality eye scan compared to what it was trained on. The reason accounted for nearly 20% of a failure to deliver a result.

The nurses had to repeat the imaging process and sometimes had to request the patient to return another day.

Again, erratic or slow internet connections meant it took very long to upload the scans to the cloud. That kept patients waiting at the clinic.

Based on these observations Google is now reworking its systems and procedures for the diabetic retinopathy algorithm.

Conclusion

“Deploying an AI system by considering a diverse set of perspectives in the design and development process is just one part of introducing new health technology that requires human interaction,” writes Beede. “It’s important to also study and incorporate real-life evaluations in the clinic, and engage meaningfully with clinicians and patients before the technology is widely deployed.”

Related Story:   Artificial Intelligence: AI Can Provide Advance Warning of Diabetes, the Silent Killer