AI-powered smart glasses are helping novice scientists work like experts
A new wearable AI system monitors your hands through smart glasses, guiding experiments and stopping mistakes before they happen
View of the lab bench as seen through LabOS glasses.
Kang Group, Stanford University
Imagine you’re standing at a lab bench working on an experiment, when you complete a step, a display inside your lab goggles tells you what to do next. A small camera in the frame takes a close look at your hands. If you reach the wrong tube, the display flashes a warning. Before you make a mistake, the system tells you how to get back on track.
Laboratory safety goggles have finally joined the ranks of smart devices. this is the promise behind LabOSAn AI “operating system” for scientific laboratories created by the Stanford-Princeton AI Coscientist Team, a group led by bioengineer Le Kang of Stanford University and computer scientist Mengdi Wang of Princeton University, with founding partners that include NVIDIA. Powered by NVIDIA’s vision-language model for processing visual data, the system is designed to provide AI with real-time knowledge of lab work to determine what causes experiments to fail or succeed and rapidly train new scientists to expert level by guiding them through experimental protocols.
Walk into a wet lab, Kang says, and “it hasn’t changed much in the last 50 years.” He explains that this matters because, a great deal of the time, science is “done in a physical laboratory, in the physical world, not on a computer.” As described in a recent preprint paper, LabOS aims to bridge this physical-digital divide.
On supporting science journalism
If you enjoyed this article, consider supporting our award-winning journalism Subscribing By purchasing a subscription, you are helping ensure a future of impactful stories about the discoveries and ideas shaping our world today.
The scientific community has long grappled with a problem known as the “replication crisis” for more than a decade. in 2016 Nature surveyMonya Baker, then editor of the journal, reported that “more than 70% of researchers have tried to reproduce another scientist’s experiments and failed,” and more than half could not reproduce their own work. Some of that failure rate is due to statistical misconduct or publication pressure. But one common reason receives less attention: Frequently, humans working in the laboratory make mistakes. A reagent added at the wrong temperature, a step skipped under time pressure, a contaminated pipet tip – these are errors that may be too small to notice but big enough to ruin an experiment.
A researcher using LabOS glasses next to a robotic arm.
Kang Group, Stanford University
The solution proposed by Wang and Kang’s team is an open-source platform and hardware kit that lets AI see what scientists see. Early pilot tests in Kang’s lab at Stanford and Wang’s lab at Princeton have researchers wear augmented reality/extended reality (AR/XR) glasses that stream video directly into the system. LabOS compares what it sees to written protocols, collecting training data as well as providing guidance to the wearer. AI can talk the scientist through each step, reminding them to keep surfaces sterile or flagging flaws in the technique.
AI requires real-time knowledge of experiments to learn what works and what doesn’t, just as robots and self-driving cars need to gather real-world data to update their systems. “We can have 1,000 chatbots, 1,000 AI scientists trying to tell real scientists what to do,” Wang says, but if the AI isn’t incorporated into a physical experiment, “we don’t have anything verifiable.”
Typically when humans work in a laboratory, learning can be slow. If an experiment fails, they try to determine what went wrong and start again. But when AI looks at an experiment and sees the results, it may be able to more quickly determine which steps caused problems and design a new experiment. By recording entire experiments, an AI can study the smallest details to determine what caused them to fail.
This observation extends beyond human guidance; LabOS also uses a robotic arm to handle difficult tasks like mixing. “It’s not like changing people,” Kang says. “We need to help people.”
So far, the aid is yielding results. In an experimental procedure that involved increasing the amount of a certain protein in cells, junior scientists with only a week of LabOS training achieved results that were almost indistinguishable from expert scientists. “As a professor, I couldn’t tell the difference,” says Kang. “The results of the experiment – they’re the same.”
“From a robotics and human-computer interaction perspective, this work highlights a promising direction,” says Kourosh Darwish, a scientist in the AI and Automation Lab at the University of Toronto’s Acceleration Consortium, who was not involved in LabOS development. Yet he notes the importance of developing standards to better evaluate such work. “As AI systems increasingly move from analytical tools to active participants in experimentation, community-level standardization and validation will be critical.”
The AI Coscientist team is already advancing this technology beyond the research bench. Recently researchers introduced medOSAdapting its AI-and-AR architecture to assist surgeons in anatomical mapping and tool alignment. Ultimately, Wang says, the broader ambition is to transform “every scientific research laboratory” — and soon, every clinic — into an “AI-comprehensible and AI-operable environment,” creating a system that can rapidly train professionals, catch mistakes, and improve human outcomes.
It’s time to stand up for science
If you enjoyed this article, I would like to ask for your support. scientific American He has served as an advocate for science and industry for 180 years, and right now may be the most important moment in that two-century history.
i have been one scientific American I’ve been a member since I was 12, and it’s helped shape the way I see the world. Science Always educates and delights me, and inspires a sense of awe for our vast, beautiful universe. I hope it does the same for you.
if you agree scientific AmericanYou help ensure that our coverage focuses on meaningful research and discovery; We have the resources to report on decisions that put laboratories across America at risk; And that we support both emerging and working scientists at a time when the value of science is too often recognised.
In return, you get the news you need, Captivating podcasts, great infographics, Don’t miss the newsletter, be sure to watch the video, Challenging games, and the best writing and reporting from the world of science. you can even Gift a membership to someone.
There has never been a more important time for us to stand up and show why science matters. I hope you will support us in that mission.
