Skip to content

Telestration: How Helena Mentis Applies Design Thinking to Surgery

Telestration: How Helena Mentis Applies Design Thinking to Surgery

Telestration: How Helena Mentis Applies Design Thinking to Surgery

Helena Mentis is the director of the Bodies in Motion Lab at University of Maryland, Baltimore County (UMBC) with research spanning human-computer interaction (HCI), computer supported cooperative work (CSCW) and medical informatics. During a recent visit to the Design Lab at UC San Diego, Mentis talked about her research on surgery in the operating room.

She examines the medical world through surgical instruments and the workflow inside the operating room. Mentis hones in on minimally invasive surgery and the reliance toward images.  She is particularly interested in how medical professionals see and share visual information in a collaborative fashion, which has grown over the past several years. She asks, “What happens if surgeons were given greater control over the image? What would happen to the workflow? Would it change anything?”

In one study at Thomas Hospital in London, surgeons were using a lot of pointing gestures to direct the operation. Confusion would arise and the surgeon would need to repeat his exact intention with others. This break in the workflow inspired Mentis’ team to ask: what if we were to build a touchless illustration system that responded to the surgeon’s gestures? Her team set out to build what she calls “telestration,” which enables surgeons to use gestures to illustrate their intentions through an interactive display.

During another operation, the surgeon encountered a soft bone and had to stop the procedure. As a result, the surgeon had take off their gloves to re-examine the tissue on the visual display. Mentis notes, “There is a tight coupling between images on display and feeling with the instrument in hand.” If the image on display could be more closely integrated with the workflow, would this save time in the operating room?After publishing her findings, people raved over how voice narration rather than gesture aided imaging and collaboration in surgery. Consequently Mentis asked, “If given the opportunity would doctors use voice or gesture?” The ensuing observations revealed that while doctors stated their preference for voice, gesture was more frequently used for shaping telestration images. While voice narration and gestures provided greater interaction with the image, surgeons actually spent more time in surgery. Mentis reasons, “There is more opportunity for collaborative discussion with the information.” Interestingly, this did add time to the overall operation, but it also yielded greater opportunities to uncover and discuss critical information.

About Helena Mentis, Ph.D.

Assistant Professor, Department of Information Systems
University of Maryland, Baltimore County

Helena Mentis, Ph.D., is an assistant professor in the Department of Information Systems at the University of Maryland, Baltimore County. Her research contributes to the areas of human-computer interaction (HCI), computer supported cooperative work (CSCW), and health informatics. She investigates how new interactive sensors can be integrated into the operating room to support medical collaboration and care. Before UMBC, she was a research fellow at Harvard Medical School, held a joint postdoctoral fellowship at Microsoft Research Cambridge and the University of Cambridge, and was an ERCIM postdoctoral scholar at Mobile Life in Sweden. She received her Ph.D. in Information Sciences and Technology from Pennsylvania State University.

Helena Mentis is the director of the Bodies in Motion Lab at University of Maryland, Baltimore County (UMBC) with research spanning human-computer interaction (HCI), computer supported cooperative work (CSCW) and medical informatics. During a recent visit to the Design Lab at UC San Diego, Mentis talked about her research on surgery in the operating room.

She examines the medical world through surgical instruments and the workflow inside the operating room. Mentis hones in on minimally invasive surgery and the reliance toward images.  She is particularly interested in how medical professionals see and share visual information in a collaborative fashion, which has grown over the past several years. She asks, “What happens if surgeons were given greater control over the image? What would happen to the workflow? Would it change anything?”

In one study at Thomas Hospital in London, surgeons were using a lot of pointing gestures to direct the operation. Confusion would arise and the surgeon would need to repeat his exact intention with others. This break in the workflow inspired Mentis’ team to ask: what if we were to build a touchless illustration system that responded to the surgeon’s gestures? Her team set out to build what she calls “telestration,” which enables surgeons to use gestures to illustrate their intentions through an interactive display.

During another operation, the surgeon encountered a soft bone and had to stop the procedure. As a result, the surgeon had take off their gloves to re-examine the tissue on the visual display. Mentis notes, “There is a tight coupling between images on display and feeling with the instrument in hand.” If the image on display could be more closely integrated with the workflow, would this save time in the operating room?After publishing her findings, people raved over how voice narration rather than gesture aided imaging and collaboration in surgery. Consequently Mentis asked, “If given the opportunity would doctors use voice or gesture?” The ensuing observations revealed that while doctors stated their preference for voice, gesture was more frequently used for shaping telestration images. While voice narration and gestures provided greater interaction with the image, surgeons actually spent more time in surgery. Mentis reasons, “There is more opportunity for collaborative discussion with the information.” Interestingly, this did add time to the overall operation, but it also yielded greater opportunities to uncover and discuss critical information.

About Helena Mentis, Ph.D.

Assistant Professor, Department of Information Systems
University of Maryland, Baltimore County

Helena Mentis, Ph.D., is an assistant professor in the Department of Information Systems at the University of Maryland, Baltimore County. Her research contributes to the areas of human-computer interaction (HCI), computer supported cooperative work (CSCW), and health informatics. She investigates how new interactive sensors can be integrated into the operating room to support medical collaboration and care. Before UMBC, she was a research fellow at Harvard Medical School, held a joint postdoctoral fellowship at Microsoft Research Cambridge and the University of Cambridge, and was an ERCIM postdoctoral scholar at Mobile Life in Sweden. She received her Ph.D. in Information Sciences and Technology from Pennsylvania State University.

Helena Mentis is the director of the Bodies in Motion Lab at University of Maryland, Baltimore County (UMBC) with research spanning human-computer interaction (HCI), computer supported cooperative work (CSCW) and medical informatics. During a recent visit to the Design Lab at UC San Diego, Mentis talked about her research on surgery in the operating room.

She examines the medical world through surgical instruments and the workflow inside the operating room. Mentis hones in on minimally invasive surgery and the reliance toward images.  She is particularly interested in how medical professionals see and share visual information in a collaborative fashion, which has grown over the past several years. She asks, “What happens if surgeons were given greater control over the image? What would happen to the workflow? Would it change anything?”

In one study at Thomas Hospital in London, surgeons were using a lot of pointing gestures to direct the operation. Confusion would arise and the surgeon would need to repeat his exact intention with others. This break in the workflow inspired Mentis’ team to ask: what if we were to build a touchless illustration system that responded to the surgeon’s gestures? Her team set out to build what she calls “telestration,” which enables surgeons to use gestures to illustrate their intentions through an interactive display.

During another operation, the surgeon encountered a soft bone and had to stop the procedure. As a result, the surgeon had take off their gloves to re-examine the tissue on the visual display. Mentis notes, “There is a tight coupling between images on display and feeling with the instrument in hand.” If the image on display could be more closely integrated with the workflow, would this save time in the operating room?After publishing her findings, people raved over how voice narration rather than gesture aided imaging and collaboration in surgery. Consequently Mentis asked, “If given the opportunity would doctors use voice or gesture?” The ensuing observations revealed that while doctors stated their preference for voice, gesture was more frequently used for shaping telestration images. While voice narration and gestures provided greater interaction with the image, surgeons actually spent more time in surgery. Mentis reasons, “There is more opportunity for collaborative discussion with the information.” Interestingly, this did add time to the overall operation, but it also yielded greater opportunities to uncover and discuss critical information.

About Helena Mentis, Ph.D.

Assistant Professor, Department of Information Systems
University of Maryland, Baltimore County

Helena Mentis, Ph.D., is an assistant professor in the Department of Information Systems at the University of Maryland, Baltimore County. Her research contributes to the areas of human-computer interaction (HCI), computer supported cooperative work (CSCW), and health informatics. She investigates how new interactive sensors can be integrated into the operating room to support medical collaboration and care. Before UMBC, she was a research fellow at Harvard Medical School, held a joint postdoctoral fellowship at Microsoft Research Cambridge and the University of Cambridge, and was an ERCIM postdoctoral scholar at Mobile Life in Sweden. She received her Ph.D. in Information Sciences and Technology from Pennsylvania State University.

Read Next

Derek Lomas and Philip Guo Recognized by Premier International HCI Conference

UC San Diego Design Lab members Derek Lomas and Philip Guo were recently recognized by…

The Design Lab’s Srishti Palani Wins Google’s PhD Fellowship

In continuing excellence among UC San Diego Design Lab researchers, Srishti Palani, a PhD student and Department of Cognitive Science at the Design Lab, was named a 2021 Google PhD Fellow for her work in Human Computer Interaction focused on improving web search and intelligent guidance during creative work. The fellowship is open to researchers in computer science and related fields. Palani was one of 60 students throughout the world to be selected for a Google Fellowship–an award that supports outstanding and promising PhD candidates of all backgrounds who seek to influence the future of technology by providing funding, mentorship, collaboration and internship opportunities.

Palani’s research takes an interdisciplinary approach involving cognitive, computer, and learning sciences to better web search and intelligent scaffolding of complex creative information work. “We use web search almost every day to search things, and it affects how we learn and work and create and collaborate. I’m really passionate about researching this area and building novel computational techniques that integrate web search into people's larger work context. Google, of course, has the most state-of-the-art web search technology that has existed in my lifetime, so I’ve always wanted to collaborate with the researchers, software engineers, and data scientists there to understand how we can get a better web search when people want to search for more complex information needs.”
Olga McConnell

Olga McConnell, Project Specialist and Executive Assistant to the Director of The Design Lab

As the Executive Assistant to the Director of The Design Lab, a project manager for the Lab’s special projects and annual events, and a lifelong learner who holds a M.A. in English Linguistics and Translation, and a M.B.A. in Business Administration and Management, Olga McConnell’s zest for knowledge is palpable. She is currently on track to complete a Project Management Certification at UC San Diego Extension at the end of 2021, and she is planning on obtaining her Project Management Professional (PMP) Certification after that. “I’m kind of addicted to getting degrees,” jokes McConnell. “I even thought the other day, maybe I’ll go to law school. And then I was like, no, enough, enough.” 

For nearly five years, McConnell was Executive Assistant to Don Norman, the Founding Director Emeritus of The Design Lab. She is now the Executive Assistant to the new Director of The Design Lab, Mai Thi Nguyen. It is Nguyen’s vision of human-technology-community interactions, along with her JEDI (justice, diversity, equity and inclusion) approach that has McConnell excited about this new chapter in the Lab’s legacy, saying, “I see how great she is as an efficient leader, so I’m really looking forward to working with her, supporting her administratively, as well as taking charge of certain projects that she has in mind.”
Ucsd San Diego

Civic Design – DSGN 160: Open Enrollment

This studio course explores how to design products, services, and policies for complex socio-technical systems. The class will follow a human-centered design process that includes user research, concept generation, prototyping, pitching, and alliance building. Students will work in teams to design solutions to civic challenges affecting people in the San Diego region. This will be a good course for intermediate to advanced design students who want to build up their portfolio and to practice their skills with sketching, storyboarding, prototyping, and evaluating services for real-world problems.

As part of this class, teams will enter the Design for San Diego 2020 challenge (D4SD.org) and interact with other innovators, experts, and mentors throughout the city to address problems related to Mobility, Health, Environment, and Housing. Top teams will have an opportunity to present their work and win prizes at events in downtown San Diego!
Design Lab Faculty

New Design Lab Faculty Working to Shape the Future of UC San Diego

As a global pioneer in design thinking, research, and invention, The Design Lab prides itself on recruiting the brightest and most innovative minds in the design field. Today, we would like to extend a warm welcome to brand new faculty members Elizabeth Eikey, Haijun Xia, and Edward Wang!

Elizabeth Eikey
From a first-generation undergraduate student at Penn State, then an inquisitive Best Buy employee and finally, to the Assistant Professor in the Department of Family Medicine and Public Health & The Design Lab at UCSD, Dr. Elizabeth Eikey has an illustrious career. Her research work at The Design Lab focuses on the intersection between technology, mental health, and equity, primarily studying the possible applications for technology in supporting mental health and well-being.

Haijun Xia
After receiving his PhD in Computer Science from University of Toronto, Xia made the move across countries to begin his time as a researcher at UC San Diego. ‘I wanted to work at The Design Lab and UC San Diego, because of the diversity of skill here,’ says the Professor, ‘We are all approaching the many challenging research questions from different angles, which is really important to develop comprehensive solutions.

Edward Wang
When Edward Wang was an undergraduate student at Harvey Mudd, he never expected himself to become a researcher, let alone becoming a professor. It was only after a Professor offered him the chance to help design a course she was planning about biosignal processing, that he began on this path. ‘As I was designing the class over summer, I had to read a bunch of papers,’ he says, ‘I couldn’t stop thinking about how cool all of it was. Especially when it branched out into computer science and how it could be involved in biosignal processes.’
Tricia Ngoon

Tricia Ngoon, UCSD & Design Lab PhD Graduate, Discusses “Adaptive Conceptual Guidance”

Currently, in the spotlight of Tricia Ngoon’s research and involvement with The Design Lab is her recently accepted paper, Shöwn: Adaptive Conceptual Guidance Aids Example Use in Creative Tasks, which will appear in the Designing Interactive Systems virtual conference this summer, 2021. Her research hypothesizes that providing “adaptive conceptual guidance” will improve a person’s implementation of examples within creative work, as opposed to providing a static example. Using the domain of web comics, “[researchers in the study] present concepts to people alongside examples as they work.” Ngoon adds that “It’s essentially a step towards coaching. For example, if [a person is] working on a comic you might present a concept to consider the framing or kind of the composition of the panel and then [show] examples of different types of framing and composition.” Ultimately, her research concluded that “these adaptive suggestions as a person is working in context really help with making a clear and more unique story. It kind of changes the way they look at their ideas because they are more likely to explore different [ones].” 
Back To Top