C-SATS is a Seattle-based health tech startup formed in 2014. C-SATS is an objective surgical skills assessment system designed to help surgeons to continuously improve. The solution identifies opportunities for technical skills improvement by crowd-sourcing expertise and applying natural language processing, computer vision, and machine learning to intraoperative surgical videos.
My roles at C-SATS: VP of User Experience, UX strategist, interim Director of Product and product manager, hands-on designer and researcher, thought leader, dish washer. I joined C-SATS in the 3rd year of its life as employee 15.
Accomplishments
- 15 additional ML labels
- 55% videos w/context, an increase from 6% within one month after release
- 45% engagement email opens, an increase from 23% (2019 Mailchimp benchmark for Medical Healthcare is 21.48%)
- 35% engagement email clickthroughs, an increase from 3% (2019 Mailchimp benchmark for Medical Healthcare is 2.69%)
- 23 observed hours of in-person live surgeries (5 prostatectomy, 2 bariatric, 1 hysterectomy)
Why Does Case Context Matter?
C-SATS has a hardware product (tablet) in the operating room that is used to record and upload surgical videos. Surgeons can optionally attach HIPPA compliant patient data that can only be discovered during the operation. Intraoperative context would provide surgical skills reviewers with insight into the complexity of the case (procedure) that may have impacted a surgeon’s performance. For example, the size of the uterus can affect the efficiency of a hysterectomy.
The Problems
From ethnographic research, I discovered that we were making it difficult for users to provide case context:
- Surgeons rarely interacted with the tablet
- Nurses were actually the primary user of the tablet
- Nurses don’t have all the case context information
- In certain hospitals, only surgeons are allowed to provide case context
- The poor usability of tablet UI (native and app) for adding case context to the video
- It was difficult to find where to update case context via the app
As a result, only 6% of the videos had case context attached. The lack of insight caused surgeons to complain that some of the reviews were irrelevant and not actionable because certain information wasn’t taken into consideration by reviewers.
A solution was needed to get more videos with case context to reviewers to improve the quality of reviews without disrupting the existing O.R. orchestration.
The Solutions
1. Optimize the Review Process
It takes up to 4 hours to process a video before it is sent to reviewers. Then it takes about another 8 hours for a case to have a completed review. Then an engagement email is triggered to inform surgeons that a case report is ready. However, surgeons typically open this engagement email 24 hours after a video is uploaded. The process was not set up to get the most valuable use of time.
For improvement, optimized videos were held for 12 hours before they were released to reviewers. This gave surgeons an opportunity to add context to the case videos via the Web app before they were released to reviewers. The optimized process replaced user pain points with user benefits.
2. Improve Usability
The original tablet UI design used a free from text box to collect case context and the tablet screen keyboard (native) was also difficult to use. The spelling errors and unstructured data that were collected made tracking performance longitudinally inefficient and slowed down the progress of natural language processing (NLP) and computer vision (CV).
For improvement, we worked with surgeons to identify the common context that is used per type of procedures. We added them to the UI as buttons (structured data tags) but kept the free form text box in case of an uncommon event or circumstance.
As a result, we identified 15 new ML labels from using structured data to improve computer vision.
3. Improve Engagement
Surgeons received a notification email after a video is uploaded and contact support was the only call to action. It was not a useful email and difficult to see what surgery the email was referring to.
The email was redesigned with one main call to action and encouraged the behavior that is beneficial to surgeons, reviewers, and C-SATS. It informed surgeons the value of adding context to videos and displayed info were already attached to the video.
What Users Said About the Product:
“The feedback I have received from C-SATS and my telementoring session is constructive, not critical. It is instructive, positive, personalized and directed at my surgical skills and techniques. That is what makes C-SATS so powerful.”
Dr. Patricia Grade, Scottsdale Shea Medical Center
“I was pleasantly surprised by the feedback I received from the C-SATS platform. The scoring method was very objective and allowed me to anonymously compare my skill to those of my colleagues.”
Dr. Vijay Singh, Northwell Health
Fun Fact
This is what you wear to an orthopedic surgery lab. If it was a live surgery, surgeons would wear a full body suit that looks like what an astronaut would wear. I’m sure you can figure out on your own why someone needs to be covered from head to toe in an orthopedic surgery …
Alix Han 