VR Environment for Bone Surgery

We are developing a visuo-haptic environment for simulating a variety of surgical procedures whose core component is bone drilling. The core environment is flexible, and has been used on otologic, craniofacial, and dental datasets. This page contains a high-level description of the system and screenshots/movies representing each of these applications. For more detailed
technical information about the underlying environment, see our publications
based on this environment:

Environment Description

Analysis and Metrics

 

Overview

Our simulation runs on a dual-CPU Windows PC, and was developed in C++. OpenGL is used for graphic rendering (optionally in stereo, via shutter glasses or a HMD), and two SensAble Phantom haptic devices are used for simulating a drill and a suction device. We typically use a Phantom 1.0 for the drill and a Phantom Omni for the suction device. Sound is rendered as a noisy sum of harmonics, whose frequencies are based on experimental recordings of surgical drill burrs.

Several features, including sound and dynamic bone dust movement, are best demonstrated via video; each section below contains a link to a short movie. Most of the movies are encoded in mp4, either as QuickTime mp4 files or as avi files. If you have trouble playing them, I recommend downloading the 3ivx codec, a nice (free) mp4 codec package.

This video (wmv) is an overview of the system, showing several clinical applications simultaneously.

haptic simulation of bone<br /><br /><br /> surgery

 

Application 1: Temporal Bone Surgery

This video (QuickTime mp4) demonstrates basic drilling on the surface of the mastoid bone. Note the simulated drill sound, whose frequency varies realistically with applied force. As bone dust accumulates, the surgeon uses his suction tool to make the surface visible again.
This video (QuickTime mp4) demonstrates the virtual nerve monitor that is incorporated into our simulation environment. The surgeon is operating deep in the ear canal and has exposed the facial nerve; when he approaches it with the drill, the visual and audio monitors indicate increased activity, a warning that the nerve is at risk of being damaged.
This video (3ivx-encoded avi) demonstrates the interface used in the simulator to select tools from a menu. The user pulls the haptic device out of the working field – as a surgeon would do before requesting a new tool – and selects a new tool from a “haptic menu”. Here the user demonstrates the drill, suction, flashlight, and camera tools.
This video (3ivx-encoded avi) demonstrates the implementation of drill surface nonuniformity (the tendency of the equator of a drill to remove bone more effectively than the pole) in our environment.
This video (3ivx-encoded avi) demonstrates the implementation of drill “running forces” (tangential forces) in our environment. The red arrow indicates the tangential force. This video demonstrates the sensitivity of this force to drill orientation.
This video (3ivx-encoded avi) demonstrates the use of our metrics console for the playback and evaluation of performance on the simulator.

Our system was originally developed to simulate temporal bone surgeries, and was built around a set of polygonal models supplied by a collaborating otologist. The temporal bone itself is converted to a voxel representation, which is used for haptic rendering and for simulating material removal. The following sequence of images shows a typical manipulation of a temporal bone.

Important features displayed here are:

  • The bone surface is re-triangulated as voxels are removed, allowing real-time rendering of the bone as a polygonal mesh
  • Shadows provide a critical depth cue; the bone surface was difficult to find initially, even with haptic feedback, in preliminary experiments in which shadows were not present.
  • A particle system is used to represent accumulated bone dust, which trains a practicing surgeon to use the suction tool frequenly to keep the relevant area adequately visible.

Click on any image for a larger version.

image of temporal bone surgical simulation image of temporal bone surgical simulation image of temporal bone surgical simulation image of temporal bone surgical simulation

Further drilling reveals the complex inner structure of the temporal bone (note the gaps representing the mastoid air cells) and a gradual change in color as the user approaches the anatomy situated behind the bone (in this case a blue tint, representing the sigmoid sinus). This is a critical cue intraoperatively. Also note that before applying suction, the accumulated bone dust obscures this important information.

The following sequence of images also includes the virtual neural monitor, which displays the simulated activity of the facial nerve (in this example). If the drill approaches the monitored nerve, the activity meter begins spiking visually and via audio. It is also possible to stream neural activity information to a remote computer, simulating the neural monitoring station often
used in otologic procedures.

image of temporal bone surgical simulation image of temporal bone surgical simulation

A final sequence of images displays a zoomed-in view that the surgeon might use for operating deep within the bone. Additional drilling also reveals the inner ear anatomy present at the end of the ear canal.

image of temporal bone surgical simulation image of temporal bone surgical simulation image of temporal bone surgical simulation

 

Application 2: Craniofacial Surgery

video<br /><br /><br /> of craniofacial surgical simulation This video (QuickTime mp4) demonstrates basic drilling on the surface of the skull, near a mandibular fracture. The user is making the first fracture typically introduced in a distraction osteogenesis procedure. Note that as the tool approaches the alveolar nerve, the virtual nerve monitor provides a visual and auditory warning. For demonstration, the user then exposes the nerve visually.
image of craniofacial surgical simulation This image displays a complete skull, originally derived from a CT scan. The CT data was isosurfaced to isolate the bone, then converted into the simulation environment’s volumetric representation.
image of craniofacial surgical simulation Here the user prepares to begin drilling near a mandibular fracture. Shadows again provide a useful depth cue, although depth presentation is enhanced significantly by the optional stereo display.
image of craniofacial surgical simulation with neurophysiology monitor Here we represent a virtual nerve in blue, for visual clarity. Note the bursts of activity on the neural monitor, which result from close contact between the drill and the virtual nerve.
image of craniofacial surgical simulator with bone dust simulation The user begins to drill on the surface of the bone. Again, bone dust accumulates and obscures the user’s view until suction is applied.
image of craniofacial surgical simulation with gui This image demonstrates the complete GUI available in the training mode of our simulation. Note the ability to toggle among multiple drill head types and sizes, the ability to connect to a remote copy of the application to demonstrate drilling technique over the network (visually and haptically), and the variety of haptic parameters that can be modified. Normally these would not be modified during a training procedure, but we use these parameters as a way of allowing an experienced surgeon to tune the haptic feedback provided by our system.

 

Application 3: Dental Surgery

http://www.stanford.edu/group/sailsbury_robotx/cgi-bin/salisbury_lab/wp-content/uploads/2013/04/tooth_movie.jpg This video (QuickTime mp4) demonstrates basic drilling on the surface of a tooth, illustrating the flexibility of our environment in loading diverse data sets.

Additional images provide more detailed views of the bone surface during simulated drilling…

image of dental simulator image of dental simulator image of dental simulator image of dental simulator

Leave a Reply