The delay on line 227 in CameraServer is only called if a buffer is present.
My team fixed it by making a local copy of the class and moving the delay out of the catch block to line 231.

In other words, we changed:
catch (VisionException ex) {
    DriverStation.reportError("Error when getting image from the camera: " + ex.getMessage(), true);
    if (dataBuffer != null) {
      synchronized (this) {
          m_imageDataPool.addLast(dataBuffer);
          Timer.delay(.1);
      }
    }
}

to:
catch (VisionException ex) {
    DriverStation.reportError("Error when getting image from the camera: " + ex.getMessage(), true);
    if (dataBuffer != null) {
      synchronized (this) {
          m_imageDataPool.addLast(dataBuffer);
      }
    }
}
Timer.delay(.1);