First findings: Video may not be Zed's cup of tea

Work:

The Zed Stereo is a unique volumetric capture solution, compared to the litany of new depth capture devices being announced and released as the immersive media market expands - Unlike its competitors, it does not make use of Structured Light. There are no LIDAR or Infrared sensors powering this camera's ability to sense stereoscopy. Instead, it uses two 2K cameras to capture a standard stereoscopic photo / video feed, and then, using post-processing, determines the depth information from the spatial conflict in the resulting image.

While this sets it apart from other capture solutions, it's reliance on proprietary post-processing methods to determine depth means that, as an out of the box video capture solution, it simply doesn't work. While you're able to use the zed stereo application to capture stereo video, the only functions that come out of the box related to producing a 3D mesh out of that footage are relegated to still videogrammetry rather than outputting a depth-kit or mimesys style 'holographic' video file. It comes with a program called "ZedFu" which lets you do photogrammetric scene capture from live footage or pre-captured stereo footage, but, the resulting model is a still capture and there's no plans from zed related to creating a similar program for producing video. 

Still, the Zed SDK is open source and their integration with matlab and Cmake3d means that you *can* take that stereo footage and use it to produce animated videogrammetric models, provided you jump through some hoops and potentially write the software to do so yourself. It remains to be seen whether or not this kind of video capture is comparable to the content produced by a kinect v2. Notably missed is the kinect's ability to instantaneously chroma-key out the background of any shot and only focus on / produce footage of a human actor. While this sort of thing is potentially possible with the Zed, it would require either 'capture space cropping' a'la the depth kit - limiting the z-buffer depth on the camera so that the only thing being captured is the actor you're filming - or, again, some sort of software suite written specifically for this use case.

However, all is not lost! We purchased the zed stereo intending to use it for video capture, and it falls down in that regard - but it may just be the perfect plug-and-play solution for greenscreen mixed reality capture, as well as a powerful / portable solution for capturing photogrammetric room models

---

Next Steps

The Zed SDK comes with unity integration that allows for greenscreen mixed reality capture of VR gameplay - the necessity to pair a vive tracker or oculus touch with an external camera and a virtual camera in the virtual environment may be negated by the Zed's capacity to occlude virtual items with real humans. We need to try and test the mixed reality capabilities of Zed.

Feedback