How to Use TensorFlow with ZED Introduction The ZED SDK can be interfaced with TensorFlow for adding 3D localization of custom objects detected with Tensorflow Object Detection API. In this tutorial, we will show you how to detect, classify and locate objects in 3D using the ZED stereo camera and TensorFlow SSD MobileNet inference model. Installation The 3D Object Detection project depends on the following libraries: Python 3 CUDA ZED SDK ZED Python API cuDNN Tensorflow Tensorflow Object Detection API OpenCV ZED SDK Install the ZED SDK and ZED Python API. cuDNN Install cuDNN. Check the support matrix for the corresponding CUDA and driver version. Tensorflow Object Detection API Install Tensorflow with GPU support by reading the following instructions for your target platform. # GPU package for CUDA-enabled GPU cards pip3 install --upgrade tensorflow-gpu Install Tensorflow Object Detection API by following these instructions and download the model repository. git clone https://github.com/tensorflow/models Test that you have correctly installed the Tensorflow Object Detection API by running the following command: python object_detection/builders/model_builder_test.py Note: If you get an import error, make sure that tensorflow/models/research/slim directories have been added to PYTHONPATH. This can be done by running the following command: # From tensorflow/models/ export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim Running 3D Object Detection Download the sample project code from GitHub. Make sure the virtualenv is active. source ~/tensorflow/bin/activate Run the code with python3. You should be detecting objects captured by your ZED camera using MobileNet and localizing them in 3D. python3 object_detection_zed.py Testing other models In this example, we’re using the computationally efficient MobileNet model for detecting objects. You can change this by updating the MODEL_NAME variable and selecting another one from Tensorflow model zoo. These models will be downloaded and extracted automatically. For example, a ResNet model can used by changing MODEL_NAME to : # Full model name required MODEL_NAME = ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03 Testing a custom model Other custom object detection models can be loaded by modifying the PATH_TO_FROZEN_GRAPH, variable typically called frozen_inference_graph.pb. Notes CUDA / cuDNN version Please refer to the Tensorflow compatibility table to know the appropriate cuDNN and CUDA versions for a given Tensorflow version. At the moment of writing, Tensorflow requires CUDA 9.0 and cuDNN 7. Architecture This sample uses 2 threads, one for the ZED images capture and one for the Tensorflow detection. While it may seem complex at first, it actually solves 2 issues: Performance is increased, as depth computation is done in parallel to inference. Tensorflow and the ZED SDK uses CUDA GPU computation and therefore requires the use of CUDA contexts. Since we currently can’t share the CUDA Context between the ZED and TF, we have to separate the GPU computation. Each CUDA context must therefore have its own thread.