IOT>> MobileNet-SSDLite-RealSense-TF>> 返回
项目作者： PINTO0309

项目描述 ：
RaspberryPi3(Raspbian Stretch) + MobileNetv2-SSDLite(Tensorflow/MobileNetv2SSDLite) + RealSense D435 + Tensorflow1.11.0 + without Neural Compute Stick(NCS)
高级语言： Python
项目地址: git://github.com/PINTO0309/MobileNet-SSDLite-RealSense-TF.git
创建时间： 2018-07-08T04:44:55Z
项目社区：https://github.com/PINTO0309/MobileNet-SSDLite-RealSense-TF
开源协议：MIT License
下载


MobileNet-SSDLite-RealSense-TF

RaspberryPi3(Raspbian Stretch) + MobileNetv2-SSDLite(Tensorflow/MobileNetv2SSDLite) + RealSense D435 + Tensorflow + without Neural Compute Stick(NCS)

Change history

Change history

[Dec 02, 2018]　Corresponds to OpenCV3.4.3,Tensorflow v1.11.0, Protobuf 3.6.1, librealsense v2.16.5, D435 Firmware v5.10.6

Environment

  • RaspberryPi3 + Raspbian Stretch
  • OpenCV 3.4.3 (Nov 25, 2018 updated)
  • VFPV3 or TBB(Intel Threading Building Blocks)
  • Tensorflow 1.11.0 (Nov 25, 2018 updated)
  • Protobuf 3.6.1 (Nov 25, 2018 updated)
  • librealsense v2.16.5 (Nov 25, 2018 updated)
  • cmake 3.11.4
  • MobileNetv2-SSDLite [MSCOCO]
  • RealSense D435 (Firmware ver v5.10.6)
  • Python3.5
  • Numpy
  • OpenGL

Videos under test by Ubuntu 16.04

Ubuntu1604

By RaspberryPi3 with OpenGL

RaspberryPi3

RaspberryPi environment construction sequence

0.【Run in Ubuntu 16.04 environment】 Realsense D435’s Firmware update

  1. $ echo 'deb http://realsense-hw-public.s3.amazonaws.com/Debian/apt-repo xenial main' | sudo tee /etc/apt/sources.list.d/realsensepublic.list
  2. $ sudo apt-key adv --keyserver keys.gnupg.net --recv-key 6F3EFCDE
  3. $ sudo apt-get update
  4. $ sudo apt-get install intel-realsense-dfu*
  5. $ mkdir v5.10.6;cd v5.10.6
  6. $ wget -O v5.10.6.zip https://downloadmirror.intel.com/28237/eng/Intel%C2%AE%20RealSense%E2%84%A2D400%20Series%20Signed%20Production%20Firmware%20v5_10_6.zip
  7. $ unzip v5.10.6.zip
  9. $ lsusb
  10. ### Below is sample.
  11. Bus 002 Device 003: ID 8086:0b07 Intel Corp.
  1. $ intel-realsense-dfu -b 002 -d 003 -f -i ./Signed_Image_UVC_5_10_6_0.bin
  2. $ intel-realsense-dfu -p
  3. FW version on device = 5.10.6.0
  4. MM FW Version = 255.255.255.255

1.【Run in RaspberryPi environment】 Extend the SWAP area

  1. $ sudo nano /etc/dphys-swapfile
  2. CONF_SWAPSIZE=2048
  4. $ sudo /etc/init.d/dphys-swapfile restart;swapon -s

2-1.Install tensorflow 1.11.0 (Raspbian Stretch)

  1. $ sudo -H pip3 install pip --upgrade
  2. $ sudo apt-get install python-pip python3-pip python3-scipy libhdf5-dev
  3. $ sudo apt-get install -y openmpi-bin libopenmpi-dev
  4. $ sudo pip3 uninstall tensorflow
  5. $ wget -O tensorflow-1.11.0-cp35-cp35m-linux_armv7l.whl https://github.com/PINTO0309/Tensorflow-bin/raw/master/tensorflow-1.11.0-cp35-cp35m-linux_armv7l_jemalloc.whl
  6. $ sudo pip3 install tensorflow-1.11.0-cp35-cp35m-linux_armv7l.whl
  7. 【Required】Restart the terminal

2-2.Install tensorflow 1.11.0 (Ubuntu16.04 x86_64)

  1. $ sudo -H pip3 install pip --upgrade
  2. $ sudo -H pip3 install tensorflow==1.11.0 --upgrade

3.Install package and update udev rule

  1. $ sudo pip3 install pillow lxml jupyter matplotlib cython
  2. $ sudo apt install -y git libssl-dev libusb-1.0-0-dev pkg-config libgtk-3-dev \
  3. libglfw3-dev at-spi2-core libdrm* python-tk libjpeg-dev libtiff5-dev \
  4. libjasper-dev libpng12-dev libavcodec-dev libavformat-dev \
  5. libswscale-dev libv4l-dev libxvidcore-dev libx264-dev qt4-dev-tools \
  6. autoconf automake libtool curl libatlas-base-dev
  7. $ cd /etc/udev/rules.d/
  8. $ sudo wget https://raw.githubusercontent.com/IntelRealSense/librealsense/master/config/99-realsense-libusb.rules
  9. $ sudo udevadm control --reload-rules && udevadm trigger

4.Install cmake-3.11.4

  1. $ cd ~
  2. $ wget https://cmake.org/files/v3.11/cmake-3.11.4.tar.gz
  3. $ tar -zxvf cmake-3.11.4.tar.gz;rm cmake-3.11.4.tar.gz
  4. $ cd cmake-3.11.4
  5. $ ./configure --prefix=/home/pi/cmake-3.11.4
  6. $ sudo make -j1
  7. $ sudo make install
  8. $ export PATH=/home/pi/cmake-3.11.4/bin:$PATH
  9. $ source ~/.bashrc
  10. $ cmake --version
  11. cmake version 3.11.4

5.Update LD_LIBRARY_PATH

  1. $ nano ~/.bashrc
  2. export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
  4. $ source ~/.bashrc

6-1.Install protobuf 3.6.1 (Raspbian Stretch)

  1. $ cd ~
  2. $ wget https://github.com/google/protobuf/releases/download/v3.6.1/protobuf-all-3.6.1.tar.gz
  3. $ tar -zxvf protobuf-all-3.6.1.tar.gz
  4. $ cd protobuf-3.6.1
  5. $ ./configure
  6. $ make -j1
  7. $ sudo make install
  8. $ nano ~/.bashrc
  9. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/<username>/protobuf-3.6.1/src/.libs
  11. $ source ~/.bashrc
  12. $ cd python
  13. $ python3 setup.py build --cpp_implementation 
  14. $ python3 setup.py test --cpp_implementation
  15. $ sudo python3 setup.py install --cpp_implementation
  16. $ export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp
  17. $ export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION=3
  18. $ sudo ldconfig
  19. $ protoc --version

6-2.Install protobuf 3.6.1 (Ubuntu16.04 x86_64)

  1. $ cd git;mkdir protobuf;cd protobuf
  2. $ wget -O protoc-3.6.1-linux-x86_64.zip https://github.com/protocolbuffers/protobuf/releases/download/v3.6.1/protoc-3.6.1-linux-x86_64.zip
  3. $ unzip protoc-3.6.1-linux-x86_64.zip
  4. $ rm protoc-3.6.1-linux-x86_64.zip
  5. $ sudo mv -f bin/* /usr/local/bin/
  6. $ sudo mv -bf include/* /usr/local/include/
  7. $ sudo chown $USER /usr/local/bin/protoc
  8. $ sudo chown -R $USER /usr/local/include/google
  9. $ sudo ldconfig
  10. $ nano ~/.bashrc
  11. export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION=2
  13. $ source ~/.bashrc

7.Install TBB (Raspbian Stretch / Intel Threading Buiding Blocks)

  1. $ cd ~
  2. $ wget https://github.com/PINTO0309/TBBonARMv7/raw/master/libtbb-dev_2018U2_armhf.deb
  3. $ sudo dpkg -i ~/libtbb-dev_2018U2_armhf.deb
  4. $ sudo ldconfig
  5. $ rm libtbb-dev_2018U2_armhf.deb

8.Install OpenCV 3.4.3(Raspbian Stretch / with TBB, with DNN, with OpenGL)

  1. $ cd ~
  2. $ wget -O opencv.zip https://github.com/Itseez/opencv/archive/3.4.3.zip
  3. $ unzip opencv.zip;rm opencv.zip
  4. $ wget -O opencv_contrib.zip https://github.com/Itseez/opencv_contrib/archive/3.4.3.zip
  5. $ unzip opencv_contrib.zip;rm opencv_contrib.zip
  6. $ cd ~/opencv-3.4.3/;mkdir build;cd build
  7. $ cmake -D CMAKE_CXX_FLAGS="-DTBB_USE_GCC_BUILTINS=1 -D__TBB_64BIT_ATOMICS=0" \
  8.         -D CMAKE_BUILD_TYPE=RELEASE \
  9.         -D CMAKE_INSTALL_PREFIX=/usr/local \
  10.         -D INSTALL_PYTHON_EXAMPLES=OFF \
  11.         -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib-3.4.3/modules \
  12.         -D BUILD_EXAMPLES=OFF \
  13.         -D PYTHON_DEFAULT_EXECUTABLE=$(which python3) \
  14.         -D INSTALL_PYTHON_EXAMPLES=OFF \
  15.         -D BUILD_opencv_python2=ON \
  16.         -D BUILD_opencv_python3=ON \
  17.         -D WITH_OPENCL=OFF \
  18.         -D WITH_OPENGL=ON \
  19.         -D WITH_TBB=ON \
  20.         -D BUILD_TBB=OFF \
  21.         -D WITH_CUDA=OFF \
  22.         -D ENABLE_NEON:BOOL=ON \
  23.         -D ENABLE_VFPV3=ON \
  24.         -D WITH_QT=OFF \
  25.         -D BUILD_TESTS=OFF ..
  26. $ make -j1
  27. $ sudo make install
  28. $ sudo ldconfig

9.Install Intel® RealSense™ SDK 2.0

  1. $ cd ~
  2. $ git clone -b v2.16.5 https://github.com/IntelRealSense/librealsense.git
  3. $ cd ~/librealsense
  4. $ git checkout -b v2.16.5
  5. $ mkdir build;cd build
  7. $ cmake .. -DBUILD_EXAMPLES=true -DCMAKE_BUILD_TYPE=Release
  8. OR
  9. $ cmake .. -DBUILD_EXAMPLES=true
  11. $ make -j1　# When running on a resource rich PC, "make -j8"
  12. $ sudo make install

10.Install OpenCV Wrapper

  1. $ cd ~/librealsense/wrappers/opencv;mkdir build;cd build
  2. $ cmake ..
  3. $ nano ../latency-tool/CMakeLists.txt
  4. target_link_libraries(rs-latency-tool ${DEPENDENCIES} pthread)
  5. $ make -j $(($(nproc) + 1))
  6. $ sudo make install
  7. $ cd ~/librealsense/build
  9. #Python3.x
  10. $ cmake .. -DBUILD_PYTHON_BINDINGS=bool:true -DPYTHON_EXECUTABLE=$(which python3)
  12. OR
  14. #Python2.x
  15. $ cmake .. -DBUILD_PYTHON_BINDINGS=bool:true -DPYTHON_EXECUTABLE=$(which python)
  17. $ make -j1
  18. $ sudo make install
  19. $ nano ~/.bashrc
  20. export PYTHONPATH=$PYTHONPATH:/usr/local/lib
  22. $ source ~/.bashrc

11.Installing the OpenGL package for Python

  1. $ sudo apt-get install python-opengl
  2. $ sudo -H pip3 install pyopengl
  3. $ sudo -H pip3 install pyopengl_accelerate
  5. $ sudo reboot

12-1.Introduction of model data of MobileNet-SSDLite (Raspbian Stretch)

  1. $ mkdir tensorflow;cd tensorflow
  2. $ git clone --recurse-submodules https://github.com/tensorflow/models.git
  3. $ nano ~/.bashrc
  4. export PYTHONPATH=$PYTHONPATH:/home/pi/tensorflow/models/research:/home/pi/tensorflow/models/research/object_detection
  6. $ source ~/.bashrc
  7. $ cd ~/tensorflow/models/research
  8. $ protoc object_detection/protos/*.proto --python_out=.
  9. $ cd ~/tensorflow/models/research/object_detection
  10. $ wget http://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz
  11. $ tar -xzvf ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz
  12. $ rm ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz

12-2.Introduction of model data of MobileNet-SSDLite (Ubuntu16.04 x86_64)

  1. $ mkdir tensorflow;cd tensorflow
  2. $ git clone --recurse-submodules https://github.com/tensorflow/models.git
  3. $ nano ~/.bashrc
  4. export PYTHONPATH=$PYTHONPATH:/home/<username>/tensorflow/models/research:/home/<username>/tensorflow/models/research/object_detection
  6. $ source ~/.bashrc
  7. $ cd ~/tensorflow/models/research
  8. $ protoc object_detection/protos/*.proto --python_out=.
  9. $ cd ~/tensorflow/models/research/object_detection
  10. $ wget http://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz
  11. $ tar -xzvf ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz
  12. $ rm ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz

13.Reduce the SWAP area to the default size

  1. $ sudo nano /etc/dphys-swapfile
  2. CONF_SWAPSIZE=100
  4. $ sudo /etc/init.d/dphys-swapfile restart;swapon -s

14.Enable OpenGL Driver

  1. $ sudo raspi-config
  2. 「7.Advanced Options」-「A7 GL Driver」-「G2 GL (Fake KMS)」 and Activate Raspberry Pi's OpenGL Driver

15.MobileNet-SSD execution

  1. $ cd ~
  2. $ git clone https://github.com/PINTO0309/MobileNet-SSDLite-RealSense-TF.git
  3. $ cd MobileNet-SSDLite-RealSense-TF
  4. $ python3 MobileNetSSDwithRealSenseTF.py