欢迎访问我的网站,希望内容对您有用,感兴趣的可以加入免费知识星球。

caffe-gpu源码编译

人工智能 迷途小书童 3年前 (2020-11-18) 3618次浏览 0个评论

软硬件环境

  • ubuntu 18.04 64bit
  • NVidia GTX 1070Ti
  • anaconda with python 3.7
  • CUDA 10.1
  • cuDNN 7.6
  • opencv 3.4.2
  • caffe 1.0.0

视频看这里

此处是 youtube 的播放链接,需要科学上网。喜欢我的视频,请记得订阅我的频道,打开旁边的小铃铛,点赞并分享,感谢您的支持。

简介

先说一下环境,使用 anacondapython 虚拟环境,支持 opencv,支持 CUDAcuDNN 加速,支持在 python 中调用 caffe。基础组件部分可以参考前面的文章,本文就不赘述了

基础环境准备

安装依赖包和工具

sudo apt install build-essential cmake git ffmpeg libatlas-base-dev libtiff-dev pkg-config python3-dev libavcodec-dev libavformat-dev libswscale-dev libtbb-dev libjpeg-dev libpng-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libx264-dev libboost-all-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libhdf5-dev
pip install protobuf

opencv

这里把 opencv 单独拿出来说,是因为 opencv 的安装方法非常多

  • apt install python3-opencv
  • conda install opencv
  • 源码编译

通过 apt install 安装最简单,也是最不容易出错的方法;其次是 conda install,最容易出问题的是自己编译源码,编译参数复杂,依赖库繁多,而且还有版本差异。

安装完成后,建议使用 opencv_version 命令来查看当前版本,默认 ubuntu 18.04 源提供的是 3.2.0 版本,conda 的会更高一些,这里是 3.4.0,源码安装的话,注意在 sudo make install 后再执行一句 sudo ldconfig。本文以 conda 的方式进行安装。

编译caffe

接下来就可以编译 caffe

git clone https://github.com/BVLC/caffe.git
cd caffe
cp Makefile.config.example Makefile.config

编辑文件 Makefile.config,主要是一些路径的修改,贴上已经修改好的

## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).
# 启用cuDNN加速
USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers
# 启用opencv
USE_OPENCV := 1
# USE_LEVELDB := 0
# USE_LMDB := 0
# This code is taken from https://github.com/sh1r0/caffe-android-lib
USE_HDF5 := 1

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
#   You should not set this flag if you will be reading LMDBs with any
#   possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3
# opencv大版本号是3,这里一定要注意
OPENCV_VERSION := 3

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.
# -gencode arch=compute_20,code=sm_21
# CUDA_ARCH := -gencode arch=compute_20,code=sm_20 
# 这里使用的是CUDA10.1,所以要注释掉前两行
CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \
        -gencode arch=compute_35,code=sm_35 \
        -gencode arch=compute_50,code=sm_50 \
        -gencode arch=compute_52,code=sm_52 \
        -gencode arch=compute_60,code=sm_60 \
        -gencode arch=compute_61,code=sm_61 \
        -gencode arch=compute_61,code=compute_61

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas

# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
# Python头文件路径,再加上numpy的头文件路径
PYTHON_INCLUDE := /home/xugaoxiang/anaconda3/include/python3.7m \
        /home/xugaoxiang/anaconda3/lib/python3.7/site-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
        # $(ANACONDA_HOME)/include/python2.7 \
        # $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include

# Uncomment to use Python 3 (default is Python 2)
# 默认是python2,这里使用python3,一定要改,不然后面会报错相应没人在用python2了吧
PYTHON_LIBRARIES := boost_python3 python3.7m
# PYTHON_INCLUDE := /usr/include/python3.5m \
#                 /usr/lib/python3.5/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.
# libpython*.so库的路径
PYTHON_LIB := /home/xugaoxiang/anaconda3/lib
# 如果设置了ANACONDA_HOME环境变量,可以使用下面的设置方法,作用一样
# PYTHON_LIB := $(ANACONDA_HOME)/lib

# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)
# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /home/xugaoxiang/anaconda3/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial/

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib

# NCCL acceleration switch (uncomment to build with NCCL)
# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
# USE_NCCL := 1

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# 启用pkg_config,方便caffe找到opencv
USE_PKG_CONFIG := 1

# N.B. both build and distribute dirs are cleared on `make clean`
# 默认编译的目录,所有的目标文件、可执行文件、库都存放在这里
BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# 是否打开debug信息
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?= @

完成后,执行

make all -j12

参数 -j 指的是使用多少个 cpu 核心,目的是加快编译速度,根据自己的实际情况设定

为了能在 python 中调用 caffe,还需要执行

make pycaffe -j12

至此,整个编译就结束了。

验证

使用 ipython 环境测试

caffe

如果细心一点,会发现,新开一个 terminal,同样打开 ipython,同样 import caffe,但是会报错,这是什么原因?

caffe

在不报错的 terminal 中,查看环境变量就会发现端倪

caffe

在编译 caffe 的过程中,会 export 环境变量 PYTHONPATH,所以我们在使用前也需要这样做

caffe

为了简便,可以将声明语句写入 ~/.bashrc 中,就不用每次都去执行了

export PYTHONPATH=/home/xugaoxiang/Works/github/caffe/python:$PYTHONPATH

Q & A

Q1

编译过程碰到了 tiff 相关的错误

caffe

这是由于之前 opencv 源码编译引起的,这里特别要注意一点,如果是从源码开始编译 opencv,那么在配置的时候一定要加上选项 -D BUILD_TIFF=ON。还有就是尽量不要同时拥有 aptconda 安装的2种环境,对新手来说比较容易出错。

Q2

进入 ipython 中,import caffe 报错

caffe

libhdf5_hl.so.100 的路径加入 LD_LIBRARY_PATH

export LD_LIBRARY_PATH=/home/xugaoxiang/anaconda3/lib:$LD_LIBRARY_PATH

Q3

进入 ipython 中,import caffe 报错

caffe

修改 Makefile.config,修改 PYTHON_LIBRARIES

PYTHON_LIBRARIES := boost_python3 python3.7m

默认的是 python2

Q4

关于 caffe 中使用源码编译的 opencv4,由于 opencv4 的版本差异,都会报错

caffe

这是由于在 opencv4 中,原来版本中的宏 CV_LOAD_IMAGE_COLORCV_LOAD_IMAGE_GRAYSCALE 已经改成了 cv::IMREAD_COLORcv::ImreadModes::IMREAD_GRAYSCALE,所以,需要在 caffe 源码目录中查找并替换,才能够编译成功

caffe

Q5

编译过程中报 hdf5 头文件找不到?即使是执行了安装命令也一样

sudo apt install libhdf5-dev
src/caffe/layers/hdf5_data_layer.cu:10:10: fatal error: hdf5.h: No such file or directory
 #include "hdf5.h"
          ^~~~~~~~
compilation terminated.
Makefile:604: recipe for target '.build_release/cuda/src/caffe/layers/hdf5_data_layer.o' failed
make: *** [.build_release/cuda/src/caffe/layers/hdf5_data_layer.o] Error 1
make: *** Waiting for unfinished jobs....

这时候需要修改 Makefile.config 文件,在 INCLUDE_DIR 后添加路径 /usr/include/hdf5/serial,在 LIBRARY_DIR 后添加 /usr/lib/x86_64-linux-gnu/hdf5/serial

参考资料

喜欢 (0)

您必须 登录 才能发表评论!

Ads Blocker Image Powered by Code Help Pro

Ads Blocker Detected!!!

请关闭 Adblock 等类似浏览器插件,然后刷新页面访问,感谢您的支持!

We have detected that you are using extensions to block ads. Please support us by disabling these ads blocker.