Brief Introduction

Firstly, in the offline training phase, we need to collect the video frame data and the corresponding vehicle speed, steering angle through the experimental vehicle. We input the video frame data and the corresponding vehicle speed and steering angle into the pre-built neural network frame by frame, and use it to train the network to predict the vehicle speed and steering angle. Secondly, in the actual vehicle test process, the frame data collected by the camera is input into the trained neural network, and the neural network calculates and predicts the vehicle speed and the steering angle. The predicted vehicle speed and steering angle are input into the control system through the vehicle's CAN bus to realize the control of the vehicle.

The overall network structure is mainly divided into four parts: spatial feature extraction, semantic segmentation subtask, time feature extraction and decision network.

The purpose of spatial feature extraction and semantic segmentation subtask is to extract better spatial feature information for subsequent motion decision making. The purpose of time feature extraction is to integrate the current time image and vehicle motion information through the historical information of images and vehicle motion to make more accurate decisions.

In order to regress the speed increment and steering angle better, we finally adopted a fully connected network as the decision network and the amount of regression control.