工作职责:

1、Brief Introduction: My scope ranges in both Pony L4 and L2 prediction teams. Since I joined Pony to present, I served as Tech Lead in L4 prediction team and own several projects. As the core member I participate in building Pony Prediction 3.0 Architecture based on deep learning, which completely replaced previous machine learning based methods and drawn high performance benefits. Recently I serve as the Team Leader in L2 prediction. I am working to integrate team members to build highly efficient prediction algorithms in limited resources.

2、Prediction 3.0: Core contributor to build the basic architecture of prediction BEV model and onboard.
∗ In Prediction 3.0, data-driven evolves to be proposal-driven. A general idea is that for an obstacle, multipletrajectory proposals are generated based on road-graph and rules. Then proposals are to be classified througha deep model and conclude to one or some topK proposals as final trajectory results.
∗ Model inputs consists of multiple-channel feature maps, where road-graph, obstacle information with historystates, signal types, motion status, traffic light status and adv motion information are gathered. Model inputswill be converted based on reference system depending on intersection and non-junction scenarios.
∗ A hourglass like encoder-decoder model is proposed to extract high dimensional feature vectors for eachposition around ADV. Given proposals above, each proposal can be represented using feature vectors andutilized for classification and regression.
∗ My work mainly contributes to non-junction scenarios where the reference system is the ADV. With the generalmodel, trajectory classification/ranking/tuning/motion estimation can all be added without much extra works.

3、Driveway Vertical Project in L4: Lead prediction and pnc to improve L4 performance in driveway scenarios. Driveway is a specific definition in Pony, you can just regard it as a T-Intersection in normal roads.
∗ Organize several regular vertical issue meetings weekly to summarize driveway issue patterns, prepare issue potential fixes, plan new feature designs and arrange weeky progress.
∗ Implement Prediction 3.0 model to driveway scenarios and deliver a fusion model which can handle both outward-driveway and into-driveway cases. Compared with previous rule based methods, the model can largely reduce false positives around driveway and can output correct predictions much earlier.
∗ In vertical-driveway statistics, Pony L4 MPCI improves 10X and MPHB improves 5X, which significantly boosts prediction performance in driveway scenarios.

4、Learning Based Motion Prediction in L4: First learning based work to estimate motion in Prediction and demonstrate yield-to-adv intentions in high precision to reduce most HBs in driveway scenarios.
∗ Based on Prediction 3.0 general model, a transformer based structure is implemented to encode history motion information. Soft-Discretization techniques are used to model discretized motion states. Multi-Head cross attention mechanisms are used to model relations among target obstacle, ADV and its neighbor obstacles.
∗ The transformer based model can obtain single-frame future motion status including velocity/acceleration, the LSTM is implemented to further smooth the motion predictions and directly regress the motion functions.
∗ This contribution is successfully launched in driveway project. It is observed that most true positive yield-to-adv intentions can be earlier detected and HBs are largely reduced.

5、Prediction in L2: As the Team Leader, I am working to build a strong team to support various L2 projects.
∗ The goals and resources are totally different between L2 and L4. L2 concerns HBs more while L4 concerns CIs
more. And usually there are no HD maps in L2 and perception qualities are worse compared with L4.
∗ My team works to rapidly develop a simple data-driven prediction architecture to rapidly support L2 projects.The prediction target simplifies to be when the obstacle will goto which lane. For each obstacle, its history observations, motion states, relationships with the candidate paths, relations with neighbor obstacles are gathered into handcrafted features. A random-forest model is used to such classification task.

6、General Acceleration: Implement a general pipeline to decrease model/onboard latency in Prediction.
∗ Cooperate with ML-Infra team to create a simple model pruning and distillation pipeline. After this work,
model latency in prediction module can be decreased 30% in average without accuracy loss.
∗ Integrate prediction BEV models in non-junction/outward-driveway/into-driveway to a ‘shared backbone‘ +‘multi-heads‘ architecture. In onboard, the gpu memory pointers of backbone feature will be held and assigned to individual processing units. Time cost in onboard can be decreased 30% without accuracy loss.

工作职责:

1、Brief Introduction: I am the deputy Team Leader of the vision based perception group in SenseTime Company. As the only interface person between the model group and the system group, I am fully responsible for the ADAS model system engineering module to ensure high-quality model docking and project delivery. Meanwhile, I lead the research and development of ADAS segmentation model and multi object tracking system. I have rich experience in the field of model acceleration, model deployment and post-processing algorithm in SDK.

2、ADAS Scene Segmentation Group: Lead the R & D of the monocular segmentation model in Sense-ADAS.

∗ Responsible for the training, deployment and quantitative acceleration of road scene perception models such aslane line, lane line attribute and driving area segmentation model, and supports the smooth delivery of allproject models of SenseTime ADAS.

∗ Rich experience in model performance improvement, and the accuracy of lane and freespace models has beenimproved by more than 10 percentage points. The model accuracy and corresponding properties meet therequirements of national standards in a number of mass production projects.

∗ Responsible for the research of semantic segmentation and monocular depth estimation model. As the coredeveloper, I deeply participate in the R & D of company-level semantic segmentation framework, which has been applied in the product lines of various business groups and has been widely praised.

3、ADAS Multi-Object Tracking System: Lead the R & D of multi-object tracking algorithms.

∗ Design the “conventional feature fusion” metrics and the smoothing algorithm based on “kalman filter +regression fitting”, which greatly improves the stability of the original MOT algorithm, and can ensure that the target TrackID does not change when the vehicle is running normally in all projects delivered.

∗ Design a novel MOT framework with the integration of deep ReId features, which improves MOTA by morethan 5 points without increasing the overall time-consuming of the algorithm. The scheme will be popularizedin the ADAS project of SenseTime ADAS.

∗ Responsible for the model training and system engineering of MOT module with high quality.

4、ADAS Model & System & Engineering Module: Lead the R & D of perception algorithms in ADAS-SDK.

∗ The goal of my team is to build a bridge between ”model development” and ”system application” to improve the quality of model delivery, and ensure the specification of algorithm research and development.

∗ Responsible for the R & D of each sub-module of perception post-processing, including but not limited to: lane line, driving area, 2-stage / 1-stage / anchor free detection, MOT, monocular ranging algorithm, etc.

∗ Successfully deliver the ADAS project of United Auto mass production. At the same time, more than 5projects are in progress. We support LDW / FCW / PCW / TSR / TLR and other functions. The performance meets the requirements of national standards.

∗ As the core R & D, participate in a number of POC projects (SAIC / Nidec, etc.), in which TensorRT is used to deploy the joint model of semantic segmentation and depth estimation on Xavier, and the segmentation accuracy reaches 0.7 in 100ms.

∗ Rich experience in model compression and optimization covering Ambrella / FPGA / QualCOMM / CUDA.

5、ADAS Small Object Detection + OCR: Lead the R & D of OCR techniques in small object detection.

∗ The object is to detect and recognize the limit-height and limit-weight in traffic signs. The pipeline algorithmof ”one-stage detection + two-stage classification + three-stage text detection and recognition” is proposed. At present, the algorithm can accurately identify the limit-height sign at 50m, and the time consumption is not more than 3ms.

∗ Optimize the MOT algorithm for small object scenarios, and add functions such as category buffer, delay display and missing detection compensations to effectively improve the stability of small object detection.

∗ Implement the whole pipeline in SenseTime ADAS SDK and support all current projects.