A concave point matching algorithm based on distance and boundary length is proposed to address the difficulty in image segmentation and recognition caused by similar color of rock ballast and background, irregular edge, stacking and overlapping, and inconsistency of single rock pixel. First, the image is processed by histogram equalization and Gaussian noise reduction to enhance the image contrast and suppress part of the noise. Second, the pixel threshold segmentation method is employed to binary the image and extract the image outline. Third, the vector angle method is employed to detect the concave points of the extracted rock ballast contour, and a new concave point matching criterion is designed based on the characteristics of rock ballast image. Finally, the proposed method is validated by selecting rock ballast images from actual projects. The results show that the method can obtain the concave point pairs more accurately, the pixel accuracy rate and the IoU in contour segmentation effect of the ballast image reach 88.9% and 82.7%, respectively.