To enhance the resource utilization of shield tunnel muck and improve the mechanical properties of recycled concrete while investigating its failure mechanism, four concrete mix proportions—no coarse aggregate (NCS), single-sized aggregate (1C10), double-sized aggregate (2C15), and well-graded aggregate (3C20)—are designed. Uniaxial compression tests and digital image correlation (DIC) techniques are employed to analyze the strength development and failure mechanisms at 28-day and 56-day curing ages. The results show the following: (1) The NCS group achieves a 28-day compressive strength of 54.19 MPa, meeting the C50 design strength requirement. However, its 56-day strength development is limited, exhibiting the largest crack width and significant brittle failure characteristics. (2) The 1C10 group achieves the highest 56-day compressive strength (79.76 MPa), with tortuous crack propagation and a multi-crack ductile failure mode. (3) The 2C15 group demonstrates the optimal compressive toughness index, indicating excellent toughness and energy absorption capacity. (4) The 3C20 group shows significant strength regression (43.30 MPa), failing to meet the design strength requirement, with cracks penetrating aggregates and exhibiting obvious brittle failure. While the NCS group meets the basic strength requirement, but single-sized or double-sized aggregate gradations can effectively optimize crack paths and failure modes. It is recommended to control the maximum aggregate size not exceeding 15 mm in practical applications to ensure the strength and durability requirements of concrete made from shield muck.