We present a high-resolution P-wave azimuthally anisotropic velocity model for the upper and middle crust beneath southern California by a novel adjoint-state traveltime tomography technique. Our model reveals significant anisotropy variations between tectonic blocks that clearly reflect both past and current plate boundary deformation. In the shallow crust, seismic anisotropy is mostly controlled by the preferred alignment of microcracks related to the present N-S compressive stress; while at deeper depths (>∼6 km), seismic anisotropy mainly records paleofabrics formed during the long-lived Farallon subduction and later extension that have not been fully reset by the present transform motion. Interestingly, our model demonstrates distinct fast axes beneath the western Transverse Ranges from its neighboring blocks, probably reflecting the large-scale vertical axis clockwise rotation of the block. In addition, we identify layered structures with distinct anisotropy features beneath the Salton Trough, which could be a result of the current transtension.