Correlations between mean transverse momentum $[p_{\rm T}]$ and anisotropic flow coefficients $v_{\rm 2}$ or $v_{\rm 3}$ are measured as a function of centrality in Pb$-$Pb and Xe$-$Xe collisions at $\sqrt{s_{\rm NN}} = 5.02$ TeV and 5.44 TeV, respectively, with ALICE. In addition, the recently proposed higher-order correlation between $[p_{\rm T}]$, $v_{\rm 2}$, and $v_{\rm 3}$ is measured for the first time, which shows an anticorrelation for the presented centrality ranges. These measurements are compared with hydrodynamic calculations using IP-Glasma and $\rm T_{R}ENTo$ initial-state shapes, the former based on the Color Glass Condensate effective theory with gluon saturation, and the latter a parameterized model with nucleons as the relevant degrees of freedom. The data are better described by the IP-Glasma rather than the $\rm T_{R}ENTo$ based calculations. In particular, Trajectum and JETSCAPE predictions, both based on the $\rm T_{R}ENTo$ initial state model but with different parameter settings, fail to describe the measurements. As the correlations between $[p_{\rm T}]$ and $v_{\rm n}$ are mainly driven by the correlations of the size and the shape of the system in the initial state, these new studies pave a novel way to characterize the initial state and help pin down the uncertainty of the extracted properties of the quark$-$gluon plasma recreated in relativistic heavy-ion collisions.