Measurements of anisotropic flow coefficients ($v_n$) and their cross-correlations using two- and multi-particle cumulant methods are reported in collisions of pp at $\sqrt{s} = 13$ TeV, p-Pb at $\sqrt{s_{_{\rm NN}}} = 5.02$ TeV, Xe-Xe at $\sqrt{s_{_{\rm NN}}} = 5.44$ TeV, and Pb-Pb at $\sqrt{s_{_{\rm NN}}} = 5.02$ TeV recorded with the ALICE detector. The multiplicity dependence of $v_n$ is studied in a very wide range from 20 to 3000 particles produced in the mid-rapidity region $|\eta|<~0.8$ for the transverse momentum range $0.2 <~ p_{\rm T} <~ 3.0$ GeV/$c$. An ordering of the coefficients $v_2 > v_3 > v_4$ is found in pp and p-Pb collisions, similar to that seen in large collision systems, while a weak $v_2$ multiplicity dependence is observed relative to nucleus-nucleus collisions in the same multiplicity range. Using a novel subevent method, $v_{2}$ measured with four-particle cumulants is found to be compatible with that from six-particle cumulants in pp and p-Pb collisions. The magnitude of the correlation between $v_n^2$ and $v_m^2$, evaluated with the symmetric cumulants SC$(m,n)$ is observed to be positive at all multiplicities for $v_2$ and $v_4$, while for $v_2$ and $v_3$ it is negative and changes sign for multiplicities below 100, which may indicate a different $v_n$ fluctuation pattern in this multiplicity range. The observed long-range multi-particle azimuthal correlations in high multiplicity pp and p-Pb collisions can neither be described by PYTHIA 8 nor by IP-Glasma+MUSIC+UrQMD model calculations, and hence provide new insights into the understanding of collective effects in small collision systems.