The $p_{\rm T}$-differential production cross sections of the prompt charmed mesons $D^0$, $D^+$, $D^{*+}$ and $D_{\rm s}^{+}$ and their charge conjugate in the rapidity interval $-0.96 < y_{\rm cms} < 0.04$ were measured in p-Pb collisions at a centre-of-mass energy $\sqrt{s_{\rm NN}} = 5.02$ TeV with the ALICE detector at the LHC. The nuclear modification factor $R_{\rm pPb}$, quantifying the D-meson yield in p-Pb collisions relative to the yield in pp collisions scaled by the number of binary nucleon-nucleon collisions, is compatible within the 15-20% uncertainties with unity in the transverse momentum interval $1 < p_{\rm T} < 24$ GeV/$c$. No significant difference among the $R_{\rm pPb}$ of the four D-meson species is observed. The results are described within uncertainties by theoretical calculations that include initial-state effects. The measurement adds experimental evidence that the modification of the momentum spectrum of D mesons observed in Pb-Pb collisions with respect to pp collisions is due to strong final-state effects induced by hot partonic matter.