The production cross sections of the prompt charmed mesons D$^0$, D$^+$, D$^{*+}$ and D$_s$ were measured at mid-rapidity in p-Pb collisions at a centre-of-mass energy per nucleon pair $\sqrt{s_{\rm NN}}=5.02$ TeV with the ALICE detector at the LHC. D mesons were reconstructed from their decays D$^0\rightarrow{\rm K}^-\pi^+$, D$^+\rightarrow{\rm K}^-\pi^+\pi^+$, D$^{*+}\rightarrow D^0\pi^+$, D$_s^+\rightarrow\phi\pi^+\rightarrow{\rm K}^-{\rm K}^+\pi^+$, and their charge conjugates. The $p_{\rm T}$-differential production cross sections were measured at mid-rapidity in the interval $1<~p_{\rm T}<~24$ GeV/$c$ for D$^0$, D$^+$ and D$^{*+}$ mesons and in $2<~p_{\rm T}<~12$ GeV/$c$ for D$_s$ mesons, using an analysis method based on the selection of decay topologies displaced from the interaction vertex. The production cross sections of the D$^0$, D$^+$ and D$^{*+}$ mesons were also measured in three $p_{\rm T}$ intervals as a function of the rapidity $y_{\rm cms}$ in the centre-of-mass system in $-1.26<~y_{\rm cms}<~0.34$. In addition, the prompt D$^0$ cross section was measured in pp collisions at $\sqrt{s}=7$ TeV and p-Pb collisions at $\sqrt{s_{\rm NN}}=5.02$ TeV down to $p_{\rm T}=0$ using an analysis technique that is based on the estimation and subtraction of the combinatorial background, without reconstruction of the D$^0$ decay vertex. The nuclear modification factor $R_{\rm pPb}(p_{\rm T})$, defined as the ratio of the $p_{\rm T}$-differential D-meson cross section in p-Pb collisions and that in pp collisions scaled by the mass number of the Pb nucleus, was calculated for the four D-meson species and found to be compatible with unity within experimental uncertainties. The results are compared to theoretical calculations that include cold-nuclear-matter effects and to transport model calculations incorporating the interactions of charm quarks with an expanding deconfined medium.