The production of charged pions, kaons and (anti)protons has been measured at mid-rapidity ($-0.5 <~ y <~ 0$) in p-Pb collisions at $\sqrt{s_{\rm NN}}=5.02$ TeV using the ALICE detector at the LHC. Exploiting particle identification capabilities at high transverse momentum ($p_{\rm T}$), the previously published $p_{\rm T}$ spectra have been extended to include measurements up to 20 GeV/$c$ for seven event multiplicity classes. The $p_{\rm T}$ spectra for pp collisions at $\sqrt{s}=7$ TeV, needed to interpolate a pp reference spectrum, have also been extended up to 20 GeV/$c$ to measure the nuclear modification factor ($R_{\rm pPb}$) in non-single diffractive p-Pb collisions. At intermediate transverse momentum ($2 <~ p_{\rm T} <~ 10$\,GeV/$c$) the proton-to-pion ratio increases with multiplicity in p-Pb collisions, a similar effect is not present in the kaon-to-pion ratio. The $p_{\rm T}$ dependent structure of such increase is qualitatively similar to those observed in pp and heavy-ion collisions. At high $p_{\rm T}$ ($>10$ GeV/$c$), the particle ratios are consistent with those reported for pp and Pb-Pb collisions at the LHC energies. At intermediate $p_{\rm T}$ the (anti)proton $R_{\rm pPb}$ shows a Cronin-like enhancement, while pions and kaons show little or no nuclear modification. At high $p_{\rm T}$ the charged pion, kaon and (anti)proton $R_{\rm pPb}$ are consistent with unity within statistical and systematic uncertainties.