Angular correlations between unidentified charged trigger particles and various species of charged associated particles (unidentified particles, pions, kaons, protons and antiprotons) are measured by the ALICE detector in p-Pb collisions at a nucleon-nucleon centre-of-mass energy of 5.02 TeV in the transverse-momentum range $0.3 < p_{\rm T} < 4$ GeV/$c$. The correlations expressed as associated yield per trigger particle are obtained in the pseudorapidity range $|\eta_{\rm lab}|<0.8$. Fourier coefficients are extracted from the long-range correlations projected onto the azimuthal angle difference and studied as a function of $p_{\rm T}$ and in intervals of event multiplicity. In high-multiplicity events, the second-order coefficient for protons, $v_2^p$, is observed to be smaller than that for pions, $v_2^\pi$, up to about $p_{\rm T} = 2$ GeV/$c$. To reduce correlations due to jets, the per-trigger yield measured in low-multiplicity events is subtracted from that in high-multiplicity events. A two-ridge structure is obtained for all particle species. The Fourier decomposition of this structure shows that the second-order coefficients for pions and kaons are similar. The $v_2^p$ is found to be smaller at low $p_{\rm T}$ and larger at higher $p_{\rm T}$ than $v_2^pi$, with a crossing occurring at about 2 GeV. This is qualitatively similar to the elliptic-flow pattern observed in heavy-ion collisions. A mass ordering effect at low transverse momenta is consistent with expectations from hydrodynamic model calculations assuming a collectively expanding system.