Two-particle angular correlations between trigger particles in the forward pseudorapidity range ($2.5 <~ |\eta| <~ 4.0$) and associated particles in the central range ($|\eta| <~ 1.0$) are measured with the ALICE detector in p-Pb collisions at a nucleon-nucleon centre-of-mass energy of 5.02 TeV. The trigger particles are reconstructed using the muon spectrometer, and the associated particles by the central barrel tracking detectors. In high-multiplicity events, the double-ridge structure, previously discovered in two-particle angular correlations at midrapidity, is found to persist to the pseudorapidity ranges studied in this Letter. The second-order Fourier coefficients for muons in high-multiplicity events are extracted after jet-like correlations from low-multiplicity events have been subtracted. The coefficients are found to have a similar transverse momentum ($p_{\rm T}$) dependence in p-going (p-Pb) and Pb-going (Pb-p) configurations, with the Pb-going coefficients larger by about $16\pm6$%, rather independent of $p_{\rm T}$ within the uncertainties of the measurement. The data are compared with calculations using the AMPT model, which predicts a different $p_{\rm T}$ and $\eta$ dependence than observed in the data. The results are sensitive to the parent particle $v_2$ and composition of reconstructed muon tracks, where the contribution from heavy flavour decays are expected to dominate at $p_{\rm T}>2$ GeV/$c$.