We present a measurement of azimuthal correlations between inclusive J/$\psi$ and charged hadrons in p-Pb collisions recorded with the ALICE detector at the CERN LHC. The J/$\psi$ are reconstructed at forward (2.03 $<~$ y $<~$ 3.53) and backward ($-$4.46 $<~$ y $<~$ $-$2.96) rapidity via their $\mu^+\mu^-$ decay channel, while the charged hadrons are reconstructed at mid-rapidity ($|\eta|$ $<~$ 1.8). The correlations are expressed in terms of associated charged-hadron yields per J/$\psi$ trigger. A rapidity gap of at least 1.5 units between the trigger J/$\psi$ and the associated charged hadrons is required. Possible correlations due to collective effects are assessed by subtracting the associated per-trigger yields in the low-multiplicity collisions from those in the high-multiplicity collisions. After the subtraction, we observe a strong indication of remaining symmetric structures at $\Delta\varphi$ $\approx$ 0 and $\Delta\varphi$ $\approx$ $\pi$, similar to those previously found in two-particle correlations at middle and forward rapidity. The corresponding second-order Fourier coefficient ($v_2$) in the transverse momentum interval between 3 and 6 GeV/$c$ is found to be positive with a significance of about 5$\sigma$. The obtained results are similar to the J/$\psi$ $v_2$ coefficients measured in Pb-Pb collisions at $\sqrt{s_{\rm NN}}$ = 5.02 TeV, suggesting a common mechanism at the origin of the J/$\psi$ $v_2$.