The production of beauty hadrons was measured via semi-leptonic decays at mid-rapidity with the ALICE detector at the LHC in the transverse momentum interval $1<~p_{\rm T}<~8$ GeV/$c$ in minimum-bias p-Pb collisions at $\sqrt{s_{\rm NN}}=5.02$ TeV and in $1.3<~p_{\rm T}<~8$ GeV/$c$ in the 20% most central Pb-Pb collisions at $\sqrt{s_{\rm NN}}=2.76$ TeV. The pp reference spectra at $\sqrt{s}=5.02$ TeV and $\sqrt{s}=2.76$ TeV, needed for the calculation of the nuclear modification factors $R_{\rm pPb}$ and $R_{\rm PbPb}$, were obtained by a pQCD-driven scaling of the cross section of electrons from beauty-hadron decays measured at $\sqrt{s}=7$ TeV. The $R_{\rm PbPb}$ is about 0.7 with an uncertainty of about 30% in the interval $3<~p_{\rm T}<~6$ GeV/$c$ and 0.47 with an uncertainty of 25% in $6<~p_{\rm T}<~8$ GeV/$c$ in Pb-Pb collisions. Below $p_{\rm T}=3$ GeV/$c$, the $R_{\rm PbPb}$ values increase with decreasing transverse momentum with systematic uncertainties of 30-45%. The $R_{\rm pPb}$ is consistent with unity within systematic uncertainties of about 20% at high $p_{\rm T}$, increasing at low $p_{\rm T}$, and is well described by theoretical calculations that include cold nuclear matter effects in p-Pb collisions. The measured $R_{\rm pPb}$ and these calculations indicate that cold nuclear matter effects are small at high transverse momentum also in Pb-Pb collisions. Therefore, the observed reduction of $R_{\rm PbPb}$ below unity for high $p_{\rm T}$ can be ascribed to an effect of the hot and dense medium formed in Pb-Pb collisions.