Transverse momentum ($p_{\rm T}$) spectra of charged particles at mid-pseudorapidity in Xe-Xe collisions at $\sqrt{s_{\rm NN}}$ = 5.44 TeV measured with the ALICE apparatus at the Large Hadron Collider are reported. The kinematic range $0.15 <~ p_{\rm T} <~ 50$ GeV/$c$ and $|\eta| <~ 0.8$ is covered. Results are presented in nine classes of collision centrality in the 0-80% range. For comparison, a pp reference at the collision energy of $\sqrt{s}$ = 5.44 TeV is obtained by interpolating between existing \pp measurements at $\sqrt{s}$ = 5.02 and 7 TeV. The nuclear modification factors in central Xe-Xe collisions and Pb-Pb collisions at a similar center-of-mass energy of $\sqrt{s_{\rm NN}}$ = 5.02 TeV, and in addition at 2.76 TeV, at analogous ranges of charged particle multiplicity density $\left\langle\rm{d}N_{\rm ch}/\rm{d}\eta\right\rangle$ show a remarkable similarity at $p_{\rm T}> 10$ GeV/$c$. The centrality dependence of the ratio of the average transverse momentum $\left\langle p_{\rm{T}}\right\rangle$ in Xe-Xe collisions over Pb-Pb collision at $\sqrt{s}$ = 5.02 TeV is compared to hydrodynamical model calculations.