We propose a theoretical analysis method for the denoising capability of Hadamard transform spectrometer (HTS). We establish the equivalent relationship between the HTS’s signal-to-noise ratio (SNR) and the coding matrix generalized Rayleigh quotient. Then, based on noise source during spectral detection, we analyze the changes of SNR with different types of spectrometers when detector noise or photon noise is dominant. The theory shows that when detector noise is dominant, the SNR of HTS with H-matrix coding is about 3.01 dB higher than S-matrix coding, and the SNR of HTS with S-matrix coding is about 10 log₁₀ ( n + 1 / 2 ) dB (n is the coding matrix order) higher than the slit-based spectrometer. When photon noise is dominant, the SNR of slit-based spectrometer and the SNR of HTS with H-matrix coding are better than S-matrix coding by about 1.51 dB. The theory is testified by computer simulations and a self-built HTS system.