In this paper, we present a multispectral image (MSI) compression method using a lossless and lossy coding scheme, which focuses on the seamless coding of the RGB bit stream to enhance the usability of the MSI. The proposed method divides the MSI data into two components: RGB and residual. The RGB component is extracted from the MSI by using the XYZ color matching functions, a color conversion matrix, and a gamma curve. The original MSI is estimated by an RGB data encoder and the difference between the original and the estimated MSI, which is referred to as the residual component in this paper. Next, the RGB and residual components are encoded by using JPEG2000, and progressive decoding is achieved from the losslessly encoded code stream. Experimental results show that a high-quality RGB image can be obtained at a low bit rate with primary encoding of the RGB component. In addition, by using the proposed method, the quality of a spectrum can be improved by decoding the residual data, and the quality is comparable to that obtained by using JPEG2000. The lossless compression ratio obtained by using this method is also similar to that obtained by using JPEG2000 with the integer Karhunen–Loeve transform.