1.

Introduction

With hundreds of narrow continuous spectral bands, hyperspectral images (HSIs)¹ better represent the semantic information of remotely sensed features.^2,3 Their rich spectral features provide a powerful tool for achieving accurate pixel-level classification.^4,5 HSI classification is widely used in precision agriculture,^6–8 mineral surveying,⁹ anomaly detection,¹⁰ and land cover mapping.¹¹ Compared with traditional visual images, HSIs have two unique features.¹² The first is that the heterogeneity of matter leads to the high-dimensional and nonlinear spectral characteristics of hyperspectral data. Second, there is a high correlation between adjacent hyperspectral spectra, and spatial correlation can supplement spectral information to provide a more accurate interpretation of surface cartography.

Classification of hyperspectral remote sensing images is fundamental to HSI processing and applications. Its ultimate aim is to assign a unique identity to each image element. Analyzing and processing the spectral and spatial characteristics of the various types of land features in the HSIs classify each image element into a category corresponding to the actual land features, thus enabling the classification of land features. Conventional methods generally use band selection and feature extraction for dimensionality reduction, compressing the original spectral image elements into a low-dimensional space, such as principal component analysis,¹³ support vector machines (SVM),¹⁴ and random forests.¹⁵ The properties of HSIs constrain these methods, and their classification results could be better. HSIs possess both spectral properties and spatial dependence, which implies a joint representation of spectral and spatial features. Therefore some researchers have proposed using a spatial–spectral feature extractor to extract HSI features. For example, spatial information was combined with multinomial logistic regression for HSI feature extraction.¹⁶ Li et al.¹⁷ proposed an edge-preserving filtering-based framework for spectral–spatial classification, significantly improving the classification accuracy of SVM with fewer computer resources. However, the above models all consist of shallow structures, such network models contain fewer hidden layers but require a more significant number of neurons, so shallow network structures do not provide a better description of HSIs.

With the development of artificial intelligence, deep-learning-based methods are widely used in remote sensing classification. For example, the spatial update super voxel stacking autoencoder method is an improved depth network. This method uses the spatial context of similar spectra within consecutive pixels for effective HSI classification.¹⁸ The convolution neural network (CNN) has a unique advantage in processing high-dimensional HSI data among these deep-learning models. For example, Chen et al.¹⁹ designed a 3D convolutional neural network capable of efficiently extracting spectral and spatial features with good classification performance. Li et al.²⁰ proposed a deep-learning framework for fully convolutional neural networks, applying deconvolution to HSI for the first time. Zhong et al.²¹ proposed a deep residual network for HSI classification, which used ResNets and CNNs of different depths and widths to investigate the effect of deep-learning model size on HSI classification accuracy. Kanthi et al.²² proposed a new 3D depth feature extraction CNN model for HSI classification using spectral and spatial information, which divides the HSI data into 3D patches and feeds them into the proposed model for depth feature extraction.

However, CNN’s reliance on a geometrically fixed structure of convolutional kernels hinders long-range dependencies between features from nonlocal locations. Transformers can effectively alleviate the problem of small perceptual range and low description efficiency caused by geometrically structured convolutional filter banks like CNN. Originally proposed as a sequence-to-sequence (seq2seq) model for machine translation, the transformer has become a mainstream model in natural language processing (NLP). This simple and efficient architecture also performs well in HSI classification. He et al.²³ applied for the first time a spatial transformation network to obtain the optimal input to a CNN for HSI classification. Zhao et al.²⁴ proposed a convolutional transform network for the fusion of spectral information and spatial location of HSIs using central position coding. Hong et al.²⁵ proposed the spectral–spatial transformer to capture the relationship between spectral bands along the spectral dimension. Meanwhile, spectral-spatial transformer network (SSTN) designed a spectral–spatial transformer network consisting of a spectral association module and a spatial attention module, using a new factorized architecture search framework to determine the hierarchical operations and block order of SSTN.²⁶

The bidirectional encoder representations from transformers (BERT) model²⁷ is based on the transformer taking its encoder part to obtain a bidirectional encoder representation model, a self-coding language model. BERT is structurally more superficial than the transformer in that it only uses the encoder part of the transformer. The BERT model was proposed in October 2018 by the Google AI Institute, and the model was initially popular in NLP. Because it is built on the transformer, BERT has powerful linguistic representation and feature extraction capabilities. However, BERT models consume substantial hardware resources so that reproducibility could be better and model convergence could be faster. RoBERTa, proposed by Liu et al.,²⁸ the model does not change the network structure of BERT but only modifies some pretraining methods, including dynamic masking and discarding the next sentence predict task. Chen et al. proposed ALBERT, which argued that the model parameters of BERT were too large and too resource-intensive. They proposed using word vector factorisation, cross-layer parameter sharing or sentence order prediction to reduce the model size and thus improve the training speed.²⁹ Cui et al.³⁰ proposed MacBERT, the model that proposes not using a mask, replacing the mask-tagged position with another similar word, and then allowing the model to correct errors to achieve better results automatically. Related improved models are MASS,³¹ UNILM,³² and SpanBert,³³ all of which have achieved good results.

He et al. argued that neurolinguistic models are more similar to HSIs in some aspects. They assembled multiple self-attention layers into transformer units. They converted the image elements of HSIs into sequences as input data to be applied to HSI classification tasks with satisfactory results. Their proposed HSI-BERT converts spectral image sets into sequences for characterization. This approach not only does not disrupt the inherent continuous spectral distribution of HSIs but also supports more flexible and dynamic input regions and is more easily generalized.³⁴ The BERT model captures richer nonlocal image element information. However, the complex distribution of spectral space in HSIs results in low efficiency of BERT model feature extraction and inconspicuous feature description in high-dimensional spectral space. Each pixel of HSI is marked by BERT sequence mode, and a multihead self-attention mechanism extracts nonlocal spatial information of HSI. Still, the representation of local spatial features of HSIs needs to be improved. Moreover, the multihead self-attention mechanism only considers the feature characterization of the spatial pixel rotation sequence. It needs to utilize the rich spectral features unique to HSIs more effectively. In order to solve the above problems based on the BERT model, this paper proposes a spatially augmented guided sequential BERT network for HSI classification studies called SAS-BERT.

The main contributions of this paper are as follows.

The remainder of this paper is organized as follows. Section 2 presents a relevant introduction to the BERT model. Section 3 presents the scheme of the proposed SAS-BERT network and its components. Section 4 presents the experimental results and analytical findings. In Sec. 5, conclusions are drawn.

2.

Related Introduction

2.1.

BERT Network Structure

The BERT model has been fine-tuned with good results on some downstream NLP tasks. The general framework of the BERT model is shown in Fig. 1, which is mainly based on the transformer’s encoder module, which will be referred to as TEn, as shown in Fig. 2, and the BERT model is obtained by stacking the TEn.

Fig. 1

Diagram of the general framework of the BERT model. The BERT model consists of an embedding layer, an encoder layer, and an FCL. The encoder layer is obtained by stacking multiple TEn.

Fig. 2

Structural diagram of TEn.

The BERT model consists of two main modules: embedding and encoder layers. The embedding layer includes token embedding, position embedding, and segment embedding, and segment embedding is not required for HSI classification. The token embedding implements a vector representation of the image element itself, and the position embedding learns the position properties of the image element.

The embedding layer is followed by the encoder layer, which contains the multiheaded self-attention (MHSA) module (Fig. 3). This mechanism improves the model’s ability to focus on different locations and multiple focus regions simultaneously. The self-attentive mechanism requires the generation of three-word vectors $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ to calculate the attention of each image element, which is obtained by multiplying the image element with three training learned weight matrices. However, in multiheaded attention, multiple learned weight matrices of $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ are randomly initialized and independently map the input vectors to different subspaces, thus enriching the feature representation of the information. Different heads of the MHSA mechanism obtain different attentions. For the input vector, if multiple attention is used and the number of heads used is $p$, the input vector is divided into $p$ independent vectors, each using self-attention to calculate the attention weight. When completed, it will be merged. Therefore, it is a parallel mechanism within a submodule. All heads work independently and in parallel. Internal to the multihead self-attention mechanism is a scaled dot product attention (Fig. 4), calculated as follows:

Eq. (1)

$$\text{attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{softmax}\left(\frac{\mathbf{Q}{\mathbf{K}}^{\mathrm{T}}}{\sqrt{d_k}}\right)\mathbf{V},$$

where $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ denote query, key, and value, respectively, and $d_k$ denotes the dimension of the vector key. After the input image is converted into a sequence, all the pixel vectors on the sequence are multiplied by three randomly initialized matrices, and three vectors, $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$, are obtained. As shown in Fig. 4, the attention mechanism takes $\mathbf{Q}$ as the target pixel that needs to predict the label, and the number of pixel vectors generates the number of $\mathbf{Q}$ vectors, so each pixel in the sequence can be the target pixel. $\mathbf{K}$, as the $\mathbf{Q}$ context pixel, matches the similarity with $\mathbf{Q}$. The similarity between the query and each key is used as the weight, and then the value of the target pixel is weighted and fused with the value of the individual pixels of the context as the output of attention.

Fig. 3

Schematic diagram of the MHSA mechanism.

Fig. 4

Schematic diagram of the scaled dot product attention mechanism.

The MHSA contains $p$ heads $(H_1,H_2,\dots ,H_p)$:

Eq. (2)

$$\mathrm{MHSA}(X)=\text{concat}(H_1,H_2,\dots ,H_p)W^O,$$

where $W^O$ is the matrix of learned parameters, and $h_i$ is the result of the attention structure of $i$. The $h_i$ of self-attention structure can be described as follows:

Eq. (3)

$$h_i=\text{attention}(X{W}_i^Q,X{W}_i^K,X{W}_i^V),$$

where $X$ is the target image element, and the learning parameter matrices $W^Q,W^K,W^V\in R^{d\times d/\mathrm{p}}$ are used for the affine projection.

3.

Modified BERT Network Structure

In this paper, the BERT network is modified by adding a spatial augmentation module and a spectral correlation module. The modified network is named the spatially augmented guided sequence-based BERT network (SAS-BERT). The spatial augmentation module guides the sequence patterns and obtains local spatial information of the HSI. The spectral correlation module establishes correlations between spectral kernels and spatial locations, enhancing the description of the spectral information of the image. A schematic diagram of the SAS-BERT framework is shown in Fig. 5. Random in Fig. 5 refers to random seeds, it means that the input samples for the 10 repeated experiments were randomly selected. Assume the original HSI dataset $I\in R^{H\times W\times b}$, where $H\times W$ is the spatial size and $b$ is the spectral dimensions. When fed into the network, it forms a one-hot category vector $Y=\{y_1,y_2,\cdots ,y_C\}\in R^{1\times 1\times C}$, where $C$ is the number of land cover categories. Taking the Pavia dataset as an example, the spectral dimensions are 103, and the land cover category is 9, which is introduced explicitly in the experimental dataset part of the fourth part. The spectral size is set to $11\times 11$. SAS-BERT consists of the spatially augmented learning module and the spe-BERT module, which contains the spectral correlation module and the multiheaded self-attentive module.

Fig. 5

Schematic diagram of the SAS-BERT framework.

3.1.

Spatially Augmented Learning Module

The MHSA mechanism of the BERT model captures rich nonlocal spatial contextual information but ignores the representation of local spatial information on HSIs. In this paper, we guide HSI’s local spatial feature extraction through the spatial augmented learning module to obtain an optimal input and then send it to the BERT module for further feature extraction and classification tasks.³⁵ The diagram of the spatially augmented learning module is shown in Fig. 6, which contains three parts that are described below.

Fig. 6

Diagram of the spatially augmented learning module.

The first section is the localization network ($F_{\mathrm{loc}}$). The localization network in Fig. 6 completes the extraction of local spatial information. The affine transformation of Eq. (4) through several hidden layers (convolution, pooling, full connection, etc.) map the coordinate point relationship between the input and output feature maps. The parameters in affine transformation are used to rotate, translate, and scale the original input image, and then the optimal input image for BERT model. The ultimate goal of using localization network is to obtain a good classification performance (as shown in red in Fig. 6). It learns the affine transformation matrix $A_{\theta }$. After several convolutions or entire connection operations, the network will input image $I$, and then a regression layer output affine transformation matrix $A_{\theta }\in R^{2\times 3}$, $A_{\theta }$ is a learnable parameter. Four transformations are carried out by changing the parameters of $A_{\theta }$: translation, scaling, rotation, and clipping. These parameters map the coordinate point relationship between the input and output feature maps. The coordinate points of the input feature map are obtained according to $A_{\theta }$:

Eq. (4)

$$F_{\mathrm{loc}}(I)=A_{\theta }=\left[\begin{array}{ccc}a& b& c\\ d& e& f\end{array}\right],$$

where $I$ is the input image; $A_{\theta }$ is an affine transformation matrix; $F_{\mathrm{loc}}(·)$ denotes a function formed by a fully connected and other layers; $a,b,c,d,e,f$ denotes different transformations by changing the values of these six parameters. For example, $c,e$ for translation, $a,e$ for scaling, $a,b,d,e$ for rotation, and $b,d$ for cropping.

The second section is the grid generator, as shown in the yellow section of Fig. 6. The position mapping relationship is based on the affine transformation matrix $A_{\theta }$. The affine transformation matrix $A_{\theta }$ and the lattice points of the output feature map are used to find the coordinates of each pixel point of the output feature map corresponding to the input image:

Eq. (5)

$$S_{\theta }(G_{(i,j)})=\left(\begin{array}{c}{x}_i^I\\ y_j^I\end{array}\right)=A_{\theta }\left(\begin{array}{c}{x}_i^O\\ y_j^O\\ 1\end{array}\right),$$

where $S_{\theta }(·)$ denotes the function formed by parametric grid sampling; $G_{(i,j)}=(x_i^O,y_j^O)$ denotes the individual grid coordinates of the output feature map; and $(x_i^I,y_j^I)$ denotes the corresponding grid coordinate points of the output feature map on the input image.

The third section is the sampler, as shown in the blue section of Fig. 6. That is, the pixel value of the output image is calculated using $(x_i^I,y_j^I)$ corresponding to the pixel value of the input image as the pixel value of the output image at $(i,j)$. The interpolation algorithm calculates the pixel values of the output image based on the location mapping. Because the grid coordinate point corresponding to the input image in the second part may be a fractional number, the interpolation algorithm is used to correct the pixel values at that location. Bilinear interpolation is used here for interpolation:

Eq. (6)

$$V_{i,j}^l=\sum _0^H\sum _0^W{I}_{n,m}^l·\max (\mathrm{0,1}-|x_i^I-m|)\max (\mathrm{0,1}-|y_j^I-n|),$$

where $V_{i,j}^l$ is the pixel value of the output feature map at channel $l$ and coordinate $(x_i^O,y_j^O)$, $I_{n,m}^l$ is the pixel value of the input feature map at channel $l$, coordinate $(n,m)$ ($(n,m)$ traverses all coordinate points of the input feature map). The smaller the values of $|x_i^I-m|$ and $|y_j^I-n|$ are, the closer the distance between $(x_i,y_j)$ and $(n,m)$ is. The larger the values of $1-|x_i^I-m|$ and $1-|y_j^I-n|$ are, the larger the value of max is, and the greater the weight will be. $(x_i^I,y_j^I)\to (n,m)$ is in a lattice point, and finally, four corresponding weights are obtained, summed, and output $V_{i,j}^l$.

The spatial augmentation module can be trained end-to-end with the entire convolutional network. It is inserted directly into the convolutional network and runs with good classification performance.

3.2.

spe-BERT Module

The spe-BERT module contains an MHSA module and a spectral correlation module. One of the essential concepts in the original BERT model is the MHSA mechanism, as shown in Fig. 4, which can link information at different positions on the input image element to obtain nonlocal long-range dependencies. However, the MHSA mechanism only considers the spatial key to the image element’s feature representation, ignoring the spectral band’s characteristics for HSI interpretation. Spectral features are the key features to distinguish ground objects in HSI. So this paper adds a spectral correlation module to the BERT module to aggregate HSI’s spectral and spatial features for better classification performance.

The schematic diagram of the spectral combined module is shown in Fig. 7. It comprises a multibranch network structure that uses two-dimensional convolution to integrate and invert the spatial information of the image element sequences along the spectral dimension. The sequenced feature cube $X$ feed into the spectral correlation module and its spectral correlation kernel is calculated as follows:

Eq. (7)

$$M_1=\sigma (\zeta (X;W_{M_1}))\in R^{mn\times k},$$

Eq. (8)

$$\mathrm{Asso}=M_1^{\mathrm{T}}·X^{\mathrm{T}}={(X·M_1)}^{\mathrm{T}}\in R^{k\times l},$$

where $\zeta (·)$ denotes a convolution operation and generates a tensor $mn\times k$; $\sigma (·)$ is the softmax function, which assigns independent weights to each image element. $M_1$ is the generated mask, which integrates the spatial information in conjunction with the input features $X$ to obtain a feature map of the spectral kernel concerning the spatial location. The output of the spectral correlation kernel is as follows:

Eq. (9)

$$M_2=\sigma (\zeta (X;W_{M_2}))\in R^{mn\times k},$$

Eq. (10)

$${\mathrm{Spe}}_{\mathrm{Asso}}={\mathrm{Asso}}^{\mathrm{T}}·M_2^{\mathrm{T}}=(M_2·\mathrm{Asso}{)}^{\mathrm{T}}\in R^{m\times n\times l},$$

where $M_2$ is another generated mask and $\zeta (·;W_{M_1})$ shares the training parameters with $\zeta (·;W_{M_2})$.

Fig. 7

Schematic diagram of the spectral combined module.

The spectral correlation module establishes the correlation between spectral kernels and spatial locations, complementing the BERT model for HSIs with missing spectral information.

3.3.

SAS-BERT

In this paper, we have chosen the Pavia University dataset of dimensions $610\times 340\times 103$ as an example to illustrate the designed SAS-BERT model. The Pavia University dataset is a selection of hyperspectral data from images taken in 2003 of the Italian city of Pavia, containing nine feature classes. Figure 8 details the SAS-BERT algorithm flow. The entire network consists of a spatial augmentation module, a spe-BERT module, and a fully connected layer (FCL). The spe-BERT module, in turn, contains the MHSA mechanism module (a module of the BERT model itself) and the spectral correlation module.

Fig. 8

Detailed framework diagram of SAS-BERT.

SAS-BERT uses a cross-entropy loss function to minimise losses:

Eq. (11)

$$L_{\mathrm{ce}}=-\frac{1}{B}\sum _{i=1}^B\sum _{j=1}^C{y}_{i,j}\log (y_{i,j}^{\prime }),$$

where $y$ and $y^{\prime }$ denote the actual and predicted one-hot label vectors, respectively. $B$ is the number of samples in a batch, and $C$ is the number of categories. $y_{i,j}$ denotes the scalar of the $j$’th category for the $i$’th sample.

Algorithm 1 describes the detailed training process of SAS-BERT in detail.

Algorithm 1

SAS-BERT algorithm.

Input: Hyperspectral image cubes $I$, model depth $d$, attention to the number of heads $h$, training batch $b$, training epoch $e$.

Output: Classification evaluation results for $X_{\text{test}}$ and predicted classification charts.

1: Begin

2: Input hyperspectral image cubes $I$;

3: for$i=0$ to epoch $e$do

4:  for$j=0$ to batch $b$do

5:   Generate the affine transformation matrix $A_{\theta }$ using the localization network;

6:   The grid generator calculates the coordinates of each pixel point of the output feature map corresponding to the input image according to $A_{\theta }$;

7:   As shown in Eq. (6), calculate the pixel values of the output image using bilinear interpolation to obtain a spatially augmented input image;

8:   Spatially augmented images for sequence conversion;

9:   for$k=1$ to depth $d$do

10:    The sequenced image is fed into the spectral correlation module to obtain a spectral kernel relation to the spatial location of the feature map $Sp{e}_{\text{Asso}}$;

11:    Divide the last dimension of the feature map into $h$ attention heads.

12:    The attentional characteristics of each attentional head are obtained by Eq. (1);

13:    Concat up the attentional features of $h$ attentional heads;

14:    Combined with the spectral correlation of the relational feature map $Sp{e}_{\text{Asso}}$;

15:    $k++$;

16:   end for

17:   $\mathrm{j}++$;

18:  end for

19:  $i++$;

20: end for

21: Load the model and feed $X_{\text{test}}$ into the model prediction;

22: Obtaining classification evaluation results and predicted classification maps.

4.

Experimental Results

All experiments in this paper are implemented on an Ubuntu 18.04 system using a GeForce RTX 2080 GPU and TensorFlow with CUDA 9.0 and Python 3.6. All subsequent training and testing experiments were conducted based on this environment.

4.3.

Parameter Adjustment

In order to improve the efficiency of the SAS-BERT network, this paper sets the neighbourhood size to $11\times 11$, the learning rate to $3\times {10}^{-4}$, the batch size to 16, the training times to 200, the dropout rate to 0.3, and samples 5%, 1% and 2% for IN, PU, and HOU, respectively, for training. This paper experiments with several factors affecting SAS-BERT’s representation of HSIs.

4.3.1.

TEn layers

Evaluating the number of TEn layers at different depths in BERT: the effects of TEn layers on HSIs classification were evaluated. The effect of TEn layer depth on classification accuracy verifies for the BERT network for the three datasets above. Letting the number of attention heads $h=10$, the classification results for different TEn layer depths on the IN, PU, and HOU datasets show in Fig. 9.

Fig. 9

Line graphs of classification accuracy for different TEn layer depths on the (a) IN, (b) PU, and (c) HOU datasets.

As shown in Fig. 9, the highest evaluation results for HSI classification accuracy were obtained on the IN and HOU datasets when the TEn layer depth was 5. When the TEn layer depth is 2, the highest evaluation result is received on the PU dataset. Therefore, the TEn layer depths selected in this paper are 5, 2, and 5 for the IN, PU, and HOU datasets, respectively.

4.3.2.

Number of self-attended heads

Evaluation of the number of heads for the MHSA mechanism: different numbers in the MHSA mechanism in the BERT model significantly impact classification performance, and this paper evaluates the effect of other numbers of attention heads on the classification results. Let the TEn layer depths $d$ of the IN, PU, and HOU datasets be 5, 2, and 5, respectively. The classification accuracies of different numbers of attention heads on the IN, PU, and HOU datasets show in Fig. 10.

Fig. 10

Line graphs of classification accuracy for different numbers of attention heads on the (a) IN, (b) PU, and (c) HOU datasets.

As shown in Fig. 10, different attention heads are set for the proposed model. The highest evaluation results were obtained when the number of attention heads was 10 on the IN and PU datasets. When the number of attention heads is 4, the highest evaluation result is received on the HOU dataset. Therefore, the number of attention heads selected in this paper for the IN, PU, and HOU datasets are 10, 10, and 4, respectively.

4.5.

Comparison with Various Algorithms

The SAS-BERT algorithm proposed in the paper is experimentally compared with other deep-learning methods. There are two types of deep-learning methods for comparison, CNN-based algorithms: CNN,¹⁹ SSRN,³⁶ HybridSN,³⁷ and ${\mathrm{LS}}^2\mathrm{CM}$,³⁸ and transformer-based methods: CTN,²⁴ SSTN,²⁶ and HSI-BERT.³⁴

In these experiments, the SAS-BERT input hyperspectral cube has a spatial size of $11\times 11$. The Adam optimizer optimizes 200 epochs. The learning rate set to $3\times {10}^{-4}$, the batch size to 16, and the dropout rate to 0.3. Various other settings for the comparison algorithms follow the original paper. The classification accuracy for the comparison experiments is shown in Table 4.

Table 4

Comparison of classification accuracy of algorithms on IN, PU, and HOU datasets.

Methods	IN			PU			HOU
	OA (%)	AA (%)	k×100	OA (%)	AA (%)	k×100	OA (%)	AA (%)	k×100
CNN	85.24 ± 1.14	87.54 ± 1.53	83.16 ± 1.31	78.63 ± 2.37	78.53 ± 1.63	70.72 ± 3.91	74.23 ± 2.91	79.90 ± 1.46	72.13 ± 3.15
SSRN	84.64 ± 9.88	80.16 ± 4.98	82.67 ± 10.81	94.89 ± 3.33	91.38 ± 6.41	93.23 ± 4.41	84.89 ± 4.94	86.52 ± 4.80	83.68 ± 5.32
HybridSN	86.80 ± 0.67	90.73 ± 0.56	84.89 ± 0.75	94.76 ± 1.56	93.15 ± 2.07	93.01 ± 2.11	88.78 ± 0.90	89.80 ± 0.68	87.86 ± 0.97
${\mathrm{LS}}^2\mathrm{CM}$	92.21 ± 2.01	91.31 ± 4.34	91.12 ± 2.30	96.04 ± 2.13	94.61 ± 2.47	94.76 ± 2.80	86.87 ± 4.09	89.50 ± 2.84	85.80 ± 4.42
CTN	93.57 ± 1.09	88.07 ± 3.44	92.66 ± 1.24	95.08 ± 0.63	93.36 ± 0.87	93.47 ± 0.83	80.36 ± 2.33	81.66 ± 2.30	78.73 ± 2.53
SSTN	94.54 ± 0.60	77.53 ± 1.37	93.77 ± 0.69	96.37 ± 0.84	91.07 ± 2.79	95.17 ± 1.13	85.45 ± 1.17	82.96 ± 1.06	84.24 ± 1.27
HSI-BERT	93.36 ± 0.88	85.01 ± 1.90	92.42 ± 1.02	96.38 ± 0.58	93.41 ± 0.90	95.19 ± 0.77	89.59 ± 2.10	87.55 ± 3.17	88.74 ± 2.28
SAS-BERT	95.49 ± 0.51	91.14 ± 1.05	94.86 ± 0.58	97.41 ± 0.32	95.63 ± 0.90	96.57 ± 0.43	91.23 ± 0.69	91.48 ± 0.54	90.52 ± 0.75

Note: Bold values represent the optimal results under the same evaluation index.

Table 4 shows the classification results of the various classification methods. It can be analyzed that SAS-BERT outperforms the compared CNN and transformer methods on the IN, PU, and HOU datasets, significantly improving the HSI-BERT method in particular. Overall, the SAS-BERT classification method performs better than the current state-of-the-art methods.

Figures 11Fig. 12–13 show the classification mapping of the comparison experiments on the Indian, Pavia, and Houston datasets. From these figures, the SAS-BERT algorithm shows more accurate classification mapping results with smoother and clearer edges, and SAS-BERT has better classification results.

Fig. 11

Classification maps on the Indian dataset: (a) a reference map of the real ground category on the Indian dataset; a classification map of (b) CNN; (c) SSRN; (d) HybridSN; (e) ${\mathrm{LS}}^2\mathrm{CM}$; (f) CTN; (g) SSTN; (h) HSI-BERT; (i) SAS-BERT; and (j) each ground cover category marker colour for the Indian dataset.

Fig. 12

Classification maps on the Pavia dataset: (a) a reference map of the real ground category on the Pavia dataset; a classification map of (b) CNN; (c) SSRN; (d) HybridSN; (e) ${\mathrm{LS}}^2\mathrm{CM}$; (f) CTN; (g) SSTN; (h) HSI-BERT; (i) SAS- BERT; and (j) each ground cover category marker colour for the Pavia dataset.

Fig. 13

Classification maps on the Houston dataset: (a) a reference map of the real ground category on the Houston dataset; a classification map of (b) CNN; (c) SSRN; (d) HybridSN; (e) ${\mathrm{LS}}^2\mathrm{CM}$; (f) CTN; (g) SSTN; (h) HSI-BERT; (i) SAS-BERT; and (j) each ground cover category marker colour for the Houston dataset.

5.

Conclusion

This paper proposes a SAS-BERT method for HSI classification based on the BERT model for feature extraction of spatial and spectral information. The method improves the performance of BERT model-based classification by aggregating augmented spatial features and spectral features to represent HSI features. It improves the representative characteristics of the local spatial information using a spatial augmentation module. The module transforms the input image so that distinct representational features characterise this input. It allows for better performance in classification tasks, which helps to minimize the overall cost of the network during training. In addition, it uses the spectral properties of HSIs so that they fully reflect the internal physical structure of matter. And correlation with spatial location is established, which significantly improves the interpretation of HSIs through feature description. Results from experiments on three widely used datasets show that the SAS-BERT model outperforms the current state-of-the-art CNN and transformer network classification models.