Spatial-Spectral Deep Transfer Learning Network for Few-Shot Hyperspectral Image Classification

Abstract

Background and Objectives: In recent decades, classification of hyperspectral images (HSIs) has made significant progress with the development of deep learning (DL). However, learning with a few samples remains a difficulty due to the high cost of annotating samples and potential errors in manual evaluation. Most DL-related methods require a sufficient number of annotated samples to learn the intricate spatial-spectral structure and spectrum of HSI data, making them unsuitable for practical situations with limited samples. To overcome the difficulty in learning with a few samples, we propose a deep transfer contrastive learning approach known as Spatial–Spectral Deep Transfer Learning Network (SSDTLN) for HSI classification. The proposed approach utilizes a spectral augmentation process to increase the diversity of pairs of samples with diverse characteristics. In addition, a spatial feature extraction component is incorporated to leverage complementary information by supporting the spatial component with pretrained representations from the source domain and the spectral residual component, focusing on precise spectral feature extraction. Moreover, contrastive learning can increase the discriminative ability of the representations with the goal of improving the ability to deal with tasks concerning high similarities between classes and large variances within classes with few samples.

Methodology: The proposed approach involved a spatial-spectral few-shot deep transfer learning approach for the classification of hyperspectral images, which was broken down into three broad phases: spatial feature extraction via a CNN-based spatial network pre-trained on the ImageNet dataset; spectral feature extraction via a residual convolutional network coupled with spectral enhancement techniques such as random spectral shift and Gaussian noise injection; spatial-spectral feature fusion via convolutional layers; and a spatial attention module (SAM). Hyperspectral image patches were then created and arranged in a C-way K-shot episode, where the labeled images comprised the support set, while the remaining comprised the query set. A mapping layer was applied for projecting the high-dimensional hyperspectral image data into a three-channel format, compatible with the spatial network. Finally, the classification was conducted via a nearest-neighbor classifier, while the optimization of the approach was conducted via a joint loss function that combined cross-entropy loss and a supervised contrastive loss.

Main Results: The quantitative and qualitative analysis of the proposed and other state-of-the-art techniques like SVM, SSRN, DFSL, S3Net, HTLN, DCLN, and CPPM are validated on four benchmark hyperspectral image (HSI) datasets known as Pavia University (PU) dataset, Salinas (SA) dataset, Indian Pines (IP) dataset, and LongKou (LK) dataset. Experimental results indicate that the proposed model outperformes other techniques in terms of classification accuracy, boundary identification, and generalization ability, using minimal samples, while also having acceptable complexity.   

Conclusions: This work successfully addresses its goal of designing a strong few-shot classification framework for hyperspectral images (HSIs) leveraging deep transfer learning and contrastive learning methods. The proposed model effectively boosts feature learning by leveraging the strengths of ImageNet-pretrained spatial networks, spectral residual learning, and spectral data augmentation methods. Additionally, the Spatial Attention Module (SAM) helps better capture the context at multiple scales, especially in boundary regions. Finally, the hybrid loss function further promotes inter-class distinction and decreases intra-class variations. Experiments conducted on four popular HSI datasets validate the superior accuracy and generalization performance of the proposed approach in few-shot learning scenarios. Even though the approach is effective, there are some issues related to computational complexity and adaptability for low-resolution and cross-sensor learning that show promising research directions.

Practical Application: The findings of this study can be effectively used in real-world applications of hyperspectral image analysis when there is a lack of annotated data, including precision farming, environmental studies, urban planning, mineral exploration, and disaster response. The developed few-shot learning framework can be effectively used for land cover mapping with only a few training samples, thereby reducing the cost of manual annotation of data. The framework's noise-robust properties make it useful for real-world applications, including UAV- or satellite-based remote sensing systems.