Fine-Tuning BERT On Coarse-Grained Labels: Exploring Hidden States For Fine-Grained Classification

Abstract

In recent years, pre-trained language models such as BERT (Bidirectional Encoder Representations from Transformers) have demonstrated exceptional performance across various natural language processing tasks. However, its effectiveness of encoding and capturing fine-grained distinctions within the hidden latent space during fine-tuning on coarse-grained labels remains relatively unexplored. To investigate this, we performed two distinct tasks: clustering and few-shot classification on fine-grained labels. The representations extracted from BERT’s hidden layers are utilized as input for these tasks. In the few-shot classification task, we demonstrate that the BERT model encodes valuable information about fine-grained labels during its fine-tuning on coarse-grained labels, allowing the few-shot classifier to classify fine-grained classes accurately even with a limited number of data samples. Additionally, in the clustering analysis, a thorough examination of the hidden layers is conducted to identify clusters that align with fine-grained label distinctions. The identification of such patterns further proves that the BERT model indeed encodes fine-grained label information within its hidden layers even when fine-tuned on coarse-grained labels. The findings contribute to a deeper understanding of the capabilities of the BERT model and provide valuable insights into harnessing its hidden latent space for fine-grained classification tasks.