TopoHuBERT project

Topological properties of attention for speech processing

We apply Topological Data Analysis (TDA) methods to speech classification problems and to introspection of the pre-trained speech model. To this end, we introduce a number of topological and algebraic features, which are derived from the transformer's attention maps and embeddings. We empirically show that a simple linear classifier built on top of such features outperforms fine-tuning classification head. In particular, we achieve an improvement of up to 9% accuracy and 5% ERR on four common datasets with the HuBERT model. On the Crema-D dataset the proposed feature set establishes a new state-of-the-art performance by reaching the performance of accuracy equal to 80.155. Last but not least we show that topological features are capable of revealing the functional roles of speech transformer heads. For example, we find the heads, that are capable to distinguish in a precise way between emotions (sad/happy), sample source (natural/generated) or recognize one of two voices without any downstream fine-tuning. To do so, we introduce a ranking function, which separates topological representations of a single head. The results demonstrate that TDA is a promising research direction for speech analysis, specifically, for the tasks that require structural prediction.

This webpage provides suplementoral and additional material for our article "Topological properties of attention for speech processing" that was accepted to INTERSPEECH 2023. Due to the strict constraints on the the size of the submitions, in our paper we had to omit or mention too briefly some of the achieved results, but here we can present them in their enterietry without any fear for page count.