In our work we calculate features on top of the HuBERT model by applying TDA methods to the sets of embeddings of different layers, and to the attention heads. One of the proposed approaches is described below, using head #4 from layer #2 for illustration purposes.
The attention matrix can be associated with a directed weighted graph. We obtain the adjacency maxtrix of an undirected graph from it as A' = 1 - max(Aattn , ATattn ).
Animation on this page transits over filtration sequence of its subgraphs increasing threshold of filtration. Heatmap on the bottom left represents symmetrized matrix filtered by the threshold, while on the top right the corresponding filtered graph is given. Bottom right quarter depicts the process of building the barcode.
Each bar represents the "life" of a connected component in the graph from "birth" (when it first appears) till "death" (when it's been connectend to another connected component). We calculate the feature Hm, sym0 as the mean length of all bars from the barcode for the input audio.
Designed in Notepad and hosted by Github. (C) TopoHuBERT team, 2023.