Mice communicate using ultrasonic vocalizations (USVs) that vary according to parameters such as sex, genetic background, and environmental stimuli. The study of USVs production provides useful models of the underlying neurobiology mechanisms of human speech and therefore many methods exist to detect USVs in mice recordings. In order to achieve a temporal analysis of these vocalizations, one must first group them into categories. The grouping of USVs is challenging due to the high volume of vocalizations present even in small recordings, which requires sophisticated analysis techniques. However, most existing tools can only recognize a predefined number of categories and do not offer temporal analysis capabilities, which limits their effectiveness in comprehensive USV analysis. In this work, we used the open-source software Analysis of Mouse VOcal Communication (AMVOC) for USVs detection and proposed an unsupervised learning approach based on features extracted from a Convolutional Autoencoder (CAE). For the evaluation of the CAE approach, we built a benchmark dataset with the help of domain experts. We utilize USVs transition matrices to propose three metrics that quantify differences in the temporal structure between different recordings. Transition matrices are tools that map the likelihood of transitions between different USV categories over time. We evaluated these metrics using a dataset consisting of mice that carry a FoxP2 mutation, a gene involved in speech function. The effects of this mutation on mice are thoroughly studied and provide a solid ground for testing our method. The proposed approach allows researchers to perform batch comparisons of the temporal structure of recordings, enabling them to extract insights and identify differences in syntax composition. Using these insights, researchers can then guide more detailed analyses by emphasizing specific areas of interest or anomalies.