Animal behavior analysis is a fundamental tool in various studies, ranging from basic neuroscience research to understanding the causes and treatments of diseases. It is widely applied not only in biological research but also across various industrial fields, including robotics. Recently, efforts have been made to accurately analyze animal behaviors using AI technology. However, there are still limitations for AI to intuitively recognize different behaviors like human observers can.

Traditional animal behavior research primarily involved filming animals with a single camera and analyzing low-dimensional data such as the time and frequency of specific movements. The analysis method provided AI with corresponding results for each piece of training data, akin to simply feeding AI with questions along with the answer key. While this method is straightforward, it requires time and labor-intensive human supervision to build the data. Observer bias is also a factor, as the analysis results can be distorted by the experimenter's subjective judgment.

To overcome those limitations, a joint research team led by Director C. Justin LEE of the Center for Cognition and Sociality within the Institute for Basic Science, and CHA Meeyoung, the Chief Investigator (CI) of the Data Science Group at the IBS Center for Mathematical and Computational Sciences (also a professor in the School of Computing at KAIST), has developed a new analytical tool called 'SUBTLE' (Spectrogram-UMAP-Based Temporal-Link Embedding). SUBTLE classifies and analyzes animal behavior through AI learning based on 3D movement information.

First, the research team recorded the movements of mice using multiple cameras, extracting the coordinates of nine key points such as the head, legs, and hips to obtain 3D action skeleton movement data over time. They then reduced this time-series data into two dimensions for embedding, a process that creates a collection of vectors corresponding to each piece of data, allowing complex data to be represented more concisely and meaningfully.

Next, the researchers clustered similar behavior states into subclusters and grouped these subclusters into superclusters representing standardized behavior patterns (repertoires), such as walking, standing, grooming, etc. During this process, they proposed a new metric called the Temporal Proximity Index (TPI) to evaluate behavior data clusters. This metric measures whether each cluster includes the same behavior state and effectively represents temporal movements, similar to how humans consider temporal information important when classifying behavior.

CI CHA Meeyoung stated, "The introduction of new evaluation metrics and benchmark data to aid in the automation of animal behavior classification is a result of the collaboration between neuroscience and data science. We expect this algorithm to be beneficial in various industries requiring behavior pattern recognition, including the robotics industry, which aims to mimic animal movements."

Director C. Justin LEE of the Center for Cognition and Sociality, who led this research commented, "We have developed an effective behavior analysis framework that minimizes human intervention while understanding complex animal behaviors by applying human behavior pattern recognition mechanisms. This framework has significant industrial applications and can also be used as a tool to gain deeper insights into the principles of behavior recognition in the brain."

Additionally, the research team transferred SUBTLE technology to Actnova, a company specializing in AI-based clinical and non-clinical behavior test analysis, in April last year. The team utilized Actnova's animal behavior analysis system, AVATAR3D, to obtain 3D movement data of animals for this research.

The research team has also made SUBTLE's code open-source, and a user-friendly graphical interface (GUI) to facilitate animal behavior analysis is available through the SUBTLE web service (http://subtle.ibs.re.kr/) for researchers who are not familiar with programming.

This research was published online on May 20th in the prestigious AI computer vision journal, the International Journal of Computer Vision (IJCV).

Figure 1. Proposing a new behavior embedding evaluation metric: TPI (Temporal Proximity Index)
(A) Displays the movement of a mouse's 3D action skeleton over time, with each color representing a standardized behavior repertoire (e.g., walking, standing).
(B) Once the behavior embedding space is created, examining the movement patterns over time can verify the quality of the embedding. A good embedding space has clusters that contain the same behavior states, showing efficient temporal movement, while a poor embedding space has clusters that contain different behavior states, resulting in inefficient temporal movement.
(C) The quality of temporal connectivity can be calculated by the total value of the product of transition probabilities between clusters and the distance between clusters (TPI). (Left) Frequent transitions to nearby clusters indicate good temporal connectivity. (Right) Few transitions to nearby clusters indicate poor temporal connectivity.
(D) Calculation method of the Temporal Proximity Index (TPI) for evaluating the temporal connectivity of the behavior embedding space.
(E) Workflow for unsupervised animal behavior analysis.

Figure 2. Schematic of the SUBTLE framework
(A) The process of obtaining and analyzing 3D coordinates of key points from the movement of a mouse.
1) On the left, it shows the process of extracting 3D raw coordinates of the mouse's movements using the AVATAR3D, while on the right, it shows the process of processing and analyzing the 3D coordinate data obtained from AVATAR3D.
2) Extract the 3D action skeleton using the avatar.
3) Extract kinematic features and wavelet spectrograms from the key point coordinates.
4) Perform nonlinear t-SNE and UMAP algorithms; the embedding using UMAP developed in this study is called SUBTLE.
(B) Results of nonlinear mapping
It shows the embedding results using t-SNE and UMAP with an increasing number of clusters (k). t-SNE exhibits a tangled thread-like shape over time, whereas UMAP displays a well-aligned grid shape temporally. Additionally, UMAP consistently achieves higher TPI scores than t-SNE across all cluster numbers.