Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL

Understandings of the three-dimensional social behaviors of freely moving large-size mammals are valuable for both agriculture and life science, yet challenging due to occlusions in close interactions. Although existing animal pose estimation methods captured keypoint trajectories, they ignored deformable surfaces which contained geometric information essential for social interaction prediction and for dealing with the occlusions. In this study, we develop a Multi-Animal Mesh Model Alignment (MAMMAL) system based on an articulated surface mesh model. Our self-designed MAMMAL algorithms automatically enable us to align multi-view images into our mesh model and to capture 3D surface motions of multiple animals, which display better performance upon severe occlusions compared to traditional triangulation and allow complex social analysis. By utilizing MAMMAL, we are able to quantitatively analyze the locomotion, postures, animal-scene interactions, social interactions, as well as detailed tail motions of pigs. Furthermore, experiments on mouse and Beagle dogs demonstrate the generalizability of MAMMAL across different environments and mammal species.

DOI:https://doi.org/10.1038/s41467-023-43483-w

aAverage 3D pose error of each keypoint (cm). Red indicates errors larger than 4.0 cm. Prefixes 'l_' and 'r_' indicate 'left' and 'right', respectively.BBox plot of MAMMAL reconstruction errors on both visible and invisible keypoints using 10 views (n = 4 animalsn = 70 timepoints). The dashed green line indicates an error of 7 cm (10% of the pig body length in BamaPig3D dataset). The 'center' part has no invisible keypoints.cIntersection over Union (IoU) versus timepoints on BamaPig3D dataset for surface estimation accuracy. 70 timepoints were used. At each timepoint, IoUs were averaged over all the labeled 2D instances of 10 views. 'MAMMAL w/o Sil' means without silhouette information during mesh fitting. Shadows, standard error mean (SEM).dAn illustration of pigs with different weights and sizes for evaluation. Train Data, one of the pigs in BamaPig2D dataset. Other pigs are new identities, including 1) Moderate, a pig with similar body size to Train Data; 2) Very Fat, a pig with large belly; 3) Juvenile, a pig with very small body size.eBox plot of MAMMAL reconstruction errors on different pigs shown ind.n = 188, 188, 211 and 178 landmarks for 'Train Data', 'Very Fat', 'Moderate' and 'Juvenile' respectively.f-h, Box plot of 3D pose error of MAMMAL and triangulation (Tri) at different view configurations on the BamaPig3D dataset (n = 4 animalsn = 70 timepointsn = 5320 landmarks) (f). Fraction of instances versus the number of correctly reconstructed keypoints of an instance. The error threshold used for determining a correct keypoint is 7 cm (g). Percentage of correctly tracked keypoints versus different thresholds (h).IA qualitative comparison between MAMMAL and Tri using 10 views. Tri produced missing parts and false poses especially for legs.jCoordinate curves of 'r_shoulder' keypoint of one pig on BamaPig3D dataset. Tri often yielded missing predictions or highly noisy predictions due to frequent occlusions. InB,eandf, black bar, median; box shoulders, interquartile range (IQR); whiskers, 1.5 times the IQR; black square dots, mean.