M3BAT: Unsupervised Domain Adaptation for Multimodal Mobile Sensing with Multi-Branch Adversarial Training

Over the years, multimodal mobile sensing has been used extensively forinferences regarding health and well being, behavior, and context. However, asignificant challenge hindering the widespread deployment of such models inreal world scenarios is the issue of distribution shift. This is the phenomenonwhere the distribution of data in the training set differs from thedistribution of data in the real world, the deployment environment. Whileextensively explored in computer vision and natural language processing, andwhile prior research in mobile sensing briefly addresses this concern, currentwork primarily focuses on models dealing with a single modality of data, suchas audio or accelerometer readings, and consequently, there is little researchon unsupervised domain adaptation when dealing with multimodal sensor data. Toaddress this gap, we did extensive experiments with domain adversarial neuralnetworks (DANN) showing that they can effectively handle distribution shifts inmultimodal sensor data. Moreover, we proposed a novel improvement over DANN,called M3BAT, unsupervised domain adaptation for multimodal mobile sensing withmulti-branch adversarial training, to account for the multimodality of sensordata during domain adaptation with multiple branches. Through extensiveexperiments conducted on two multimodal mobile sensing datasets, threeinference tasks, and 14 source-target domain pairs, including both regressionand classification, we demonstrate that our approach performs effectively onunseen domains. Compared to directly deploying a model trained in the sourcedomain to the target domain, the model shows performance increases up to 12AUC (area under the receiver operating characteristics curves) onclassification tasks, and up to 0.13 MAE (mean absolute error) on regressiontasks.