Browsing by Author "Wang, Rui"

The goal of my thesis is to understand the genetics of a complex behavioral trait, vocal learning, which serves as a critical substrate for human spoken language. With the available genomes of 23 mammals, I developed a novel approach based on molecular homoplasy to reveal Single Non-random Amino Acids Patterns (SNAAPs) that are associated with convergent traits, a task that proved intractable for standard approaches, e.g. dN/dS analyses. Of 73 genes I identified in mammalian vocal learners, ~25% function in neural connectivity, auditory or speech processing. Remarkably, these include a group of 6 genes from the ROBO1 axon guidance pathway. In birds, I found ROBO1 and its ligand SLIT1 show convergent differential expression in the motor output song nucleus of the three independent lineages of vocal learners but not in analogous brain areas of vocal non-learners, and ROBO1 is developmentally regulated during song learning critical periods in songbirds. In a different set of genes, I came across an unexpected discovery of the excess sharing of homoplastic substitutions in humans and domesticated species. I revealed biased nucleotide transitions (mostly favoring A/G mutation) for above amino acid substitutions and found that this rule was significantly relaxed during domestication for artificial selection. Overall, my thesis has resulted in a novel approach for studying convergent complex traits and provided critical insights into the evolution of vocal learning specifically, and complex traits generally.

The performance of machine learning systems has been dramatically improving in recent years thanks to the advent of pre-trained deep neural networks. Such models are generally adopted as the foundation for feature extraction, yielding state-of-the-art results when being further trained (or fine-tuned) on downstream tasks. Nonetheless, the pre-trained deep neural networks are generally huge in size, e.g., with millions or billions of parameters, thus demanding abundant data and computation resources during fine-tuning. Therefore, it is of pragmatic merit to investigate efficient training approaches with such pre-trained deep neural networks for low-resource scenarios, i.e., when there is limited computation budget or annotated data for the down stream tasks. In this dissertation, we consider low-resource scenarios for efficient training with the following tasks: i) natural language understanding, ii) fair text generation, and iii) compositional image retrieval.

We first study natural language understanding through its sub-tasks of sequence classification and sequence labeling.We propose an attention-based architecture and a data-free distillation framework, respectively, both designed for the scenario where there are limited data annotations. These approaches improves the data efficiency in fine-tuning pre-trained deep neural networks for better understanding of natural language. We then explore strategies in fine-tuning pre-trained language models for demographic fairness in text generation. Specifically, we propose to minimize the mutual information between the semantics in the generated text sentences and their demographic polarity, i.e., the demographic group to which the sentence is referring. We develop a computational efficient approach in estimating the upper bound of such mutual information via importance sampling, reducing the number of model inference required during training. Finally, for compositional image retrieval, we propose a visual prompt tuning mechanism and a self-supervised auxiliary task that adapt a pre-trained vision-language model to perform image retrieval tasks with only few annotations, while improving the computation efficiency by allowing the pre-trained parameters to be frozen during training.

Overall, my research work draws attention to the data and computational efficiency of the current large pre-trained deep neural networks, improving the flexibility in deploying such models in the considered low-resource scenarios.