Browsing by Subject "EEG"

Humans make decisions in highly complex physical, economic and social environments. In order to adaptively choose, the human brain has to learn about- and attend to- sensory cues that provide information about the potential outcome of different courses of action. Here I present three event-related potential (ERP) studies, in which I evaluated the role of the interactions between attention and reward learning in economic decision-making. I focused my analyses on three ERP components (Chap. 1): (1) the N2pc, an early lateralized ERP response reflecting the lateralized focus of visual; (2) the feedback-related negativity (FRN), which reflects the process by which the brain extracts utility from feedback; and (3) the P300 (P3), which reflects the amount of attention devoted to feedback-processing. I found that learned stimulus-reward associations can influence the rapid allocation of attention (N2pc) towards outcome-predicting cues, and that differences in this attention allocation process are associated with individual differences in economic decision performance (Chap. 2). Such individual differences were also linked to differences in neural responses reflecting the amount of attention devoted to processing monetary outcomes (P3) (Chap. 3). Finally, the relative amount of attention devoted to processing rewards for oneself versus others (as reflected by the P3) predicted both charitable giving and self-reported engagement in real-life altruistic behaviors across individuals (Chap. 4). Overall, these findings indicate that attention and reward processing interact and can influence each other in the brain. Moreover, they indicate that individual differences in economic choice behavior are associated both with biases in the manner in which attention is drawn towards sensory cues that inform subsequent choices, and with biases in the way that attention is allocated to learn from the outcomes of recent choices.

Physiological signals measured from the body, such as brain activity and motor behavior, can be used to infer different physiological states or processes in humans. Signal processing and machine learning often play a fundamental role in this assessment, providing unique approaches to analyzing and interpreting physiological data for a variety of applications, such as medical diagnosis and human-computer interaction. In this work, these approaches were utilized and adapted for two separate applications: brain-computer interfaces (BCIs) and the assessment of visual-motor skill in virtual reality (VR).

The goal of BCI technology is to allow people with severe motor impairments to control a device without the need for voluntary muscle control. Conventional BCIs operate by converting electrophysiological signals measured from the brain into meaningful control commands, eliminating the need for physical interaction with the system. However, despite encouraging improvements over the last decade, BCI use remains primarily in research laboratories. One of the biggest obstacles limiting their daily in-home use is the significant amount of time and expertise that is often required to set up the biosensors (electrodes) for recording brain activity. The most common modality for brain recording is electroencephalography (EEG), which typically employs gel-based “wet” electrodes for recording signals with high signal-to-noise ratios (SNRs). However, while wet electrodes record higher quality signals than dry electrodes, they often hinder frequent use because of the complex and time-consuming process of applying the electrodes to the scalp. Therefore, in this research, a signal processing solution was implemented to help mitigate noise in a dry electrode system to facilitate a more practical BCI device for everyday use in people with severe motor impairments. This solution utilized a Bayesian algorithm that automatically determined the amount of EEG data to collect online based on the quality of incoming data. The hypothesis for this research was that the algorithm would detect the need for additional data collection in low SNR scenarios, such as those in the dry electrode systems, and collect sufficient data to improve BCI performance. In addition to this solution, two anomaly detection techniques were implemented to characterize the differences between the wet and dry electrode recordings to determine if any additional types of signal processing would further improve BCI performance with dry electrodes. Taken as a whole, this research demonstrated the impact of noise in dry electrode recordings on BCI performance and showed the potential of a signal processing approach for noise mitigation. However, further signal processing efforts are likely necessary for full mitigation and adoption of dry electrodes for use in the home.

The second study presented in this work focused on signal processing and machine learning techniques for assessing visual-motor skill during a simulated marksmanship task in immersive VR. Immersive VR systems offer flexible control of an interactive environment, along with precise position and orientation tracking of realistic movements. These systems can also be used in conjunction with brain monitoring techniques, such as EEG, to record neural signals as individuals perform complex motor tasks. In this study, these elements were fused to investigate the psychophysiological mechanisms underlying visual-motor skill during a multi-day simulated marksmanship training regimen. On each of 3 days, twenty participants performed a task where they were instructed to shoot simulated clay pigeons that were launched from behind a trap house using a mock firearm controller. Through the practice of this protocol, participants significantly improved their shot accuracy and precision. Furthermore, systematic changes in the variables extracted from the EEG and kinematic signals were observed that accompanied these improvements in performance. Using a machine learning approach, two predictive classification models were developed to automatically determine the combinations of EEG and kinematic variables that best differentiated successful (target hit) from unsuccessful (target miss) trials and high-performing participants (top fourth) from low-performing participants (bottom fourth). Finally, in order to capture the more complex patterns of human motion in the spatiotemporal domain, time series methods for motion trajectory prediction were developed that utilized the raw tracking data to estimate the future motion of the firearm controller. The objective of this approach was to predict whether the controller’s virtually projected ray would intersect with the target before the trigger was pulled to shoot, with the eventual goal of alerting participants in real-time when shooting may be suboptimal.

Overall, the findings from this research project point towards a comprehensive psychophysiological signal processing approach that can be used to characterize and predict human performance in VR, which has the potential to revolutionize the design of current simulation-based training programs for realistic visual-motor tasks.

A series of fusion techniques are developed and applied to EEG and pupillary recording analysis in a rapid serial visual presentation (RSVP) based image triage task, in order to improve the accuracy of capturing single-trial neural/pupillary signatures (patterns) associated with visual target detection.

The brain response to visual stimuli is not a localized pulse, instead it reflects time-evolving neurophysiological activities distributed selectively in the brain. To capture the evolving spatio-temporal pattern, we divide an extended (``global") EEG data epoch, time-locked to each image stimulus onset, into multiple non-overlapping smaller (``local") temporal windows. While classifiers can be applied on EEG data located in multiple local temporal windows, outputs from local classifiers can be fused to enhance the overall detection performance.

According to the concept of induced/evoked brain rhythms, the EEG response can be decomposed into different oscillatory components and the frequency characteristics for these oscillatory components can be evaluated separately from the temporal characteristics. While the temporal-based analysis achieves fairly accurate detection performance, the frequency-based analysis can improve the overall detection accuracy and robustness further if frequency-based and temporal-based results are fused at the decision level.

Pupillary response provides another modality for a single-trial image triage task. We developed a pupillary response feature construction and selection procedure to extract/select the useful features that help to achieve the best classification performance. The classification results based on both modalities (pupillary and EEG) are further fused at the decision level. Here, the goal is to support increased classification confidence through inherent modality complementarities. The fusion results show significant improvement over classification results using any single modality.

For crucial image triage tasks, multiple image analysts could be asked to evaluate the same set of images to improve the probability of detection and reduce the probability of false positive. We observe significant performance gain by fusing the decisions drawn by multiple analysts.

To develop a practical real-time EEG-based application system, sometimes we have to work with an EEG system that has a limited number of electrodes. We present methods of ranking the channels, identifying a reduced set of EEG channels that can deliver robust classification performance.

At every moment in life we are receiving input from multiple sensory modalities. We are limited, however, in the amount of information we can selectively attend to and fully process at any one time. The ability to integrate the relevant corresponding multisensory inputs together and to segregate other sensory information that is conflicting or distracting is therefore fundamental to our ability to successfully navigate through our complex environment. Such multisensory integration and segregation is done on the basis of temporal, spatial, and semantic cues, often aided by selective attention to particular inputs from one or multiple modalities. The precise nature of how attention interacts with multisensory perception, and how this ramifies behaviorally and neurally, has been largely underexplored. Here, in a series of six cognitive experiments in humans using auditory and visual stimuli, along with electroencephalography (EEG) measures of brain activity and behavioral measures of task performance, I examine the interactions between attention, stimulus conflict, and multisensory processing. I demonstrate that attention can spread across modalities in a pattern that closely follows the temporal linking of multisensory stimuli, while also engendering the spatial linking of such multisensory stimuli. When stimulus inputs either within audition or across modalities conflict, I observe an electrophysiological signature of the processing of this conflict that is similar to what had been previously observed within the visual modality. Moreover, using neural measures of attentional distraction, I show that when task-irrelevant stimulus input from one modality conflicts with task-relevant input from another, attention is initially pulled toward the conflicting irrelevant modality, thereby contributing to the observed impairment in task performance. Finally, I demonstrate that there are individual differences in multisensory temporal processing in the population, in particular between those with extensive action-video-game experience versus those with little. However, everyone appears to be susceptible to multisensory distraction, a finding that should be taken into serious consideration in today's complex world of multitasking.

Episodic memory, as a cognitive construct, exists only in relation to those other cognitive constructs that reference it. It is, as Ribot suggests: the tactile, the muscular, the auditory and so forth. And it is even more than this, extending to a breadth of cognitive operations, including, for example, attention and cognitive control, both of which are generally believed to facilitate episodic memory encoding and episodic memory retrieval. Without these types of sensory and cognitive referents, episodic memory does not exist. Accordingly, these types of referents are critical to an understanding of episodic memory. Therefore, in this dissertation I examine how different cognitive constructs serve to facilitate episodic memory.

Chapter 2 examines attention-related subsequent memory effects. Many studies of subsequent memory rely upon a reverse inference, i.e. increased activity in attention-related networks during memory encoding is related to better subsequent memory, ergo increased attention predicts better memory. However, it is only through direct manipulation of attentional states and the examination of specific neural markers that this claim can be strongly established. Additionally, attention is a multifaceted process, and claims that attention in general facilitates memory ignore the fact that attention consists of a set of rapidly enfolding processes. To address these issues, I designed a modified visual-search EEG experiment with a subsequent long-term memory test. The utilization of a visual-search paradigm has advantages, as the search process evokes a series of independent and well-established attention-related EEG markers which can be linked to subsequent memory. All of the attentional effects examined were found to also predict subsequent memory, suggesting that these attentional processes associated with visual search, aid long-term memory formation as well.

Chapter 3 examines how large-scale network dynamics affect long-term memory retrieval. Until now, all studies of long-term memory have focused on individual regions, pair-wise connections between regions, or, very rarely, complex interactions between a small subset of regions. In a pair of fMRI studies, I use mathematical concepts from network science to examine the large-scale brain networks associated with successful remembering and forgetting. In doing so, I demonstrate that the hippocampus increases its integration with the rest brain when individuals successfully remember an item as compared to when they do not.

Chapter 4 examines how individual items are represented in the brain with machine-learning techniques and fMRI data. Studies of episodic memory often focus on things that are common across a set of items, while ignoring the uniqueness of individual events. However, an event’s uniqueness is what defines it as being episodic with respect to memory. A primary reason unique events are not often studied is the difficulty of decoding brain states associated with individual events. In Chapter 4, I develop a machine-learning framework, utilizing cross-subject single-item decoding, to predict what image or word a left-out subject is viewing. This establishes a robust way to detect individual events which could be used in service of better understanding episodic memory.

By examining long-term memory from these perspectives, I provide evidence of how different cognitive constructs facilitate episodic memory. In Chapter 2, I focus on the role of attentional processes with respect to episodic memory encoding, in Chapter 3, I focus on how large-scale network interactions facilitate episodic memory retrieval, and in Chapter 4 I focus on the representational nature of unique events. In all cases, the examination centers on how diverse processes coordinate in order to facilitate episodic memory.

Electroencephalography (EEG), a neuroscience method which requires sustained access to the scalp and hair, has many clinical and research applications. It is an essential feature of the rapidly growing consumer neurotechnology market. Neuroethicists have criticized EEG for being unaccommodating to phenotypic differences in hair type - a flaw which contributes to the systematic exclusion of minority groups from research. This exclusion legitimizes concerns about the generalizability of EEG research and effectiveness of EEG-based technologies. The following report employs a review of the most current literature across neuroscience, ethics, and technology publication sources to demonstrate how exclusion EEG research creates gaps in theoretical knowledge that disproportionately impact minorities and have profound implications for medical and consumer products. This paper summarizes the many applications of EEG and examines the impact of exclusion on EEG-based research and technology development. It outlines the risks of maintaining exclusion and provides policy recommendations for how to mitigate those risks by prioritizing inclusion in research methods.

Over the past decade there has been an explosion of interest in exploring how attention and reward value can interact with one another during cognition and behavior. This interdisciplinary work has already provided many critical findings that have revolutionized what have traditionally been isolated fields of study. However, there are still many unexplored aspects by which attention and value can interact with one another. Here I advance upon this interdisciplinary work by investigating several aspects of the dynamic interplay between attention and value. In the first three studies I look at how reward can influence attention and assess its neural impact using EEG. I first detail the neural mechanisms underlying reward-driven salience, a phenomenon that describes how reward-associated items receive higher priority during attentional orienting. These findings provide evidence that value-driven salience generates a unique increase in the strength of attentional orienting. In the second study I investigate whether associating distractors with rewards can lead to larger impairments in sustained spatial attention. Results indicate that sustained spatial attention can be resistant to a distractor’s reward-history, highlighting an important boundary condition for reward-related distraction. The objective of the third study was to investigate the neural processes underlying reward expectation and outcome processing, with a focus on how these reward expectations influence attention and attentional orienting. A core finding from this experiment is that outcome valence modulates the strength of attentional orienting, while uncertain outcomes lead to elongated attentional processing. In the fourth and final study I turn to investigating how attention can influence decision making. A burgeoning body of work has shown that attention can be highly predictive of choice, but it has not yet determined how inattention can influence decision processes. To this end I investigated if and how attentional distractors influence nutritional decision making using a combination of behavioral and eye-tracking measures. The results indicate that distractors can interrupt the decision process but that they do not reset it. Collectively, these studies demonstrate the diverse ways in which attention and reward can interact with one another, and how studying these interactions can help us better understand a number of real-world behaviors and circumstances.

Substantial correlative evidence links the synchronized, oscillatory neural firing patterns that emerge in Parkinson’s disease (PD) in the frequency range of 13-30Hz (termed “beta band”) with the development of bradykinesia and akinesia. Yet, a causal link between these beta frequency oscillations and symptoms of bradykinesia has not been demonstrated. I tested the hypothesis that the synchronized beta oscillations that emerge in PD are causal of symptoms of bradykinesia/akinesia through studies in intact and parkinsonian animals as well as PD patients.

Regarding the rat studies, I designed novel stimulation patterns to mimic the temporal characteristics of the beta oscillatory bursting pattern seen in single units in PD rats and patients. I applied these beta frequency patterned stimulus trains along with continuous frequency controls over a range of amplitudes via stimulating electrodes implanted unilaterally into the subthalamic nucleus (STN) of healthy and PD rats and assessed the effects on unit activity in the substantia nigra reticulata (SNr) and performance in motor tasks designed to assess forelimb bradykinesia and gross locomotor activity. I quantified the degree of unit entrainment in the SNr as a function of pattern and amplitude by calculation of the excitatory effective pulse fraction (eEPF) [1]. I further quantified the increase in SNr unit spectral beta frequency power due to the applied stimulation paradigms. I found that the beta-patterned paradigms were superior to low frequency controls at entrainment and induction of beta power in downstream substantia nigra reticulata (SNr) neurons. However, I found no deleterious effects on motor performance across a wide battery of validated behavioral tasks.

In PD patients, my objective was to determine how beta frequency oscillatory activity varies with disease progression and severity in human PD patients using cortical electroencephalogram (EEG). I recorded EEG in twenty-five PD patients of varying disease severity after overnight abstinence from PD medication. I recorded EEG at rest (eyes open and closed) and while patients performed various hand motor tasks. These tasks included one-handed isometric grip and rapid open/close movements. For each EEG channel data-stream for each patient, I calculated the total percent of spectral power in the beta band (PSP-β) with respect to movement state. I used stepwise regression to predict UPDRSIII scores from the normalized PSP-β values calculated for each channel during the ‘Eyes Open Rest’ state and found a significant, predictive regression equation. I assessed the relationship between UPDRSIII score and cortical coherence using linear regression and found significant, positive correlations between UPDRSIII score and coherence at beta band frequencies between pre-motor-motor and motor-somatosensory cortical areas. I observed phase amplitude coupling (PAC) between beta and gamma (30-200Hz) frequencies at rest and found it to be significantly altered by task, but found no effect of motor symptom progression on mean PAC.

My data suggest that certain metrics of beta band activity in pre-motor, motor, and somatosensory brain regions at rest may serve as a marker for degree of motor impairment, but that beta frequency oscillations may be an epiphenomenon and not necessary or sufficient for the generation of bradykinesia/akinesia in PD.

Research on biomarkers of brain disorders is an actively developing area. Biomarkers may allow for the early detection of diseases, which is essential for early intervention and improved outcomes. Biomarkers for monitoring the changes in the patient’s state can potentially increase the efficiency of clinical trials. Digital biomarkers, which emerged in recent years, rely on applications of machine learning methods to the data gathered by low-cost sensors, often embedded in consumer devices. Digital biomarkers have the potential to provide low-cost and more objective, granular, and sensitive to change metrics than traditional clinical ratings used in assessments of neurological and neurodevelopmental disorders. On the other hand, in traditional electrophysiological methods measuring brain activity, such as electroencephalography (EEG), biomarkers historically were based on visual analysis by clinicians, classical signal processing measures, or event-related potential (ERP) technique. Search for machine learning-based EEG biomarkers is an active area of research. This dissertation aims to build novel digital behavioral and EEG-based biomarkers and outcome measures by applying machine learning to behavioral, EEG, and concurrently recorded behavioral and EEG data. Machine learning models for the detection of gaze, human face and body landmarks, and automatic speech recognition achieve good performance on publicly available datasets. However, applying these models to a new clinical dataset immediately incurs a dataset shift problem, since the conditions under which real clinical video and audio data are recorded differe from the training dataset (e.g. different video camera angles, or audio noise). Furthermore, clinical datasets are in general much smaller than those used for training such models, and there are not enough human resources in the clinical setting to perform data labeling, making re-training not feasible. Yet, the question remains – whether the predictions from pre-trained models can provide valuable insight into human behavior and neurophysiology in the clinical setting, and whether they can be a source of clinically relevant findings. In this dissertation, we first explore this question in two use cases: (1) building digital measures of caregiver-child interaction in neurodevelopmental disorders using pre-trained pose detection deep learning models; (2) creating a digital biomarker of ataxic dysarthria using pre-trained automatic speech recognition deep learning models. We show that in the first case, our method enables to distinguish different clusters of caregiver responsiveness which are associated with a child’s caregiver- and clinician-reported socialization, communication, and language abilities, thus demonstrating the feasibility of using digital measures of caregiver-child interaction in clinical trials. In the second case, we demonstrate the convergent validity of our novel biomarker with clinician-reported scores and the greater sensitivity to change than clinician-reported scores on a longitudinal dataset. Second, we propose a novel deep learning model for detecting seizures in neonates from EEG data. We demonstrate the model’s high generalizability by evaluating it on an independent dataset from another hospital and show that model by design can be applied in different facilities with different EEG hardware. This approach has the potential to be clinically validated and will allow to scale up studies of neonatal seizures by increasing the sample sizes (including data from multiple clinical centers). Finally, we turn to the problem of combining EEG and behavioral biomarkers, which can improve biomarker sensitivity, but also provide new insights into brain-behavior relationships. In the study of autism, we propose a new metric of attentional preference to social/non-social stimuli and show that not only it distinguishes between autistic and neurotypical children, but also is differently associated with brain activity as measured by EEG. Then we turn to the question of scaling up EEG and behavior studies and provide the tool that allows measuring participants’ attention to the screen during EEG recording. This tool will allow to reduce human effort and make measurements of participants’ visual attention more objective, thus scaling up data preprocessing and allowing for multi-center studies of concurrent EEG and behavior.