Browsing by Author "Pajic, Miroslav"

Continuous deep brain stimulation (cDBS) of either subthalamic nucleus (STN) or globus pallidus (GP) is an effective therapy in Parkinson’s Disease (PD) but is inherently limited by lack of responsiveness to dynamic, fluctuating symptoms intrinsic to the disease. Adaptive DBS (aDBS) adjusts stimulation in response to neural biomarkers to improve both efficacy and battery life. This thesis discusses 1) the development of dual target STN+GP aDBS with a novel, external adaptive controller and 2) the outcomes from a first in-human clinical trial in PD patients (n = 6; NCT #03815656) in order to assess efficacy of the aDBS controller.We performed random amplitude experiments to probe system dynamics and thus estimated initial aDBS parameters. We then implemented an innovative proportional plus integral (PI) aDBS using a novel distributed architecture. The PI aDBS controller was first evaluated in the clinic settings and then compared to cDBS in the home settings. The results showed that the PI aDBS control reduced average power delivered while preserving improved Unified Parkinson’s Disease Rating Scale (UPDRS) III scores in the clinic and reduced beta oscillations during blinded testing in the home setting. Thus, we demonstrated that the novel PI aDBS may enhance chronic, symptomatic treatment of PD.

Today’s safety-critical Cyber-Physical Systems (CPS) are networked with distributed sensing, control, and actuation components, and external network connectivity to accomplish high performance and utility requirements. Intelligent CPS devices are revolutionizing many application domains including vehicular and industrial systems; yet, while their vulnerabilities lead to high-profile incidents, their security has commonly been an afterthought. Standard defense mechanisms relying on crypto-techniques incur high computation/bandwidth overheads, precluding deployment on resource-constrained platforms. In this dissertation, we introduce design-time methodologies for integrating security in such CPS by exploiting physics of controlled systems to relax stringent security requirements conflicting with resource availability. To provide guarantees for control quality despite attacks, we perform security-aware deadline-driven real-time scheduling of authentication services on processors (where computation time is the bottleneck), the shared network (where bandwidth is limited), and in an end-to-end fashion (where both computation time and bandwidth are constrained).

Another challenge, achieving CPS modularity, may be addressed with distribution of control functionalities. Since challenges in control distribution are domain-specific, we focus on cyber-manufacturing, enabling higher autonomy and reconfigurability levels of manufacturing resources. We develop a method for distribution of event-driven sequential control, starting from an existing centralized design, and formally mapping it to distributed controllers guaranteeing correctness. To ensure reliable and resilient operation in the presence of communication faults and attacks, we incorporate stochastic channel and non-deterministic attack models to verify system-level safety and liveness properties of the distributed system. If properties are violated, we improve code generation for the target platforms to include patches necessary to satisfy properties. Additionally, we consider distribution of Computerized Numerical Control (CNC); we propose an architecture where functionalities of a centralized controller are mapped to a high-level planner, and low-level controllers that accompany manufacturing resources (i.e., axes). We show that maintaining sufficient synchronization among controllers guarantees insignificant accuracy loss due to motion control distribution. We define requirements for cyber-physical synchronization critical for implementing distributed CNC due to clock skews in distributed systems, and provide design-time tradeoffs by investigating real-time/bandwidth requirements for different distributed architectures.

While the presented methods are applicable in other domains, validation is based on multiple vehicular and industrial systems.

High dimensional random dynamical systems are ubiquitous, including-but not limited to- cyber-physical systems, daily return on different stocks of S\&P 1500 and velocity profile of interacting particle systems around McKeanVlasov limit. Mathematically speaking, observed time series data can be captured via a stable $n-$ dimensional linear transformation `$A$' and additive randomness. System identification aims at extracting useful information about underlying dynamical system, given a length $N$ trajectory from it (corresponds to an $n \times N$ dimensional data matrix). We use spectral theorem for non-Hermitian operators to show that spatio-temperal correlations are dictated by the \emph{discrepancy between algebraic andgeometric multiplicity of distinct eigenvalues} corresponding to state transition matrix. Small discrepancies imply that original trajectory essentially comprises of multiple \emph{lower dimensional random dynamical systems living on $A$ invariant subspaces and are statistically independent of each other}. In the process, we provide first quantitative handle on decay rate of finite powers of state transition matrix $\|A^{k}\|$ . It is shown that when a stable dynamical system has only one distinct eigenvalue and discrepancy of $n-1$: $\|A\|$ has a dependence on $n$, resulting dynamics are \emph{spatially inseparable} and consequently there exist at least one row with covariates of typical size $\Theta\big(\sqrt{N-n+1}$ $e^{n}\big)$ i.e., even under stability assumption, covariates can \emph{suffer from curse of dimensionality }.

In the light of these findings we set the stage for non-asymptotic error analysis in estimation of state transition matrix $A$ via least squares regression on observed trajectory by showing that element-wise error is essentially a variant of well-know Littlewood-Offord problem and(can be extremely sensitive to dimension of the state space and number of iterations). We also show that largest singular value of the data matrix can be cursed by dimensionality even when state-transition matrix is stable. Overarching theme of this thesis is new theoretical results on spectral theorem for non-Hermitian operators, non-asymptotic behavior of high dimensional dynamical systems , which we incorporate with the work of Talagrand on concentration of measure phenomenon to better understand behavior of the structured random matrices(data matrix) and subsequently the performance of different learning algorithms with dependent data. Besides, we also show that there exists stable linear Gaussians with process level Talagrands' inequality linear in dimension of the state space(previously an open problem), along with deterioration of mixing times with increase in discrepancy between algebraic and geometric multiplicity of $A$.

In this work, we address the safe navigation problem for the robot equipped with the neural network controller. Our goal is to propose a neural network controller representation that can efficiently and safely learn a safe policy. By following the learned safe policy, the robot can reach the goal state while avoiding hitting obstacles and walls all the time. We use Hamilton Jacobi safety analysis to improve the safety awareness of the policy and integrate it within the value iteration network to generalize to the new, unseen domains outside the training set. Applying the transfer learning techniques, we can learn a reward function that maps each state to a reasonable reward value. We use the learned reward function to construct the unknown part in the discrete-time Hamilton Jacobi value function and integrate this Hamilton Jacobi value function into the value iteration network to construct our Hamilton Jacobi value iteration network model. Finally, we compare the performance of our model with the value iteration network model in the grid world domains to show our model can safely learn a safe policy that generalizes to the new, unseen domains.

Artificial intelligence (AI) and deep learning (DL) have recently shown success in domains related to healthcare and its decision-making systems. However, most of the existing methods are developed upon benchmark environments which are often defined with simplistic dynamics and allow access to data that are well-structured, pre-processed, and with substantial amount. It is intractable to leverage such methods to facilitate real-world applications; as limited access to the real-world healthcare environments leads to significantly reduced sample efficiency during training. Moreover, strict safety protocols are usually enforced in practice upon deployment to human participants, while the policy selection cri- teria weighs human feedback (HF) more than environmental returns; both of which could be intractable to be captured in simulations. From data logging perspectives, data irregularities are often encountered in healthcare facilities, e.g., missingness due to malfunctioned devices.

This dissertation aims to introduce AI/ML methods that can overcome limitations including insufficient and imperfect data as well as complying with safety protocols, which are applicable to real-world decision-making processes in healthcare systems, with focuses on (i) sample-efficient reinforcement learning (RL) based frameworks that can synthesize control policies of medical devices to maximize both environmental returns and HF in offline manners, with off-policy evaluation (OPE) facilitating the evaluation of RL policies without online interactions, i.e., for improved safety and efficiency upon deployment of the RL policies. (ii) DL-based analyses of multivariate healthcare data constituted by multiple modalities to facilitate clinical decision making systems, by tackling data irregularities and capturing underlying factors important to automated disease diagnoses and prognoses.

To tackle (i), we introduce an algorithmic OPE framework, variational latent branching model (VLBM), which can be integrated into most existing offline RL methods for effi- cient and safe policy evaluation and selection upon deployment. Specifically, it leverages variational inference to learn the transition function of MDPs by formulating the envi- ronmental dynamics as a compact latent space, from which the next states and rewards are then sampled. Its efficacy is validated by benchmark environments including Mujoco and Adroit. Then, an OPE for human feedback (OPEHF) method is developed on top of VLBM’s framework to capture the HF participants could have provided once the policies are deployed, further ensuring the satisfaction of human participants who received proce- dures or medical devices guided by RL agents. At last, we design a full-stack offline RL policy optimization pipeline, into which both OPE methods are integrated, toward training control policies of a implantable deep brain stimulation (DBS) device for treatment of Parkinson’s disease (PD), by adjusting the stimulation amplitude in real time. The goal is to reduce the energy used for generating the stimulus, while maintain the same level of treatment (i.e., control) efficacy as continuous DBS (cDBS) (i.e., constantly stimulating with the highest amplitude possbile, determined by clinicians). The efficacy is validated a cohort of 5 human participants, where results show that, and OPE components are able to pinpoint high-performing policies among the policy candidates trained using different offline RL algorithms or hyper-parameter sets.

In terms of (ii), we introduce three frameworks to address the following challenges, respectively. (ii.a) Data missingness, e.g., due to the non-periodical logging of patient vitals or lab results. (ii.b) High dimensionality and multi-modality within healthcare data, given that substantial amount of lab/vital results need to be recorded, and such information could come in the format of images (e.g., CT scan), tabular (e.g., demographic information) etc. The efficacy of each framework is validated by retrospective studies pertaining to the identification, prognoses and treatment of ophthalmic diseases. The results show that our frameworks are able to accurately identify the presence of diseases and automatically design treatment plans.

With the advent of Machine Learning and the existence of Electronic Health Records, with most non-federal acute care hospitals a large number of office-based physicians already having opted for having a certified EHRs, each patient has essentially become a big data problem for medical predictions. This is also true in the field of Ophthalmology with its various specific modalities. Across two projects we explore how electronic health records can be used to make predictive model for various condition using machine learning.

Using patient histories and demographics such as age, gender, and race, body mass index (BMI), medications, biologicals, comorbidities, past medical history, and visual acuities we model a risk classifier for progression of age-related macular degeneration from its dry for to its wet form, which is a much faster progressing form of the disease. We found older age, use of biologicals such as anti-VEGF agents, and lover visual acuity to be associated with increased risk of progression of the disease. Our model gave an indicative tool with accuracy of 0.778±0.045, F1 score of 0.795±0.038 and sensitivity of 0.86±0.068. Also using imaging modalities such as SD-OCTs we model the detection of hydro-chloroquine toxicity related retinopathy, and attempt propose a prediction model. Our Model was able to detect hydro-chloroquine toxicity induced retinopathy with a precision of 0.72, recall of 0.92, F1 score of 0.81, and accuracy of 0.81.

Using the two projects we showed that using data extracted from electronic health records we can make effective models for various tasks using machine learning fairly well.

Cyber-Physical Systems (CPS) feature synergetic integration of multiple subsystems to control physical environments through cycles of sensing and actuation. The correct-by-design paradigm aims to provide guarantees on the performance of CPS by utilizing formally-proven algorithms for synthesis and validation of various system components. This paradigm postulates the ability to both derive adequate abstractions of the system, and mathematically formalize design requirements. Therefore, developing mathematical tools that allow system designers to easily model CPS, and to capture design requirements, is imperative to such paradigm.

This dissertation provides theoretical and experimental contributions towards modeling and development of assured and adaptive CPS. In particular, we propose Delayed Action Game (DAG) to aid with modeling CPS where part of the state is hidden from the controller. The formalism deploys the concept of delaying actions as means to hide them from other players without the usage of private variables, allowing the use of off-the-shelf model checkers for analysis. Based on a DAG model, we design an algorithm that utilizes model checkers to synthesize optimal strategies. In addition, we propose Context-Aware Probabilistic Temporal Logic (CAPTL) to aid with formalizing temporal requirements that can naturally described as a set of objectives that are prioritized based on some probabilistic conditions. Furthermore, we develop the algorithm that allow for synthesizing optimal strategies for a Markov Decision Process (MDP) that satisfy a given CAPTL-based requirement.

We deploy the theoretical frameworks in two application domains: human-robot interaction and digital microfluidics, with the goal of designing systems to be more adaptive to their environments. First, we develop protocols for supervisory systems where a human operator, supervising a number of Unmanned Aerial Vehicles (UAVs), can intermittently perform geolocation tasks to aid in detection of possible attacks. We model the system as a DAG, and further use it to synthesize security-aware human-UAV protocols that both provide UAV path plans, increasing the chances of attack detection, and specify the time instances at which the operator is advised to perform a geolocation task. Second, we propose a stochastic game-based framework for droplet routing in Micro-Electrode-Dot Array (MEDA) biochips. The framework utilizes the ability to sense microelectrode health to synthesize routing plans that adapt to the microelectrode degradation levels in run-time. Using multiple real-life bioassays for evaluation, we show that the framework increases the probability of successful completion of benchmark bioassays. Finally, we adapt the framework to utilize Deep Reinforcement Learning (DRL) algorithms to achieve the same task.

The increase in network connectivity has also resulted in several high-profile attacks on cyber-physical systems. An attacker that manages to access a local network could remotely affect control performance by tampering with sensor measurements delivered to the controller. Recent results have shown that with network-based attacks, such as Man-in-theMiddle attacks, the attacker can introduce an unbounded state estimation error if measurements from a suitable subset of sensors contain false data when delivered to the controller. While these attacks can be addressed with the standard cryptographic tools that ensure data integrity, their continuous use would introduce significant communication and computation overhead. Consequently, we study effects of intermittent data integrity guarantees on system performance under stealthy attacks. We consider linear estimators equipped with a general type of residual-based intrusion detectors (including χ 2 and CUSUM detectors), and show that even when integrity of sensor measurements is enforced only intermittently, the attack impact is significantly limited; specifically, the state estimation error is bounded or the attacker cannot remain stealthy. Furthermore, we present methods to: (1) evaluate the effects of any given integrity enforcement policy in terms of reachable state-estimation errors for any type of stealthy attacks, and (2) design an enforcement policy that provides the desired estimation error guarantees under attack. Finally, on three automotive case studies we show that even with less than 10% of authenticated messages we can ensure satisfiable control performance in the presence of attacks.

Autonomous systems (AS), enhanced by the capabilities of reinforcement learning (RL), are expected to perform increasingly sophisticated tasks across various civilian and industrial application domains. This expectation arises from their promising ability to make decisions solely based on perception without human intervention. In addition to high efficiency, AS often require robustness and safety guarantees for real-world deployment. In this thesis, we propose model-free RL approaches that obtain controllers for AS operating in unknown, stochastic, and potentially adversarial environments directly from linear temporal logic (LTL) specifications defined on state labels, such as safety and liveness requirements. This ensures that the learned controllers satisfy the desired properties, avoiding unintended consequences, and remain robust against adversarial behavior.We first derive a novel rewarding and discounting mechanism from the LTL specifications for Markov decision processes. We show that a policy learned by a model-free RL algorithm, which maximizes the sum of these discounted rewards, also maximizes the probability of satisfying the LTL specifications. We generalize this approach to multiple objectives, where the utmost priority is given to ensuring safety. Satisfaction of the other LTL specifications takes a secondary role, and the tertiary objective is to enhance the quality of control. We then extend our results to zero-sum stochastic games to ensure the robustness of learned controllers against any unpredictable nondeterministic environment behavior. Addressing the scalability challenges inherent in learning controllers for stochastic games, we propose heuristics and approximate methods to further accelerate the learning process. We illustrate how our approach can be utilized to learn controllers that are resilient against stealthy attackers, capable of disrupting the agent's actuation without being detected. We further discuss an approach for cases where state labels are absent. This approach aims to learn a labeling function that translates raw state information into object properties applicable in LTL specifications, thereby enabling the learning of controllers from LTL specifications. We conclusively show the effectiveness of our approaches in successfully learning optimal controllers through numerous case studies. These controllers maximize the probability of satisfying LTL specifications in the worst case, thereby exhibiting resilience against adversarial behavior. Moreover, our methods demonstrate scalability across a broad spectrum of LTL specifications, consistently surpassing the performance of existing approaches.

Cyber-physical systems (CPS) consist of computing and physical components, such as sensors, actuators, and processors, and these components communicate through network to control and monitor the physical processes. Although many cyber-physical systems (CPS) operate in safety critical scenarios and the heterogeneous component connectivity provides numerous possible points of attack, most of existing systems are only weakly protected by legacy components, such as intrusion detectors. On the other hand, security-aware resource allocation can significantly reduce the security-related overhead and thus system cost; where the idea is to focus on protecting the critical system components and communication links, which if compromised could significantly degrade system performance. Yet, to achieve this, we need methods to analyze system vulnerability, in terms of performance degradation under attack, for different types of attacks especially the ones that are potentially stealthy to the deployed intrusion detection mechanisms. In this dissertation, we analyze the vulnerability cyber-physical systems modeled by a control system to adversarial attacks. The contribution of the thesis is fivefold. First, the vulnerability of systems with linear time invariant (LTI) model subject to bounded noise is analyzed. For such systems, we also show how performance guarantees can be achieved when the system is equipped to intermittent data authentication. Second, for a nonlinear dynamical system equipped with extended Kalman filter and Chi-square ID, we leverage machine learning methods and develop learning-enabled attack generators capable of designing stealthy attacks that maximally degrade system operation. We show how such problem can be cast within a learning-based grey-box framework where only parts of the run-time information are known to the attacker. Third, we study performance of perception-based cyber-physical systems in the presence of attacks and provide methods for modeling and analysis of their vulnerability to stealthy attacks on both physical and perception-based sensing. Here, we define the notion stealthiness which is independent of the deployed ID and we provide the condition for which the system will be vulnerable to such stealthy yet impactful attacks. Fourth, we focus on analyzing vulnerability of general nonlinear dynamical control systems to stealthy false data injection attack on sensors. We use a similar notion of stealthiness as in the third contribution where the attack is considered to be stealthy if it undetected from any existing ID. We show that even for such strong notion of stealthiness, there are a large class of systems that are vulnerable to these stealthy effective attacks. Finally, we consider stealthy perception-based attacks on unmanned aerial vehicles. Specifically, we introduce a method to consistently attack both the sensor measurements and camera images over time, in order to cause control performance degradation (e.g., by failing the mission) while remaining stealthy (i.e., undetected by the deployed anomaly detector). We show that stealthy, yet effective attacks can be designed by changing images of the ground vehicle’s landing markers as well as suitably falsifying sensing data. We illustrate the effectiveness of our attacks in Gazebo 3D robotics simulator.

Unmanned aerial vehicles (UAVs) have been widely used in various domains of industry and people's daily life. However, the vulnerability of UAVs has been underestimated and compromised UAV systems could put autonomous missions at risk. This paper studies the vulnerability of UAVs and implements an effective and stealthy perception-based attack. This stealthy attack does not rely on the vulnerability of deep neural networks but makes use of fake image transformations and sensor signal manipulations. In particular, two specific autonomous missions are investigated: ($i$) ground vehicle tracking (GVT), and ($ii$) vertical take-off and landing (VTOL) of a quadcopter on a moving ground vehicle. Experimental results with Gazebo simulations show that this stealthy attack causes significant deviation from the UAV's designated trajectories, and remains undetected from state-of-art intrusion detectors.

This paper implements the first closed-loop adaptive deep brain stimulation (DBS) system for Parkinson’s patients that updates with multiple input streams from disparate data sources analyzed in real time. Input data streams include the brain's local field potential (LFP) from DBS leads, hand tremors, and heart rate. This approach is designed to be evaluated on 6 patients with Parkinson's disease implanted with the Medtronic Summit™ RC+S systems and has the potential to be more effective in simultaneously controlling multiple symptoms commonly presented in Parkinson’s patients. In the case of the DBS control of both the bradykinesia and tremor, the system demonstrated in this paper has the ability to overcome the challenge of the “breakout tremor” presented in previous studies. The system in-lab testing characterization results indicated that the system could be used to control closed-loop deep brain stimulation systems with a high degree of accuracy and robustness.