Browsing by Subject "Deep neural networks"
- Results Per Page
- Sort Options
Item Open Access Efficient and Scalable Deep Learning(2019) Wen, WeiDeep Neural Networks (DNNs) can achieve accuracy superior to traditional machine learning models, because of their large learning capacity and the availability of large amounts of labeled data. In general, larger DNNs can obtain higher accuracy. However, there are two obstacles which hinder us building larger DNNs: (1) inference of large DNNs is slow which limits their deployment to small devices; (2) training large DNNs is also slow which slows down research exploration. To remove those obstacles, this dissertation focuses on acceleration of DNN inference and training. To accelerate DNN inference, original DNNs are compressed while keeping original accuracy. More specific, Structurally Sparse Deep Neural Networks (SSDNNs) are proposed to remove neural components. In Convolutional Neural Networks (CNNs), neurons, filters, channels and layers can be removed; in Recurrent Neural Networks (RNNs), hidden sizes can be reduced. The study shows that SSDNNs can achieve higher speedup than sparse DNNs which have non-structured sparsity. Besides SSDNNs, a Force Regularization is proposed to enforce DNNs to lower-rank space, such that DNNs can be decomposed to lower-rank architectures with fewer ranks than traditional methods. The dissertation also demonstrates that SSDNNs and Force Regularization are orthogonal and can be combined for higher speedup. To accelerate DNN training, distributed deep learning is required. However, two problems hinder us using more compute nodes for higher training speed: Communication Bottleneck and Generalization Gap. Communication Bottleneck is that communication time will increase and dominate when the distributed systems scale to many compute nodes. To reduce gradient communication in Stochastic Gradient Descent (SGD), SGD with low-precision gradients (TernGrad) is proposed. Moreover, in distributed deep learning, a large batch size is required to exploit system computing power; unfortunately, accuracy will decrease when the batch size is very large, which is referred to as the Generalization Gap. One hypothesis to explain Generalization Gap is that large-batch SGD sticks at sharp minima. The dissertation proposes a stochastic smoothing (SmoothOut) to escape sharp minima. The dissertation will show that TernGrad overcomes Communication Bottleneck and SmoothOut helps to close the Generalization Gap.
Item Open Access Privacy and Robustness of Deep Neural Networks(2021) Li, BaiConcerns related to security and confidentiality have been raised when applying ma- chine learning to real-world applications. In this dissertation, I mainly discuss ap- proaches to defending models against membership inference attacks and adversarial attacks. Membership inference attacks attempt to infer the training set information from observing the model output, while adversarial attacks aim to alter the output of models via introducing minimal perturbations in the input.This dissertation consists of three main parts. In the first part, I show that stochastic gradient Markov chain Monte Carlo (SG-MCMC) – a class of scalable Bayesian posterior sampling algorithms – satisfies strong differential privacy, when carefully chosen stepsizes are employed. I develop theory on the performance of the proposed differentially-private SG-MCMC method and conduct experiments to support the analysis, showing that a standard SG-MCMC sampler with minor mod- ification can reach state-of-the-art performance in terms of both privacy and utility on Bayesian learning. In the second part, I introduce a framework that is scalable and provides certified bounds on the norm of the input manipulation for constructing adversarial examples. A connection between robustness against adversarial pertur- bation and additive random noise is established, and a training strategy that can significantly improve the certified bounds is proposed. In the third part, I conduct experiments to understand the behavior of fast adversarial training. Fast adversar- ial training is a promising approach that remarkably reduces computation time for adversarially robust training, yet it can only run for a limited number of training epochs, resulting in sub-optimal performance. I show the key to its success is the ability to recover from overfitting to weak attacks. I then extend the findings to improve fast adversarial training, demonstrating superior robust accuracy to strong adversarial training, with much-reduced training time.
Item Open Access Single Image Super Resolution:Perceptual quality & Test-time Optimization(2019) Chen, LeiImage super resolution is defined as recovering a high-resolution image given a low-resolution image input. It has a wide area of applications in modern digital image processing, producing better results in areas including satellite image processing, medical image processing, microscopy image processing, astrological studies and surveillance area. However, image super resolution is an ill-posed question since there exists non-deterministic answer in the high resolution image space, making it difficult to find the optimal solution.
In this work, various research directions in the area of single image super resolution are thoroughly studied. Each of the proposed methods' achievements as well as limitations including computational efficiency, perceptual performance limits are compared. The main contribution in this work including implementing a perceptual score predictor and integrating as part of the objective function in the upsampler algorithm. Apart from that, a test-time optimization algorithm is proposed, aiming at further enhance the image quality for the obtained super-resolution image from any upsampler. The proposed methods are implemented and tested using Pytorch. Results are compared on baseline applied datasets including Set5, Set14, Urban100 and DIV2K.
Results from perceptual score predictor was evaluated on both PSNR precision index and perceptual index, which is a combination of perceptual evaluation Ma score and NIQE score. With new objective function, the upsampler achieved to move along the trade-off curve of precision and perception. The test-time optimization algorithm achieved slightly improvements in both precision and perception index. Note that the proposed test time optimization does not require training of new neural network, thus, is computationally efficient.