Ferroelectric memory and architecture for deep neural network training in resistive crossbar arrays

Abstract

Deep neural networks (DNN) can perform cognitive tasks such as speech recognition and object detection with high accuracy. However, the computational cost to perform inference tasks with DNNs is a challenge for mobile applications, whereas the time and energy required to train DNN models can be prohibitive even at large data centers. The computational cost of Deep Neural Networks (DNN) is dominated by memory access and multiply accumulate operations. For this reason, it has been proposed to use resistive crossbar arrays to minimize the movement of weights and perform efficient multiply accumulate operations. These architectures store the weight value in multilevel resistive memory elements, and perform matrix-vector multiplications in the analog domain. One of the main challenges of these architectures is the limited resolution and nonlinearity of resistive memories available today. In this thesis, this limitation is addressed in two ways: by developing a model to design and optimize multilevel memories based on ferroelectric materials, and by designing an architecture to mitigate the limitations of resistive crossbars for DNN training.

First, ferroelectrics are studied for multilevel memory devices in resistive crossbar arrays. Ferroelectrics are ceramic materials that can have two nonvolatile polarization states. In their polycrystalline form, these materials are composed of a multitude of grains with independent polarization states, allowing for dense, nonvolatile, multilevel memories compatible with standard semiconductor fabrication processes. However, modeling the dynamics of polycrystalline ferroelectrics is challenging due to the statistical variations in the composition of its grains. For this purpose, a model to extract the statistical properties of a ferroelectric film and a Monte Carlo simulation that can describe and predict its polarization dynamics and variability were developed. This model provides the tools to characterize and optimize ferroelectric materials, and to design and evaluate devices, circuits and architectures for deep learning and other applications.

Secondly, architecture improvements to train DNN models in resistive crossbar arrays are presented. An accurate scheme for parallel weight update in resistive crossbar arrays is proposed and evaluated. By using pulse width- and frequency-modulated signals, the value of resistive elements in a crossbar array can be updated in parallel with higher accuracy than existing techniques based on stochastic multiplication. This scheme produces an unbiased multiplication with stochastic rounding, which is optimal for training neural networks with limited resolution. Finally, the mapping of DNN models to hardware with nonnegative weights is studied. To analyze different mapping schemes, a general vector-matrix multiplication is decomposed into a vector-matrix multiplication with nonnegative weight elements performed in a crossbar array, followed by a limited set of addition and subtraction operations described by a connection matrix. The mathematical conditions for the existence of such decomposition are derived and applied to fully connected and convolutional layers. Based on this analysis, an efficient mapping scheme is designed, which mitigates the effect of weight nonlinearity and limited resolution. These architectures are evaluated with low-level simulations of DNN training implemented in MATLAB and by extending the Keras open-source framework to incorporate nonideal weight elements and the connection matrix decomposition.