Syracuse

ID: S1

Title: Attribute-based object localization

Abstract: Despite the recent advances in object detection, it is still a challenging task to localize
a free-form textual phrase in an image. Unlike locating objects over a deterministic number of
classes, localizing textual phrases involves a massively larger search space. Thus, along with
learning from the visual cues, it is necessary to develop an understanding of these textual
phrases and its relation to the visual cues to reliably reason about locations of described by the
phrases. Spatial attention networks are known to learn this relationship and enable the
language-encoding recurrent networks to focus its gaze on salient objects in the image. Thus,
we propose to utilize spatial attention networks to refine region proposals for the phrases from
a Region Proposal Network (RPN) and localize them through reconstruction. Utilizing
in-network RPN and attention allows for an independent/self-sufficient model and
interpretable results respectively.
Deliverable: Code and benchmarks used in the experiments.

Proposed budget: $50K/year for two years.

Performing site: Syracuse (PI: Qinru Qiu).

ID: S2

Title: Biologically plausible spike-domain backpropagation for in-hardware learning

Abstract: Asynchronous event-driven computation and communication through spikes enable
massively parallel, extremely energy efficient and highly robust neuromorphic hardware
specialized for spiking neural networks (SNN). However, the lack of a unified and effective
learning algorithm limits the SNN to shallow networks with low accuracies. While
backpropagation algorithm, which utilizes gradient descent to train networks, has been
successfully used in Artificial Neural Networks (ANNs), it is neither biologically plausible nor
neuromorphic implementation friendly. In this project, we propose to develop methods to
achieve backpropagation in spiking neural networks. This will enable error propagation through
spiking neurons in a more biologically plausible way and hence makes the in-hardware learning
feasible on existing neuromorphic processors.
Deliverable: Code and benchmarks used in the experiments.

Proposed budget: $50K/year for two years.

Performing site: Syracuse (PI: Qinru Qiu).

ID: S3

Title: Fast prediction for UAV traffic/communication in a metropolitan area

Abstract: With the rapid increase of the UAV applications in delivery, surveillance, and rescue
mission, there is an urgent need for UAV traffic management that ensures the safety and
timeliness of the missions. Accurate and fast UAV traffic prediction and resource usage
estimation is a key technique in the traffic management system. It allows us to 1) evaluate
different mission schedules or 4G/5G resource allocation schemes in short period of time; and
2) adjust mission control policy in real-time. A good prediction model should not only consider
the UAV mission information, but also the environment information such as weather, 4G/5G
base station distribution, etc. Its complexity is much beyond the traditional analytical approach.
We propose to solve this problem using machine learning. The model has a combined of
convolutional neural network (CNN) and recurrent neural network (RNN). The inputs are
multi-channel time varying streams, such as weather map, cellular network usage map,
geographical constraints, and UAV launching/landing information, etc. The outputs will be
predicted UAV conflict probability map and 4G/5G channel congestion map. Compression and
acceleration of the model will also be studied for real-time prediction.
Deliverable: Model and implementation of the UAV traffic/communication prediction
framework. Technical publications.

Proposed budget: $50K/year for two years.

Performing site: Syracuse (PI: Qinru Qiu).

ID: S4

Title: FPGA-based Neuromorphic System for Sensor Data Processing

Abstract: Human brain’s computational power, energy efficiency, and its ability to process
real-time sensor data are attractive features for the evolving trend of the Internet of Things.
Inspired by the architecture of the human brain, brain-inspired computing is based on spiking
neural networks (SNN). SNN consists of networks of homogeneous computing elements i.e.
neurons communicating with spikes. The sparsely distributed asynchronous events i.e. spikes,
allow for a highly parallel and distributed computing architecture. Thus, a neuromorphic
hardware is highly desirable for embedded self-contained applications such as real-time sensor
data processing. However, several challenges prohibit the wide utilization of neuromorphic
hardware: limited weight precision, I/O precision, training difficulty caused by the discrete
nature of SNN. In this work, we built a flexible and reconfigurable FPGA-based neuromorphic
system for sensor data processing. An innovative workflow is proposed to mitigate
aforementioned issues. To demonstrate its effectiveness, we trained a neural network to
recognize sign language, and then used the proposed workflow to map it to the neuromorphic
system.
Deliverable: 1) Technical publication on premium ML conference or journal; 2) Code and
benchmarks used in the experiment. 3) A complete FPGA-based neuromorphic system
implementation.

Proposed budget: $50K/year for two years.

Performing site: Syracuse (PI: Qinru Qiu).

ID: S5

Title: Autonomous Waypoint Planning and Trajectory Generation for Multi-rotor UAVs

Abstract: Safe and effective operations for multi-rotor unmanned aerial vehicles (UAVs)
demand obstacle avoidance strategies and advanced trajectory planning and control schemes
for stability and energy efficiency. To solve those problems in one framework analytically is
extremely challenging when the UAV needs to fly large distance in a complex environment. To
address this challenge, we propose a two-level strategy that ensures a global optimal solution.
At the higher-level, deep reinforcement learning (DRL) is adopted to select a sequence of
waypoints which lead the UAV from its current position to the destination. At the lower-level,
an optimal trajectory is generated between each pair of adjacent waypoints. While the goal of
trajectory generation is to maintain the stability of the UAV, the goal of the waypoints planning
is to select waypoints with the lowest control thrust consumption throughout the entire trip
while avoiding collisions with obstacles.
Deliverable: Technical publication on cyber-physical system conference or journal, and the
software implementation of the proposed DRL framework.

Proposed budget: $50K/year for two years.

Performing site: Syracuse (PI: Qinru Qiu).

ID: S6

Title: Multi-modal Fusion with Non-linear Dependence

Abstract: Fusion of data from multiple sources/sensors has been shown to significantly improve
inference performance. However, since each sensor carries a unique physical trait, sensor
heterogeneity is the first critical challenge for multi-modal fusion. Sensors are said to be
heterogeneous if their respective observation models cannot be described by the same
probability density function. Also, multiple sensor modalities tend to be dependent due to
non-linear cross-modal interactions. This dependence can be non-linear and even more
complex. Copula-based dependence modeling approach is a flexible parametric
characterization of the joint distribution of multivariate sensor observations. It addresses the
sensor heterogeneity and cross-modal non-linear dependence. We propose to design
copula-based optimal fusion rules for multi-modal inference problems.
Deliverable: 1) Many submissions to statistical signal processing conferences or TSP journal; 2)
Code and benchmarks used in the experiment.

Proposed budget: $50K/year for two years.

Performing site: Syracuse (PI: Pramod Varshney).

ID: S7

Title: Compressive Sensing for multimodal data

Abstract: We consider the problem of sparse signal reconstruction from compressed
measurements when the receiver has multiple measurement vectors. Most of the works in
compressed sensing with multiple measurement vectors assume that sparse signals have
non-zero elements in common. This assumption is valid when multiple sensors observe a
phenomenon with the same signal modality. In this work, we will consider that several sensors
observe the same phenomenon with signals from different modalities. These multimodal
signals do not share joint-sparse representation. However, these signals are expected to be
statistically dependent as they are observed from the same phenomenon. We seek to extend
the concept of multiple measurement vectors that leverages dependence structure among
signals from different modalities. We approach the problem in two different ways. First, we
model heterogeneous dependence among the multiple sparse signals using Copula functions.
Several copula functions model different dependencies among random variables and the one
that best represents the dependencies among multimodal sparse signals should be selected
during the reconstruction of the multiple sparse signals. Second, we will consider learning of
dependent structures among multimodal signals using generative model-based techniques.
Generative models can be used when there are enough data to learn the dependencies among
the sparse signals. We will exploit the learned dependencies during signal reconstruction and
enhance the reconstruction performance.
Deliverable: 1) Many submissions to statistical signal processing conferences or TSP journal; 2)
Code and benchmarks used in the experiment.

Proposed budget: $50K/year for two years.

Performing site: Syracuse (PI: Pramod Varshney).