Developing useful interfaces between brains and machines is a grand challenge of neuroengineering. An effective interface has the capacity to not only interpret neural signals, but predict the intentions of the human to perform an action in the near future; prediction is made even more challenging outside well-controlled laboratory experiments. This paper describes our approach to detect and to predict natural human arm movements in the future, a key challenge in brain computer interfacing that has never before been attempted. We introduce the novel Annotated Joints in Long-term ECoG (AJILE) dataset; AJILE includes automatically annotated poses of 7 upper body joints for four human subjects over 670 total hours (more than 72 million frames), along with the corresponding simultaneously acquired intracranial neural recordings. The size and scope of AJILE greatly exceeds all previous datasets with movements and electrocorticography (ECoG), making it possible to take a deep learning approach to movement prediction. We propose a multimodal model that combines deep convolutional neural networks (CNN) with long short-term memory (LSTM) blocks, leveraging both ECoG and video modalities. We demonstrate that our models are able to detect movements and predict future movements up to 800 msec before movement initiation. Further, our multimodal movement prediction models exhibit resilience to simulated ablation of input neural signals. We believe a multimodal approach to natural neural decoding that takes context into account is critical in advancing bioelectronic technologies and human neuroscience.

This paper proposes a novel approach for the analysis of movement and behavior of the Plainfin midshipman (Porichthys notatus) in the wild. It is based on underwater video recordings of the fish in their natural habitat taken inside their nests during reproductive months. During this time, alpha male Plainfin midshipmen rarely leave their nests as they are guarding their eggs, so the proposed approach addresses the issue of detecting subtle motion and nesting behavior as the fish remains relatively sedentary. To the best of our knowledge, this is the first paper to propose an automated method to analyze subtle movements of a highly territorial animal in its natural habitat.Motion detection uses the displacement of SURF (Interest point algorithm) key-point movements from frame to frame to analyze the amount of movement by the fish. Kmeans clustering and other outlier removal techniques are then used to differentiate fish motion from small moving objects in the background and foreground. The analysis of fish behavior uses similarity-based periodicity detection combined with the K-neighbors classifier. Experimental validation with respect to expert-annotated ground truth shows excellent performance for both motion and behavior detection approaches.