Journals

  • Regulation of arousal via online neurofeedback improves human performance in a demanding sensory-motor task

    Abstract Our state of arousal can significantly affect our ability to make optimal decisions, judgments, and actions in real-world dynamic environments. The Yerkes–Dodson law, which posits an inverse-U relationship between arousal and task performance, suggests that there is a state of arousal that is optimal for behavioral performance in a given task. Here we show that we can use online neurofeedback to shift an individual’s arousal from the right side of the Yerkes–Dodson curve to the left toward a state of improved performance. Specifically, we use a brain–computer interface (BCI) that uses information in the EEG to generate a neurofeedback signal that dynamically adjusts an individual’s arousal state when they are engaged in a boundary-avoidance task (BAT). The BAT is a demanding sensory-motor task paradigm that we implement as an aerial navigation task in virtual reality and which creates cognitive conditions that escalate arousal and quickly results in task failure (e.g., missing or crashing into the boundary). We demonstrate that task performance, measured as time and distance over which the subject can navigate before failure, is significantly increased when veridical neurofeedback is provided. Simultaneous measurements of pupil dilation and heart-rate variability show that the neurofeedback indeed reduces arousal. Our work demonstrates a BCI system that uses online neurofeedback to shift arousal state and increase task performance in accordance with the Yerkes–Dodson law.   Data available via IEEE DataPort: https://ieee-dataport.org/open-access/regulation-arousal-online-neurofeedback-improves-human-performance-demanding-sensory

  • Correlation of neural activity with behavioral kinematics reveals distinct sensory encoding and evidence accumulation processes during active tactile sensing

    March 29, 2018 | I. Delis, Jacek Dmochowski, Paul Sajda, Q. Wang

    Many real-world decisions rely on active sensing, a dynamic process for directing our sensors (e.g. eyes or fingers) across a stimulus to maximize information gain. Though ecologically pervasive, limited work has focused on identifying neural correlates of the active sensing process. In tactile perception, we often make decisions about an object/surface by actively exploring its shape/texture. Here we investigate the neural correlates of active tactile decision-making by simultaneously measuring electroencephalography (EEG) and finger kinematics while subjects interrogated a haptic surface to make perceptual judgments. Since sensorimotor behavior underlies decision formation in active sensing tasks, we hypothesized that the neural correlates of decision-related processes would be detectable by relating active sensing to neural activity. Novel brain-behavior correlation analysis revealed that three distinct EEG components, localizing to right-lateralized occipital cortex (LOC), middle frontal gyrus (MFG), and supplementary motor area (SMA), respectively, were coupled with active sensing as their activity significantly correlated with finger kinematics. To probe the functional role of these components, we fit their single-trial-couplings to decision-making performance using a hierarchical-drift-diffusion-model (HDDM), revealing that the LOC modulated the encoding of the tactile stimulus whereas the MFG predicted the rate of information integration towards a choice. Interestingly, the MFG disappeared from components uncovered from control subjects performing active sensing but not required to make perceptual decisions. By uncovering the neural correlates of distinct stimulus encoding and evidence accumulation processes, this study delineated, for the first time, the functional role of cortical areas in active tactile decision-making.

  • Compact Convolutional Neural Networks for Classification of Asynchronous Steady-state Visual Evoked Potentials

    March 19, 2018 | Nick Waytowich, V. Lawhern, J.O. Garcia, Jennifer A. Cummings, Josef Faller, Paul Sajda, J.M. Vettel

    Steady-State Visual Evoked Potentials (SSVEPs) are neural oscillations from the parietal and occipital regions of the brain that are evoked from flickering visual stimuli. SSVEPs are robust signals measurable in the electroencephalogram (EEG) and are commonly used in braincomputer interfaces (BCIs). However, methods for high-accuracy decoding of SSVEPs usually require hand-crafted approaches that leverage domain-specific knowledge of the stimulus signals, such as specific temporal frequencies in the visual stimuli and their relative spatial arrangement. When this knowledge is unavailable, such as when SSVEP signals are acquired asynchronously, such approaches tend to fail. In this paper, we show how a compact convolutional neural network (Compact-CNN), which only requires raw EEG signals for automatic feature extraction, can be used to decode signals from a 12-class SSVEP dataset without the need for any domain-specific knowledge or calibration data. We report across subject mean accuracy of approximately 80 % (chance being 8.3 %) and show this is substantially better than current state-of-the-art hand-crafted approaches using canonical correlation analysis (CCA) and Combined-CCA. Furthermore, we analyze our Compact-CNN to examine the underlying feature representation, discovering that the deep learner extracts additional phase and amplitude related features associated with the structure of the dataset. We discuss how our Compact-CNN shows promise for BCI applications that allow users to freely gaze/attend to any stimulus at any time (e.g., asynchronous BCI) as well as provides a method for analyzing SSVEP signals in a way that might augment our understanding about the basic processing in the visual cortex.

  • Tracking Deceased-Related Thinking With Neural Pattern Decoding of a Cortical-Basal Ganglia Circuit

    March 6, 2018 | Noam Schneck, Stefan Haufe, Tao Tu, George A. Bonanno, Kevin N. Ochsner, Paul Sajda, J. John Mann

    Background Deceased-related thinking is central to grieving and potentially critical to processing of the loss. Self-report measurements might fail to capture important elements of deceased-related thinking and processing. Here, we used a machine learning approach applied to functional magnetic resonance imaging—known as neural decoding—to develop a measure of ongoing deceased-related processing. Methods Twenty-three subjects grieving the loss of a first-degree relative, spouse, or partner within 14 months underwent two functional magnetic resonance imaging tasks. They first viewed pictures and stories related to the deceased, a living control person, and a demographic control figure while providing ongoing valence and arousal ratings. Second, they performed a 10-minute Sustained Attention to Response Task with thought probes every 25 to 35 seconds to identify deceased, living, and self-related thoughts. Results A conjunction analysis, controlling for valence/arousal, identified neural clusters in basal ganglia, orbital prefrontal cortex, and insula associated with both types of deceased-related stimuli versus the two control conditions in the first task. This pattern was applied to functional magnetic resonance imaging data collected during the Sustained Attention to Response Task and discriminated deceased-related but not living or self-related thoughts, independently of grief severity and time since loss. Deceased-related thoughts on the Sustained Attention to Response Task correlated with self-reported avoidance. The neural model predicted avoidance over and above deceased-related thoughts. Conclusions A neural pattern trained to identify mental representations of the deceased tracked deceased-related thinking during a sustained attention task and also predicted subject-level avoidance. This approach provides a new imaging tool to be used as an index of processing the deceased for future studies of complicated grief. Also note Commentary: “Can’t Stop Remembering: Neural Decoding of Representations of the Deceased Predicts Subsequent Intrusive Thinking and Coping Strategies”

  • Extracting multidimensional stimulus-response correlations using hybrid encoding-decoding of neural activity

    March 6, 2018 | Jacek Dmochowski, J. Ki, P. DeGuzman, Paul Sajda, L. Parra

    In neuroscience, stimulus-response relationships have traditionally been analyzed using either encoding or decoding models. Here we propose a hybrid approach that decomposes neural activity into multiple components, each representing a portion of the stimulus. The technique is implemented via canonical correlation analysis (CCA) by temporally filtering the stimulus (encoding) and spatially filtering the neural responses (decoding) such that the resulting components are maximally correlated. In contrast to existing methods, this approach recovers multiple correlated stimulus-response pairs, and thus affords a richer, multidimensional analysis of neural representations. We first validated the technique’s ability to recover multiple stimulus-driven components using electroencephalographic (EEG) data simulated with a finite element model of the head. We then applied the technique to real EEG responses to auditory and audiovisual narratives experienced identically across subjects, as well as uniquely experienced video game play. During narratives, both auditory and visual stimulus-response correlations (SRC) were modulated by attention and tracked inter-subject correlations. During video game play, SRC varied with game difficulty and the presence of a dual task. Interestingly, the strongest component extracted for visual and auditory features of film clips had nearly identical spatial distributions, suggesting that the predominant encephalographic response to naturalistic stimuli is supramodal. The diversity of these findings demonstrates the utility of measuring multidimensional SRC via hybrid encoding-decoding.

  • Combined EEG/TMS/fMRI studies asking whether phase matters

    March 6, 2018 | MS George, Truman R. Brown, Jordan Muraskin, GT Saber, J Doose, H Moss, Robin Goldman, Paul Sajda

    The effects of TMS have been largely studied in the motor system, using motor evoked potentials (MEP) as the output. There is a large between-pulse variability in MEP’s, requiring multiple pulses and averaging. One potential source of the MEP variability might be the EEG phase cycle of the motor cortex relative to the TMS pulse. We have succeeded in performing TMS in the fMRI scanner with high fidelity EEG. As the first step in understanding whether the phase of a brain region matters with respect to the effect of a TMS pulse, we have started in motor system, using local and secondary BOLD response, and assessing whether the local alpha phase influences the TMS pulse impact.

  • A multimodal encoding model applied to imaging decision-related neural cascades in the human brain

    Abstract Perception and cognition in the brain are naturally characterized as spatiotemporal processes. Decision-making, for example, depends on coordinated patterns of neural activity cascading across the brain, running in time from stimulus to response and in space from primary sensory regions to the frontal lobe. Measuring this cascade is key to developing an understanding of brain function. Here we report on a novel methodology that employs multi-modal imaging for inferring this cascade in humans at unprecedented spatiotemporal resolution. Specifically, we develop an encoding model to link simultaneously measured electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) signals to infer high-resolution spatiotemporal brain dynamics during a perceptual decision. After demonstrating replication of results from the literature, we report previously unobserved sequential reactivation of a substantial fraction of the pre-response network whose magnitude correlates with a proxy for decision confidence. Our encoding model, which temporally tags BOLD activations using time localized EEG variability, identifies a coordinated and spatially distributed neural cascade that is associated with a perceptual decision. In general the methodology illuminates complex brain dynamics that would otherwise be unobservable using fMRI or EEG acquired separately.

  • A comparison of single-trial EEG classification and EEGinformed fMRI across three MR compatible EEG recording systems

    Simultaneously recorded electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) can be used to non-invasively measure the spatiotemporal dynamics of the human brain. One challenge is dealing with the artifacts that each modality introduces into the other when the two are recorded concurrently, for example the ballistocardiogram (BCG). We conducted a preliminary comparison of three different MR compatible EEG recording systems and assessed their performance in terms of single-trial classification of the EEG when simultaneously collecting fMRI. We found tradeoffs across all three systems, for example varied ease of setup and improved classification accuracy with reference electrodes (REF) but not for pulse artifact subtraction (PAS) or reference layer adaptive filtering (RLAF).

  • Attentional Bias to Reminders of the Deceased as Compared to a Living Attachment in Grieving

    March 6, 2018 | Noam Schneck, Tao Tu, C. A. Michel, George A. Bonanno, Paul Sajda, J. John Mann

    Background: Grieving individuals demonstrate attentional bias towards reminders of the deceased vs. neutral stimuli. We sought to assess bias towards reminders of the deceased vs. a living-attachment figure and to evaluate similarities and differences in the neural correlates of deceased and living-related attention. We also sought to identify grief process variables associated with deceased-related attentional bias. Methods: Twenty-five subjects grieving the death of a first-degree relative or partner within 14-months performed an emotional Stroop task, using words related to a deceased or a living attachment figure, and a standard Stroop task, to identify general selective attention, during functional magnetic resonance imaging (fMRI). Subjects rated word-sadness, complicated grief (CG) symptoms, depression severity, attachment style, emotional pain, nonacceptance, yearning and intrusions. Results: We identified an attentional bias to deceased vs. living-related words, independent of age, depression severity/history, loss-type, word-sadness, medication use and time since loss. Attentional bias correlated with CG severity and intrusive thinking. A conjunction analysis identified joint activation in fusiform gyrus, posterior cingulate and temporal parietal junction across living and deceased-related attention vs. general selective attention. Insecure-avoidant attachment style correlated with decreased engagement of this network in deceased-related attention. Conclusions: We have demonstrated an attentional bias to reminders of the deceased vs. a living attachment in grieving. Overlapping neural circuits related to living and deceased related attention suggests that the bereaved employ similar processes in attending to the deceased as they do in attending to the living. Deceased-related attentional bias appears to be linked primarily to intrusive thinking about the loss.

  • SigViewer: Visualizing Multimodal Signals Stored in XDF (Extensible Data Format) Files*

    March 6, 2018 | Yida Lin, Clemens Brunner, Paul Sajda, Josef Faller

    Multimodal biosignal acquisition is facilitated by recently introduced software solutions such as Lab Streaming Layer (LSL) and its associated data format XDF (Extensible Data Format). However, there are no stand-alone applications that can visualize multimodal time series stored in XDF files. We extended SigViewer, an open source cross-platform Qt C++ application with the capability of loading, resampling, annotating, and visualizing signals stored in XDF files and successfully applied the tool for post-hoc visual verification of the accuracy of a system that aims to predict the phase of alpha oscillations within the electroencephalogram in real-time.

  • Network configurations in the human brain reflect choice-bias during rapid face processing

    Network interactions are likely to be instrumental in processes underlying rapid perception and cognition. Specifically, high-level and perceptual regions must interact to balance pre-existing models of the environment with new incoming stimuli. Simultaneous EEG/fMRI allows for the temporal characterization of brain-network interactions combined with improved anatomical localization of regional activity. In this paper we use simultaneous EEG/fMRI and multivariate dynamical systems (MDS) analysis to characterize network relationships between constitute brain areas that reflect a subject’s choice for a face versus non-face categorization task. Our simultaneous EEG and fMRI analysis on 21 human subjects (12 males, 9 females) identifies early perceptual and late frontal subsystems that are selective to the categorical choice of faces versus non-faces. We analyze the interactions between these subsystems using a MDS in the space of the BOLD signal. Our main findings show that differences between face choice and house choice networks are seen in the network interactions between the early and late subsystems, and that the magnitude of the difference in network interaction positively correlates with the behavioral false positive rate of face choices. We interpret this to reflect the role of saliency and expectations likely encoded in frontal “late” regions on perceptual processes occurring in “early” perceptual regions.

  • Maternal serum α-fetoprotein screening: III. Pitfalls in evaluating black gravid women

    January 18, 2017 | J. N. Macri, R. V. Kasturi, M. G. Hu, D. A. Krantz, T. J. Douros, Paul Sajda, E. J. Cook

    Maternal serum α-fetoprotein levels are higher in black women. Misinterpretation of maternal serum α-fetoprotein screening results can subject black gravid women to unnecessary invasive diagnostic procedures and their calculable risks. Screening errors for black women can result from the use of normative data bases established with maternal serum samples drawn from other racial groups or the use of such data bases in conjunction with a published correction factor. Because the incidence of open neural tube defects is lower for blacks than for others, excessive false positive results for blacks (estimated to be 8817 to 28,215 cases annually) would be a pernicious misapplication of maternal serum α-fetoprotein screening. We address the problem outlined above and recommend independently developed, valid, normative data bases.

  • Extraction of Depth-From-Occlusion by a Physiologically-Based Network

    January 18, 2017 | Paul Sajda, L. H. Finkel
  • A hierarchical neural network architecture that learns target context: applications to digital mammography

    January 17, 2017 | Paul Sajda, C. Spence, J. Pearson

    An important problem in image analysis is finding small objects in large images. The problem is challenging because: 1) searching a large image is computationally expensive; and 2) small targets (on the order of a few pixels in size) have relatively few distinctive features which enable them to be distinguished from non-targets. To overcome these challenges the authors have developed a hierarchical neural network architecture which combines multiresolution pyramid processing with neural networks. Here the authors discuss the application of their hierarchical neural network architecture to the problem of detecting microcalcifications in digital mammograms. Microcalcifications are cues for breast tumors. 30% to 50% of breast carcinomas have microcalcifications visible in mammograms while 60% to 80% of all breast tumors eventually show microcalcifications via histology. Similar to the building/ATR problem, microcalcifications are generally very small point-like objects (>10 pixels in mammograms) which are hard to detect. Radiologists must often exploit other information in the imagery (e.g. location of blood vessels, ducts, etc.) in order to detect these microcalcifications. Here the authors examine how well their hierarchical neural network architecture learns and exploits contextual information in mammograms.

  • Integrating multi-resolution and contextual information for improved microcalcification detection in CAD

    January 17, 2017 | Paul Sajda, C. Spence, J. Pearson, RM Nishikawa
  • Dealing with position uncertainty when training an image search system

    January 17, 2017 | C. Spence, Paul Sajda

    We formulate an error function for the supervised learning of image search/detection tasks when the positions of the objects to be found are uncertain or ill-defined. The need for this uncertain object position (UOP) error function arises in at least two ways. First, point-like objects frequently have positions that are inaccurately specified. We illustrate this with the problem of detecting microcalcifications in mammograms. The second type of position uncertainty occurs with extended objects whose boundaries are not accurately defined. In this case we usually only need the detector to respond at one pixel within each object. As an example of this, we present results for neural networks trained to detect clusters of buildings in aerial photographs. We are currently applying the UOP error function to the detection of masses in mammograms, which also have poorly-defined boundaries. In all of these examples, neural networks trained with the UOP error function perform much better than networks trained with the conventional cross-entropy error function.

  • Large scale observer study to measure the benefits of computer-aided detection to screening mammography

    January 17, 2017 | RM Nishikawa, C. Gatsonis, M.D. Schnall, M.L. Giger, Paul Sajda, M. Chen
  • Response error correction-a demonstration of improved human-machine performance using real-time EEG monitoring

    January 17, 2017 | L. Parra, C. Spence, Adam Gerson, Paul Sajda

    We describe a brain–computer interface (BCI) system, which uses a set of adaptive linear preprocessing and classification algorithms for single-trial detection of error related negativity (ERN). We use the detected ERN as an estimate of a subject’s perceived error during an alternative forced choice visual discrimination task. The detected ERN is used to correct subject errors. Our initial results show average improvement in subject performance of 21% when errors are automatically corrected via the BCI. We are currently investigating the generalization of the overall approach to other tasks and stimulus paradigms.

  • A multi-scale probabilistic network model for detection, synthesis and compression in mammographic image analysis

    January 17, 2017 | Paul Sajda, C. Spence, L. Parra

    We develop a probabilistic network model over image spaces and demonstrate its broad utility in mammographic image analysis, particularly with respect to computer-aided diagnosis. The model employs a multi-scale pyramid decomposition to factor images across scale and a network of tree-structured hidden variables to capture long-range spatial dependencies. This factoring makes the computation of the density functions local and tractable. The result is a hierarchical mixture of conditional probabilities, similar to a hidden Markov model on a tree. The model parameters are found with maximum likelihood estimation using the expectation-maximization algorithm. The utility of the model is demonstrated for three applications: (1) detection of mammographic masses for computer-aided diagnosis; (2) qualitative assessment of model structure through mammographic synthesis; and (3) compression of mammographic regions of interest.

  • Multiresolution hierarchical blind recovery of biochemical markers of brain cancer in MRSI

    January 17, 2017 | S. Du, Paul Sajda, X. Mao, D.C. Shungu

    We present a multi-resolution hierarchical application of the constrained non-negative matrix factorization (cNMF) algorithm for blindly recovering constituent source spectra in magnetic resonance spectroscopic imaging (MRSI). cNMF is an extension of non-negative matrix factorization (NMF) that includes a positivity constraint on amplitudes of recovered spectra. We apply cNMF hierarchically, with spectral recovery and subspace reduction constraining which observations are used in the next level of processing. The decomposition model recovers physically meaningful spectra which are highly tissue-specific, for example spectra indicative of tumor proliferation, given a processing hierarchy that proceeds coarse-to-fine. We demonstrate the decomposition procedure on /sup 1/H long TE brain MRS data. The results show recovery of markers for normal brain tissue, low proliferative tissue and highly proliferative tissue. The coarse-to-fine hierarchy also makes the algorithm computationally efficient, thus it is potentially well-suited for use in diagnostic work-up.

  • Nonnegative matrix factorization for rapid recovery of constituent spectra in magnetic resonance chemical shift imaging of the brain

    January 17, 2017 | Paul Sajda, S. Du, Truman R. Brown, R. Stoyanova, D.C. Shungu, X. Mao, L. Parra

    We present an algorithm for blindly recovering constituent source spectra from magnetic resonance (MR) chemical shift imaging (CSI) of the human brain. The algorithm, which we call constrained nonnegative matrix factorization (cNMF), does not enforce independence or sparsity, instead only requiring the source and mixing matrices to be nonnegative. It is based on the nonnegative matrix factorization (NMF) algorithm, extending it to include a constraint on the positivity of the amplitudes of the recovered spectra. This constraint enables recovery of physically meaningful spectra even in the presence of noise that causes a significant number of the observation amplitudes to be negative. We demonstrate and characterize the algorithm’s performance using /sup 31/P volumetric brain data, comparing the results with two different blind source separation methods: Bayesian spectral decomposition (BSD) and nonnegative sparse coding (NNSC). We then incorporate the cNMF algorithm into a hierarchical decomposition framework, showing that it can be used to recover tissue-specific spectra given a processing hierarchy that proceeds coarse-to-fine. We demonstrate the hierarchical procedure on /sup 1/H brain data and conclude that the computational efficiency of the algorithm makes it well-suited for use in diagnostic work-up.

  • Guest editorial: Special issue on blind source separation and de-convolution in imaging and image processing

    January 17, 2017 | Paul Sajda, Y. Zeevi
  • A 3D morphological analysis of trabecular bone based on individual trabeculae segmentation

    January 17, 2017 | X. S. Liu, P. K. Saha, F. W. Wehrli, Paul Sajda, X. E. Guo
  • Quantification of the roles of trabecular microarchitecture and trabecular type in determining the elastic modulus of human trabecular bone

    January 17, 2017 | X. S. Liu, Paul Sajda, P. K. Saha, F. W. Wehrli, X. E. Guo

    The roles of microarchitecture and types of trabeculae in determining elastic modulus of trabecular bone have been studied in μCT images of 29 trabecular bone samples by comparing their Young’s moduli calculated by finite element analysis (FEA) with different trabecular type-specific reconstructions. The results suggest that trabecular plates play an essential role in determining elastic properties of trabecular bone.

  • HiRes—a tool for comprehensive assessment and interpretation of metabolomic data

    January 17, 2017 | Q. Zhao, R. Stoyanova, S. Du, Paul Sajda, Truman R. Brown

    The increasing role of metabolomics in system biology is driving the development of tools for comprehensive analysis of high-resolution NMR spectral datasets. This task is quite challenging since unlike the datasets resulting from other ‘omics’, a substantial preprocessing of the data is needed to allow successful identification of spectral patterns associated with relevant biological variability. HiRes is a unique stand-alone software tool that combines standard NMR spectral processing functionalities with techniques for multi-spectral dataset analysis, such as principal component analysis and nonnegative matrix factorization. In addition, HiRes contains extensive abilities for data cleansing, such as baseline correction, solvent peak suppression, removal of frequency shifts owing to experimental conditions as well as auxiliary information management. Integration of these components together with multivariate analytical procedures makes HiRes very capable of addressing the challenges for assessment and interpretation of large metabolomic datasets, greatly simplifying this otherwise lengthy and difficult process and assuring optimal information retrieval.

  • Automated tissue segmentation and blind recovery of 1H MRS imaging spectral patterns of normal and diseased human brain

    January 13, 2017 | S. Du, X. Mao, Paul Sajda, D.C. Shungu

    Constrained non-negative matrix factorization (cNMF) with iterative data selection is described and demonstrated as a data analysis method for fast and automatic recovery of biochemically meaningful and diagnostically specific spectral patterns of the human brain from 1H MRS imaging (1H MRSI) data. To achieve this goal, cNMF decomposes in vivo multidimensional 1H MRSI data into two non-negative matrices representing (a) the underlying tissue-specific spectral patterns and (b) the spatial distribution of the corresponding metabolite concentrations. Central to the proposed approach is automatic iterative data selection which uses prior knowledge about the spatial distribution of the spectra to remove voxels that are due to artifacts and undesired metabolites/tissues such as the strong lipid and water components. The automatic recovery of diagnostic spectral patterns is demonstrated for long-TE1H MRSI data on normal human brain, multiple sclerosis, and serial brain tumor. The results show the ability of cNMF with iterative data selection to automatically and simultaneously recover tissue-specific spectral patterns and achieve segmentation of normal and diseased human brain tissue, concomitant with simplification of information content. These features of cNMF, which permit rapid recovery, reduction and interpretation of the complex diagnostic information content of large multi-dimensional spectroscopic imaging data sets, have the potential to enhance the clinical utility of in vivo1H MRSI.

  • Brain-computer interfaces

    January 13, 2017 | Paul Sajda, K-R Mueller, K. V. Shenoy

    The human brain is perhaps the most fascinating and complex signal processing machine in existence. It is capable of transducing a variety of environmental signals (the senses, including taste, touch, smell, sound, and sight) and extracting information from these disparate signal streams, ultimately fusing this information to enable behavior, cognition, and action. What is perhaps surprising is that the basic signal processing elements of the brain, i.e., neurons, transmit information at a relatively slow rate compared to transistors, switching about 106 times slower in fact. The brain has the advantage of having a tremendous number of neurons, all operating in parallel, and a highly distributed memory system of synapses (over 100 trillion in the cerebral cortex) and thus its signal processing capabilities may largely arise from its unique architecture. These facts have inspired a great deal of study of the brain from a signal processing perspective. Recently, scientists and engineers have focused on developing means in which to directly interface with the brain, essentially measuring neural signals and decoding them to augment and emulate behavior. This research area has been termed brain computer interfaces and is the topic of this issue of IEEE Signal Processing Magazine.

  • Complete volumetric decomposition of individual trabecular plates and rods and its morphological correlations with anisotropic elastic moduli in human trabecular bone

    January 13, 2017 | X. S. Liu, Paul Sajda, P. K. Saha, F. W. Wehrli, G. Bevill, T. M. Keaveny, X. E. Guo

    Trabecular plates and rods are important microarchitectural features in determining mechanical properties of trabecular bone. A complete volumetric decomposition of individual trabecular plates and rods was used to assess the orientation and morphology of 71 human trabecular bone samples. The ITS-based morphological analyses better characterize microarchitecture and help predict anisotropic mechanical properties of trabecular bone. Introduction: Standard morphological analyses of trabecular architecture lack explicit segmentations of individual trabecular plates and rods. In this study, a complete volumetric decomposition technique was developed to segment trabecular bone microstructure into individual plates and rods. Contributions of trabecular type-associated morphological parameters to the anisotropic elastic moduli of trabecular bone were studied. Materials and Methods: Seventy-one human trabecular bone samples from the femoral neck (FN), tibia, and vertebral body (VB) were imaged using μCT or serial milling. Complete volumetric decomposition was applied to segment trabecular bone microstructure into individual plates and rods. The orientation of each individual trabecula was determined, and the axial bone volume fractions (aBV/TV), axially aligned bone volume fraction along each orthotropic axis, were correlated with the elastic moduli. The microstructural type-associated morphological parameters were derived and compared with standard morphological parameters. Their contributions to the anisotropic elastic moduli, calculated by finite element analysis (FEA), were evaluated and compared. Results: The distribution of trabecular orientation suggested that longitudinal plates and transverse rods dominate at all three anatomic sites. aBV/TV along each axis, in general, showed a better correlation with the axial elastic modulus (r2 = 0.95∼0.99) compared with BV/TV (r2 = 0.93∼0.94). The plate-associated morphological parameters generally showed higher correlations with the corresponding standard morphological parameters than the rod-associated parameters. Multiple linear regression models of six elastic moduli with individual trabeculae segmentation (ITS)-based morphological parameters (adjusted r2 = 0.95∼0.98) performed equally well as those with standard morphological parameters (adjusted r2 = 0.94∼0.97) but revealed specific contributions from individual trabecular plates or rods. Conclusions: The ITS-based morphological analyses provide a better characterization of the morphology and trabecular orientation of trabecular bone. The axial loading of trabecular bone is mainly sustained by the axially aligned trabecular bone volume. Results suggest that trabecular plates dominate the overall elastic properties of trabecular bone.

  • Dynamic simulation of three dimensional architectural and mechanical alterations in human trabecular bone during menopause

    January 13, 2017 | X. S. Liu, A. H. Huang, X. Henry Zhang, Paul Sajda, B. Ji, X. E. Guo

    A three dimensional (3D) computational simulation of dynamic process of trabecular bone remodeling was developed with all the parameters derived from physiological and clinical data. Contributions of the microstructural bone formation deficits: trabecular plate perforations, trabecular rod breakages, and isolated bone fragments, to the rapid bone loss and disruption of trabecular microarchitecture during menopause were studied. Eighteen human trabecular bone samples from femoral neck (FN) and spine were scanned using a micro computed tomography (μCT) system. Bone resorption and formation were simulated as a computational cycle corresponding to 40-day resorption/160-day formation. Resorption cavities were randomly created over the bone surface according to the activation frequency, which was strictly based on clinical data. Every resorption cavity was refilled during formation unless it caused trabecular plate perforation, trabecular rod breakage or isolated fragments. A 20-year-period starting 5 years before and ending 15 years after menopause was simulated for each specimen. Elastic moduli, standard and individual trabeculae segmentation (ITS)-based morphological parameters were evaluated for each simulated 3D image. For both spine and FN groups, the time courses of predicted bone loss pattern by microstructural bone formation deficits were fairly consistent with the clinical measurements. The percentage of bone loss due to trabecular plate perforation, trabecular rod breakage, and isolated bone fragments were 73.2%, 18.9% and 7.9% at the simulated 15 years after menopause. The ITS-based plate fraction (pBV/BV), mean plate surface area (pTb.S), plate number density (pTb.N), and mean rod thickness (rTb.Th) decreased while rod fraction (rBV/BV) and rod number density (rTb.N) increased after the simulated menopause. The dynamic bone remodeling simulation based on microstructural bone formation deficits predicted the time course of menopausal bone loss pattern of spine and FN. Microstructural plate perforation could be the primary cause of menopausal trabecular bone loss. The combined effect of trabeculae perforation, breakage, and isolated fragments resulted in fewer and smaller trabecular plates and more but thinner trabecular rods.

  • Micromechanical analyses of vertebral trabecular bone based on individual trabeculae segmentation of plates and rods

    January 13, 2017 | X. S. Liu, G. Bevill, T. M. Keaveny, Paul Sajda, X. E. Guo

    Trabecular plates play an important role in determining elastic moduli of trabecular bone. However, the relative contribution of trabecular plates and rods to strength behavior is still not clear. In this study, individual trabeculae segmentation (ITS) and nonlinear finite element (FE) analyses were used to evaluate the roles of trabecular types and orientations in the failure initiation and progression in human vertebral trabecular bone. Fifteen human vertebral trabecular bone samples were imaged using micro computed tomography (μCT), and segmented using ITS into individual plates and rods by orientation (longitudinal, oblique, and transverse). Nonlinear FE analysis was conducted to perform a compression simulation for each sample up to 1% apparent strain. The apparent and relative trabecular number and tissue fraction of failed trabecular plates and rods were recorded during loading and data were stratified by trabecular orientation. More trabecular rods (both in number and tissue fraction) failed at the initiation of compression (0.1–0.2% apparent strain) while more plates failed around the apparent yield point (>0.7% apparent strain). A significant correlation between plate bone volume fraction (pBV/TV) and apparent yield strength was found (r2=0.85). From 0.3% to 1% apparent strain, significantly more longitudinal trabecular plate and transverse rod failed than other types of trabeculae. While failure initiates at rods and rods fail disproportionally to their number, plates contribute significantly to the apparent yield strength because of their larger number and tissue volume. The relative failed number and tissue fraction at apparent yield point indicate homogeneous local failure in plates and rods of different orientations.

  • Maximum likelihood in cost-sensitive learning: Model specification, approximations, and upper bounds

    January 13, 2017 | Jacek Dmochowski, L. Parra, Paul Sajda

    The presence of asymmetry in the misclassification costs or class prevalences is a common occurrence in the pattern classification domain. While much interest has been devoted to the study of cost-sensitive learning techniques, the relationship between cost-sensitive learning and the specification of the model set in a parametric estimation framework remains somewhat unclear. To that end, we differentiate between the case of the model including the true posterior, and that in which the model is misspecified. In the former case, it is shown that thresholding the maximum likelihood (ML) estimate is an asymptotically optimal solution to the risk minimization problem. On the other hand, under model misspecification, it is demonstrated that thresholded ML is suboptimal and that the risk-minimizing solution varies with the misclassification cost ratio. Moreover, we analytically show that the negative weighted log likelihood (Elkan, 2001) is a tight, convex upper bound of the empirical loss. Coupled with empirical results on several real-world data sets, we argue that weighted ML is the preferred cost-sensitive technique.

  • Accuracy of high-resolution in vivo micro magnetic resonance imaging for measurements of microstructural and mechanical properties of human distal tibial bone

    January 12, 2017 | X. S. Liu, X. Henry Zhang, C. S. Rajapakse, M. J. Wald, J. Magland, K. K. Sekhon, M. F. Adam, Paul Sajda, F. W. Wehrli, X. E. Guo

    Micro magnetic resonance imaging (µMRI) is an in vivo imaging method that permits 3D quantification of cortical and trabecular bone microstructure. µMR images can also be used for building microstructural finite element (µFE) models to assess bone stiffness, which highly correlates with bone’s resistance to fractures. In order for µMRI-based microstructural and µFE analyses to become standard clinical tools for assessing bone quality, validation with a current gold standard, namely, high-resolution micro computed tomography (µCT), is required. Microstructural measurements of 25 human cadaveric distal tibias were performed for the registered µMR and µCT images, respectively. Next, whole bone stiffness, trabecular bone stiffness, and elastic moduli of cubic subvolumes of trabecular bone in both µMR and µCT images were determined by voxel-based µFE analysis. The bone volume fraction (BV/TV), trabecular number (Tb.N*), trabecular spacing (Tb.Sp*), cortical thickness (Ct.Th), and structure model index (SMI) based on µMRI showed strong correlations with µCT measurements (r2 = 0.67 to 0.97), and bone surface-to-volume ratio (BS/BV), connectivity density (Conn.D), and degree of anisotropy (DA) had significant but moderate correlations (r2 = 0.33 to 0.51). Each of these measurements also contributed to one or many of the µFE-predicted mechanical properties. However, model-independent trabecular thickness (Tb.Th*) based on µMRI had no correlation with the µCT measurement and did not contribute to any mechanical measurement. Furthermore, the whole bone and trabecular bone stiffness based on µMRI were highly correlated with those of µCT images (r2 = 0.86 and 0.96), suggesting that µMRI-based µFE analyses can directly and accurately quantify whole bone mechanical competence. In contrast, the elastic moduli of the µMRI trabecular bone subvolume had significant but only moderate correlations with their gold standards (r2 = 0.40 to 0.58). We conclude that most microstructural and mechanical properties of the distal tibia can be derived efficiently from µMR images and can provide additional information regarding bone quality. © 2010 American Society for Bone and Mineral Research.

  • Linking neuronal variability to perceptual decision making via neuroimaging

    January 12, 2017 | Paul Sajda, Marios Philiastides, Hauke R. Heekeren, R. Ratcliff
  • Guest Editorial Special Issue on Brain–Machine Interface

    January 12, 2017 | M. Sawan, P. Mohseni, Paul Sajda, J. C. Sanchez

    The 19 papers in this special issue cover a variety of brain-machine/computer interfaces and include: 1) monitoring and data recording methods, 2) spike-based detection techniques, 3) adaptive/automatic biosignal processing approaches, 4) bioamplifiers implementation and validation, 5) ambulatory/implantable electronics, 6) electronics for brain science, and vision-based interfaces, 7) electrical stimulation, and 8) experimental and clinical case studies.

  • Spatiotemporal properties of intracellular calcium signaling in osteocytic and osteoblastic cell networks under fluid flow

    January 12, 2017 | D. Jing, X. Lu, E. Luo, Paul Sajda, P. L. Leong, X. E. Guo

    Mechanical stimuli can trigger intracellular calcium (Ca2 +) responses in osteocytes and osteoblasts. Successful construction of bone cell networks necessitates more elaborate and systematic analysis for the spatiotemporal properties of Ca2 + signaling in the networks. In the present study, an unsupervised algorithm based on independent component analysis (ICA) was employed to extract the Ca2 + signals of bone cells in the network. We demonstrated that the ICA-based technology could yield higher signal fidelity than the manual region of interest (ROI) method. Second, the spatiotemporal properties of Ca2 + signaling in osteocyte-like MLO-Y4 and osteoblast-like MC3T3-E1 cell networks under laminar and steady fluid flow stimulation were systematically analyzed and compared. MLO-Y4 cells exhibited much more active Ca2 + transients than MC3T3-E1 cells, evidenced by more Ca2 + peaks, less time to the 1st peak and less time between the 1st and 2nd peaks. With respect to temporal properties, MLO-Y4 cells demonstrated higher spike rate and Ca2 + oscillating frequency. The spatial intercellular synchronous activities of Ca2 + signaling in MLO-Y4 cell networks were higher than those in MC3T3-E1 cell networks and also negatively correlated with the intercellular distance, revealing faster Ca2 + wave propagation in MLO-Y4 cell networks. Our findings show that the unsupervised ICA-based technique results in more sensitive and quantitative signal extraction than traditional ROI analysis, with the potential to be widely employed in Ca2 + signaling extraction in the cell networks. The present study also revealed a dramatic spatiotemporal difference in Ca2 + signaling for osteocytic and osteoblastic cell networks in processing the mechanical stimulus. The higher intracellular Ca2 + oscillatory behaviors and intercellular coordination of MLO-Y4 cells provided further evidences that osteocytes may behave as the major mechanical sensor in bone modeling and remodeling processes.

  • Towards Serious Games for Improved BCI

    January 10, 2017 | B. Lance, J. Touryan, Y-K Wang, S-W Lu, C-H Chuang, P. Khooshabeh, Paul Sajda, A. R. Marathe, T-P Jung, Chin-Teng Lin, K. McDowell

    Brain-computer interface (BCI) technologies, or technologies that use online brain signal processing, have a great promise to improve human interactions with computers, their environment, and even other humans. Despite this promise, there are no current serious BCI technologies in widespread use, due to the lack of robustness in BCI technologies. The key neural aspect of this lack of robustness is human variability, which has two main components: (1) individual differences in neural signals and (2) intraindividual variability over time. In order to develop widespread BCI technologies, it will be necessary to address this lack of robustness. However, it is currently unknown how neural variability affects BCI performance. To accomplish these goals, it is essential to obtain data from large numbers of individuals using BCI technologies over considerable lengths of time. One promising method for this is through the use of BCI technologies embedded into games with a purpose (GWAP). GWAP are a game-based form of crowdsourcing which players choose to play for enjoyment and during which the player performs key tasks which cannot be automated but that are required to solve research questions. By embedding BCI paradigms in GWAP and recording neural and behavioral data, it should be possible to much more clearly understand the differences in neural signals between individuals and across different time scales, enabling the development of novel and increasingly robust adaptive BCI algorithms.

  • Ex Vivo Hyperspectral Autofluorescence (AF) Imaging of Retinal Pigment Epithelium (RPE) in Human Donor Eyes with Age-related Macular Degeneration (AMD)

    January 10, 2017 | T. B. Ami, Y. Tong, Paul Sajda, Z. Ablonczy, C. Curcio, T. Smith, T. Ach

    Purpose We have shown how hyperspectral imaging can extract individual signals of abundant fluorophores ex-vivo in normal RPE [1]. Herein we analyze donor eyes with AMD, hypothesizing that the spectral signatures would vary with perturbations in RPE physiology. Methods Hyperspectral AF images were captured from 7 locations in 5 RPE/Bruch’s-membrane (BrM) flat-mounts from donor eyes with early to late AMD. Imaging was performed at 2 excitation bands, 436-460 and 480-510 nm, with emission captured between 420 and 720 nm in 10 nm intervals using a Nuance FX camera. Results Gaussian mixture modeling and mathematical factorization were applied to extract 1 BrM spectrum and 4 abundant emission spectra from RPE organelles, the latter peaking at mean wavelengths of 519±7, 574±8, 599±4, and 644±10 nm (436 nm excitation). The 519 nm peak was blue-shifted ~50 nm relative to the corresponding signal from normal eyes [1]. Spatial abundance images showed unique signals localized to RPE granule aggregates, melanosomes, and basal laminar deposits (BLamD) (Figure). Conclusions In AMD, some AF signals from RPE become spatially discrete [2], and a morphological diversity of RPE granule populations is well demonstrated using hyperspectral imaging. Differences in spectra and their spatial distributions between normal and AMD eyes may extend our knowledge of RPE pathophysiology in AMD. 1. Smith, Post, Johri, Lee, Ablonczy, Curcio, Ach, Sajda: Hyperspectral signal recovery of unknown fluorophores in the RPE. Biomed Opt Express 2014. 2. Ach, ARVO 2014

  • Hyperspectral Autofluorescence (AF) Characterization of Drusen in Donor Eyes with Age-Related Macular Degeneration (AMD)

    January 10, 2017 | Y. Tong, T. B. Ami, A. Johri, R. Post, Z. Ablonczy, Paul Sajda, C. Curcio, T. Ach, T. Smith

    Purpose Drusen are one of the hallmark lesions of AMD. Our purpose is to characterize drusen by their spectral AF signatures, for a better molecular understanding of AMD. Methods Retinal pigment epithelium (RPE)/Bruch’s membrane (BrM)-flatmounts were prepared from 5 donors with AMD. Hyperspectral AF imaging was performed at 11 locations containing drusen at 2 excitation bands, 436-460 and 480-510 nm, with emissions captured between 420 and 700 nm in 10 nm intervals, with the Nuance FX camera. Abundant individual spectra with corresponding spatial localizations were recovered with custom software based on a nonnegative tensor factorization algorithm [1]. Results In a typical example the original AF image (RGB) is presented with 5 spatial abundances (Fig. 1) of 5 recovered individual spectra (Fig. 2). In particular, spectrum C4 (Fig. 2, cyan line) localizes precisely and specifically to the drusen and also a diffuse region (upper left Fig. 1, abundance C4) with a peak wavelength of 510 nm. A spectrum with this characteristic peak and shape was recovered from all drusen and variably from diffuse regions in 10 of the 11 locations, with one spectrum peaking at 490 nm. Further, the dominant fluorophore (spectrum C1 and abundance C1 herein) consistently shows moderate co-localization with drusen, indicating this fluorophore overlying as granules and/or within drusen. This fluorophore spectrum is remarkably blue-shifted about 50 nm from the corresponding dominant fluorophore spectrum recovered from previously reported normal donors [1]. Conclusions With hyperspectral AF image analysis, we found a single spectral AF signature for drusen and diffuse sub-RPE deposits in AMD that appears to be highly sensitive for drusen. The possibility that this signal also localizes to basal linear deposit is being addressed. A second drusen-associated AF spectral signature shares commonalities with RPE lipofuscin. These hyperspectral AF characterizations of drusen may aid the molecular understanding of AMD.

  • Imaging decision-related neural cascades in the human brain

    January 10, 2017 | Jordan Muraskin, Truman R. Brown, Jennifer Walz, Bryan Conroy, Robin Goldman, Paul Sajda

    Perceptual decisions depend on coordinated patterns of neural activity cascading across the brain, running in time from stimulus to response and in space from primary sensory regions to the frontal lobe. Measuring this cascade and how it flows through the brain is key to developing an understanding of how our brains function. However observing, let alone understanding, this cascade, particularly in humans, is challenging. Here, we report a significant methodological advance allowing this observation in humans at unprecedented spatiotemporal resolution. We use a novel encoding model to link simultaneously measured electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) signals to infer the high-resolution spatiotemporal brain dynamics taking place during rapid visual perceptual decision-making. After demonstrating the methodology replicates past results, we show that it uncovers a previously unobserved sequential reactivation of a substantial fraction of the pre-response network whose magnitude correlates with decision confidence. Our results illustrate that a temporally coordinated and spatially distributed neural cascade underlies perceptual decision-making, with our methodology illuminating complex brain dynamics that would otherwise be unobservable using conventional fMRI or EEG separately. We expect this methodology to be useful in observing brain dynamics in a wide range of other mental processes.

  • Multidimensional stimulus-response correlation reveals supramodal neural responses to naturalistic stimuli

    January 6, 2017 | Jacek Dmochowski, J. Ki, P. DeGuzman, Paul Sajda, L. Parra

    In neuroscience, stimulus-response relationships have traditionally been analyzed using either encoding or decoding models. Here we combined both techniques by decomposing neural activity into multiple components, each representing a portion of the stimulus. We tested this hybrid approach on encephalographic responses to auditory and audiovisual narratives identically experienced across subjects, as well as uniquely experienced video game play. The highest stimulus-response correlations (SRC) were detected for dynamic visual features. During narratives both auditory and visual SRC were modulated by attention and tracked correlations between subjects. During video game play, SRC was modulated by task difficulty and attentional state. Importantly, the strongest component extracted for visual and auditory features had nearly identical spatial distributions, suggesting that the predominant encephalographic response to naturalistic stimuli is supramodal. The variety of novel findings demonstrates the utility of measuring multidimensional stimulus-response correlations.

  • Neural mechanisms underlying catastrophic failure in human–machine interaction during aerial navigation

    Objective. We investigated the neural correlates of workload buildup in a fine visuomotor task called the boundary avoidance task (BAT). The BAT has been known to induce naturally occurring failures of human–machine coupling in high performance aircraft that can potentially lead to a crash—these failures are termed pilot induced oscillations (PIOs). Approach. We recorded EEG and pupillometry data from human subjects engaged in a flight BAT simulated within a virtual 3D environment. Main results. We find that workload buildup in a BAT can be successfully decoded from oscillatory features in the electroencephalogram (EEG). Information in delta, theta, alpha, beta, and gamma spectral bands of the EEG all contribute to successful decoding, however gamma band activity with a lateralized somatosensory topography has the highest contribution, while theta band activity with a fronto-central topography has the most robust contribution in terms of real-world usability. We show that the output of the spectral decoder can be used to predict PIO susceptibility. We also find that workload buildup in the task induces pupil dilation, the magnitude of which is significantly correlated with the magnitude of the decoded EEG signals. These results suggest that PIOs may result from the dysregulation of cortical networks such as the locus coeruleus (LC)—anterior cingulate cortex (ACC) circuit. Significance. Our findings may generalize to similar control failures in other cases of tight man-machine coupling where gains and latencies in the control system must be inferred and compensated for by the human operators. A closed-loop intervention using neurophysiological decoding of workload buildup that targets the LC-ACC circuit may positively impact operator performance in such situations.

  • Brain Dynamics of Post-Task Resting State are Influenced by Expertise: Insights From Baseball Players

    January 6, 2017 | Jordan Muraskin, S. Dodhia, G. Lieberman, J.O. Garcia, T. Verstynen, J.M. Vettel, Jason Sherwin, Paul Sajda

    Post-task resting state dynamics can be viewed as a task-driven state where behavioral performance is improved through endogenous, non-explicit learning. Tasks that have intrinsic value for individuals are hypothesized to produce post-task resting state dynamics that promote learning. We measured simultaneous fMRI/EEG and DTI in Division-1 collegiate baseball players and compared to a group of controls, examining differences in both functional and structural connectivity. Participants performed a surrogate baseball pitch Go/No-Go task before a resting state scan, and we compared post-task resting state connectivity using a seed-based analysis from the supplementary motor area (SMA), an area whose activity discriminated players and controls in our previous results using this task. Although both groups were equally trained on the task, the experts showed differential activity in their post-task resting state consistent with motor learning. Specifically, we found (1) differences in bilateral SMA–L Insula functional connectivity between experts and controls that may reflect group differences in motor learning, (2) differences in BOLD-alpha oscillation correlations between groups suggests variability in modulatory attention in the post-task state, and (3) group differences between BOLD-beta oscillations that may indicate cognitive processing of motor inhibition. Structural connectivity analysis identified group differences in portions of the functionally derived network, suggesting that functional differences may also partially arise from variability in the underlying white matter pathways. Generally, we find that brain dynamics in the post-task resting state differ as a function of subject expertise and potentially result from differences in both functional and structural connectivity.

  • Fusing multiple neuroimaging modalities to assess group differences in perception-action coupling

    January 4, 2017 | Jason Sherwin, Jordan Muraskin, G. Lieberman, J.O. Garcia, T. Verstynen, J.M. Vettel, Paul Sajda

    In the last few decades, noninvasive neuroimaging has revealed macroscale brain dynamics that underlie perception, cognition, and action. Advances in noninvasive neuroimaging target two capabilities: 1) increased spatial and temporal resolution of measured neural activity; and 2) innovative methodologies to extract brain–behavior relationships from evolving neuroimaging technology. We target the second. Our novel methodology integrated three neuroimaging methodologies and elucidated expertise-dependent differences in functional (fused EEG-fMRI) and structural (dMRI) brain networks for a perception–action coupling task. A set of baseball players and controls performed a Go/No-Go task designed to mimic the situation of hitting a baseball. In the functional analysis, our novel fusion methodology identifies 50-ms windows with predictive EEG neural correlates of expertise and fuses these temporal windows with fMRI activity in a whole-brain 2-mm voxel analysis, revealing time-localized correlations of expertise at a spatial scale of millimeters. The spatiotemporal cascade of brain activity reflecting expertise differences begins as early as 200 ms after the pitch starts and lasts up to 700 ms afterwards. Network differences are spatially localized to include motor and visual processing areas, providing evidence for differences in perception–action coupling between the groups. Furthermore, an analysis of structural connectivity reveals that the players have significantly more connections between cerebellar and left frontal/motor regions, and many of the functional activation differences between the groups are located within structurally defined network modules that differentiate expertise. In short, our novel method illustrates how multimodal neuroimaging can provide specific macroscale insights into the functional and structural correlates of expertise development.

  • Spectral Transfer Learning Using Information Geometry for a User-Independent Brain-Computer Interface

    December 20, 2016 | Nick Waytowich, V. Lawhern, A. Bohannon, K. Ball, B. Lance

    Recent advances in signal processing and machine learning techniques have enabled the application of Brain-Computer Interface (BCI) technologies to fields such as medicine, industry, and recreation; however, BCIs still suffer from the requirement of frequent calibration sessions due to the intra- and inter-individual variability of brain-signals, which makes calibration suppression through transfer learning an area of increasing interest for the development of practical BCI systems. In this paper, we present an unsupervised transfer method (spectral transfer using information geometry, STIG), which ranks and combines unlabeled predictions from an ensemble of information geometry classifiers built on data from individual training subjects. The STIG method is validated in both off-line and real-time feedback analysis during a rapid serial visual presentation task (RSVP). For detection of single-trial, event-related potentials (ERPs), the proposed method can significantly outperform existing calibration-free techniques as well as outperform traditional within-subject calibration techniques when limited data is available. This method demonstrates that unsupervised transfer learning for single-trial detection in ERP-based BCIs can be achieved without the requirement of costly training data, representing a step-forward in the overall goal of achieving a practical user-independent BCI system.

  • Simultaneous decomposition of multiple hyperspectral data sets: signal recovery of unknown fluorophores in the retinal pigment epithelium

    December 16, 2016 | R.T. Smith, R. Post, A. Johri, M. Lee, Z. Ablonczy, C. Curcio, T. Ach, Paul Sajda

    Upon excitation with different wavelengths of light, biological tissues emit distinct but related autofluorescence signals. We used non-negative matrix factorization (NMF) to simultaneously decompose co-registered hyperspectral emission data from human retinal pigment epithelium/Bruch’s membrane specimens illuminated with 436 and 480 nm light. NMF analysis was initialized with Gaussian mixture model fits and constrained to provide identical abundance images for the two excitation wavelengths. Spectra recovered this way were smoother than those obtained separately; fluorophore abundances more clearly localized within tissue compartments. These studies provide evidence that leveraging multiple co-registered hyperspectral emission data sets is preferential for identifying biologically relevant fluorophore information.

  • NEDE: An Open-Source Scripting Suite for Developing Experiments in 3D Virtual Environments

    December 16, 2016 | David Jangraw, A. Johri, M. Gribetz, Paul Sajda

    Background As neuroscientists endeavor to understand the brain’s response to ecologically valid scenarios, many are leaving behind hyper-controlled paradigms in favor of more realistic ones. This movement has made the use of 3D rendering software an increasingly compelling option. However, mastering such software and scripting rigorous experiments requires a daunting amount of time and effort. New method To reduce these startup costs and make virtual environment studies more accessible to researchers, we demonstrate a naturalistic experimental design environment (NEDE) that allows experimenters to present realistic virtual stimuli while still providing tight control over the subject’s experience. NEDE is a suite of open-source scripts built on the widely used Unity3D game development software, giving experimenters access to powerful rendering tools while interfacing with eye tracking and EEG, randomizing stimuli, and providing custom task prompts. Results Researchers using NEDE can present a dynamic 3D virtual environment in which randomized stimulus objects can be placed, allowing subjects to explore in search of these objects. NEDE interfaces with a research-grade eye tracker in real-time to maintain precise timing records and sync with EEG or other recording modalities. Comparison with existing methods Python offers an alternative for experienced programmers who feel comfortable mastering and integrating the various toolboxes available. NEDE combines many of these capabilities with an easy-to-use interface and, through Unity’s extensive user base, a much more substantial body of assets and tutorials. Conclusions Our flexible, open-source experimental design system lowers the barrier to entry for neuroscientists interested in developing experiments in realistic virtual environments.

  • Evoked Neural Responses to Events in Video

    December 16, 2016 | D. Rosenthal, P. DeGuzman, L. Parra, Paul Sajda

    In contrast to static imagery, detection of events of interest in video involves evidence accumulation across space and time; the observer is required to integrate features from both motion and form to decide whether a behavior constituents a target event. Do such events that extend in time elicit evoked responses of similar strength as evoked responses associated with instantaneous events such as the presentation of a static target image? Using a set of simulated scenarios, with avatars/actors having different behaviors, we identified evoked neural activity discriminative of target vs. distractor events (behaviors) at discrimination levels that are comparable to static imagery. EEG discriminative activity was largely in the time-locked evoked response and not in oscillatory activity, with the exception of very low EEG frequency bands such as delta and theta, which simply represent bands dominating the event related potential (ERP). The discriminative evoked response activity we see is observed in all target/distractor conditions and is robust across different recordings from the same subjects. The results suggest that we have identified a robust neural correlate of target detection in video, at least in terms of the stimulus set we used—i.e., dynamic behavior of an individual in a low clutter environment. Additional work is needed to test a larger variety of behaviors and more diverse environments.

  • Neurally and ocularly informed graph-based models for searching 3D environments

    December 16, 2016 | Paul Sajda, David Jangraw

    OBJECTIVE: As we move through an environment, we are constantly making assessments, judgments and decisions about the things we encounter. Some are acted upon immediately, but many more become mental notes or fleeting impressions-our implicit ‘labeling’ of the world. In this paper, we use physiological correlates of this labeling to construct a hybrid brain-computer interface (hBCI) system for efficient navigation of a 3D environment. APPROACH: First, we record electroencephalographic (EEG), saccadic and pupillary data from subjects as they move through a small part of a 3D virtual city under free-viewing conditions. Using machine learning, we integrate the neural and ocular signals evoked by the objects they encounter to infer which ones are of subjective interest to them. These inferred labels are propagated through a large computer vision graph of objects in the city, using semi-supervised learning to identify other, unseen objects that are visually similar to the labeled ones. Finally, the system plots an efficient route to help the subjects visit the ‘similar’ objects it identifies. MAIN RESULTS: We show that by exploiting the subjects’ implicit labeling to find objects of interest instead of exploring naively, the median search precision is increased from 25% to 97%, and the median subject need only travel 40% of the distance to see 84% of the objects of interest. We also find that the neural and ocular signals contribute in a complementary fashion to the classifiers’ inference of subjects’ implicit labeling. SIGNIFICANCE: In summary, we show that neural and ocular signals reflecting subjective assessment of objects in a 3D environment can be used to inform a graph-based learning model of that environment, resulting in an hBCI system that improves navigation and information delivery specific to the user’s interests.

  • Your eyes give you away: prestimulus changes in pupil diameter correlate with poststimulus task-related EEG dynamics

    December 16, 2016 | Linbi Hong, Jennifer Walz, Paul Sajda

    Pupillary measures have been linked to arousal and attention as well as activity in the brainstem’s locus coeruleus norepinephrine (LC-NE) system. Similarly, there is evidence that evoked EEG responses, such as the P3, might have LC-NE activity as their basis. Since it is not feasible to record electrophysiological data directly from the LC in humans due to its location in the brainstem, an open question has been whether pupillary measures and EEG variability can be linked in a meaningful way to shed light on the nature of the LC-NE role in attention and arousal. We used an auditory oddball task with a data-driven approach to learn task-relevant projections of the EEG, for windows of data spanning the entire trial. We investigated linear and quadratic relationships between the evoked EEG along these projections and both prestimulus (baseline) and poststimulus (evoked dilation) pupil diameter measurements. We found that baseline pupil diameter correlates with early (175–200 ms) and late (350–400 ms) EEG component variability, suggesting a linear relationship between baseline (tonic) LC-NE activity and evoked EEG. We found no relationships between evoked EEG and evoked pupil dilation, which is often associated with evoked (phasic) LC activity. After regressing out reaction time (RT), the correlation between EEG variability and baseline pupil diameter remained, suggesting that such correlation is not explainable by RT variability. We also investigated the relationship between these pupil measures and prestimulus EEG alpha activity, which has been reported as a marker of attentional state, and found a negative linear relationship with evoked pupil dilation. In summary, our results demonstrate significant relationships between prestimulus and poststimulus neural and pupillary measures, and they provide further evidence for tight coupling between attentional state and evoked neural activity and for the role of cortical and subcortical networks underlying the process of target detection.

  • Prestimulus alpha power predicts fidelity of sensory encoding in perceptual decision making

    December 16, 2016 | Paul Sajda, Bin Lou, Y. Li, Marios Philiastides

    Pre-stimulus α power has been shown to correlate with the behavioral accuracy of perceptual decisions. In most cases, these correlations have been observed by comparing α power for different behavioral outcomes (e.g. correct vs incorrect trials). In this paper we investigate such covariation within the context of behaviorally-latent fluctuations in task-relevant post-stimulus neural activity. Specially we consider variations of pre-stimulus α power with post-stimulus EEG components in a two alternative forced choice visual discrimination task. EEG components, discriminative of stimulus class, are identified using a linear multivariate classifier and only the variability of the components for correct trials (regardless of stimulus class, and for nominally identical stimuli) are correlated with the corresponding pre-stimulus α power. We find a significant relationship between the mean and variance of the pre-stimulus α power and the variation of the trial-to-trial magnitude of an early post-stimulus EEG component. This relationship is not seen for a later EEG component that is also discriminative of stimulus class and which has been previously linked to the quality of evidence driving the decision process. Our results suggest that early perceptual representations, rather than temporally later neural correlates of the perceptual decision, are modulated by pre-stimulus state.

  • Simultaneous EEG-fMRI Reveals a Temporal Cascade of Task-Related and Default-Mode Activations During a Simple Target Detection Task

    December 16, 2016 | Jennifer Walz, Robin Goldman, M. Carapezza, Jordan Muraskin, Truman R. Brown, Paul Sajda

    Focused attention continuously and inevitably fluctuates, and to completely understand the mechanisms responsible for these modulations it is necessary to localize the brain regions involved. During a simple visual oddball task, neural responses measured by electroencephalography (EEG) modulate primarily with attention, but source localization of the correlates is a challenge. In this study we use single-trial analysis of simultaneously-acquired scalp EEG and functional magnetic resonance image (fMRI) data to investigate the blood oxygen level dependent (BOLD) correlates of modulations in task-related attention, and we unravel the temporal cascade of these transient activations. We hypothesize that activity in brain regions associated with various task-related cognitive processes modulates with attention, and that their involvements occur transiently in a specific order. We analyze the fMRI BOLD signal by first regressing out the variance linked to observed stimulus and behavioral events. We then correlate the residual variance with the trial-to-trial variation of EEG discriminating components for identical stimuli, estimated at a sequence of times during a trial. Post-stimulus and early in the trial, we find activations in right-lateralized frontal regions and lateral occipital cortex, areas that are often linked to task-dependent processes, such as attentional orienting, and decision certainty. After the behavioral response we see correlates in areas often associated with the default-mode network and introspective processing, including precuneus, angular gyri, and posterior cingulate cortex. Our results demonstrate that during simple tasks both task-dependent and default-mode networks are transiently engaged, with a distinct temporal ordering and millisecond timescale.

  • Grand challenges in mapping the human brain: NSF workshop report

    December 16, 2016 | B. He, T. Coleman, G.M. Genin, G. Glover, X. Hu, N. Johnson, T. Liu, S. Makeig, Paul Sajda, K. Ye

    This report summarizes the outcomes of the NSF Workshop on Mapping and Engineering the Brain, held at Arlington, VA, during August 13-14, 2013. Three grand challenges were identified, including high spatiotemporal resolution neuroimaging, perturbation-based neuroimaging, and neuroimaging in naturalistic environments. It was highlighted that each grand challenge requires groundbreaking discoveries, enabling technologies, appropriate knowledge transfer, and multi- and transdisciplinary education and training for success.

  • EEG precursors of detected and missed targets during free-viewing search

    December 16, 2016 | J.C. Dias, Paul Sajda, Jacek Dmochowski, L. Parra

    When scanning a scene, the target of our search may be in plain sight and yet remain unperceived. Conversely, at other times the target may be perceived in the periphery prior to fixation. There is ample behavioral and neurophysiological evidence to suggest that in some constrained visual-search tasks, targets are detected prior to fixational eye movements. However, limited human data are available during unconstrained search to determine the time course of detection, the brain areas involved, and the neural correlates of failures to detect a foveated target. Here, we recorded and analyzed electroencephalographic (EEG) activity during free-viewing visual search, varying the task difficulty to compare neural signatures for detected and unreported (“missed”) targets. When carefully controlled to remove eye-movement-related potentials, saccade-locked EEG shows that: (a) “Easy” targets may be detected as early as 150 ms prior to foveation, as indicated by a premotor potential associated with a button response; (b) object-discriminating occipital activity emerges during the saccade to target; and (c) success and failures to detect a target are accompanied by a modulation in alpha-band power over fronto-central areas as well as altered saccade dynamics. Taken together, these data suggest that target detection during free viewing can begin prior to and continue during a saccade, with failure or success in reporting a target possibly resulting from inhibition or activation of fronto-central processing areas associated with saccade control.

  • Fast bootstrapping and permutation testing for assessing reproducibility and interpretability of multivariate FMRI decoding models

    December 16, 2016 | Bryan Conroy, Jennifer Walz, Paul Sajda

    Multivariate decoding models are increasingly being applied to functional magnetic imaging (fMRI) data to interpret the distributed neural activity in the human brain. These models are typically formulated to optimize an objective function that maximizes decoding accuracy. For decoding models trained on full-brain data, this can result in multiple models that yield the same classification accuracy, though some may be more reproducible than others—i.e. small changes to the training set may result in very different voxels being selected. This issue of reproducibility can be partially controlled by regularizing the decoding model. Regularization, along with the cross-validation used to estimate decoding accuracy, typically requires retraining many (often on the order of thousands) of related decoding models. In this paper we describe an approach that uses a combination of bootstrapping and permutation testing to construct both a measure of cross-validated prediction accuracy and model reproducibility of the learned brain maps. This requires re-training our classification method on many re-sampled versions of the fMRI data. Given the size of fMRI datasets, this is normally a time-consuming process. Our approach leverages an algorithm called fast simultaneous training of generalized linear models (FaSTGLZ) to create a family of classifiers in the space of accuracy vs. reproducibility. The convex hull of this family of classifiers can be used to identify a subset of Pareto optimal classifiers, with a single-optimal classifier selectable based on the relative cost of accuracy vs. reproducibility. We demonstrate our approach using full-brain analysis of elastic-net classifiers trained to discriminate stimulus type in an auditory and visual oddball event-related fMRI design. Our approach and results argue for a computational approach to fMRI decoding models in which the value of the interpretation of the decoding model ultimately depends upon optimizing a joint space of accuracy and reproducibility.

  • Simultaneous EEG-fMRI Reveals Temporal Evolution of Coupling between Supramodal Cortical Attention Networks and the Brainstem

    Cortical and subcortical networks have been identified that are commonly associated with attention and task engagement, along with theories regarding their functional interaction. However, a link between these systems has not yet been demonstrated in healthy humans, primarily because of data acquisition and analysis limitations. We recorded simultaneous EEG–fMRI while subjects performed auditory and visual oddball tasks and used these data to investigate the BOLD correlates of single-trial EEG variability at latencies spanning the trial. We focused on variability along task-relevant dimensions in the EEG for identical stimuli and then combined auditory and visual data at the subject level to spatially and temporally localize brain regions involved in endogenous attentional modulations. Specifically, we found that anterior cingulate cortex (ACC) correlates strongly with both early and late EEG components, whereas brainstem, right middle frontal gyrus (rMFG), and right orbitofrontal cortex (rOFC) correlate significantly only with late components. By orthogonalizing with respect to event-related activity, we found that variability in insula and temporoparietal junction is reflected in reaction time variability, rOFC and brainstem correlate with residual EEG variability, and ACC and rMFG are significantly correlated with both. To investigate interactions between these correlates of temporally specific EEG variability, we performed dynamic causal modeling (DCM) on the fMRI data. We found strong evidence for reciprocal effective connections between the brainstem and cortical regions. Our results support the adaptive gain theory of locus ceruleus–norepinephrine (LC–NE) function and the proposed functional relationship between the LC–NE system, right-hemisphere ventral attention network, and P300 EEG response.

  • Musical experts recruit action-related neural structures in harmonic anomaly detection: Evidence for embodied cognition in expertise

    December 16, 2016 | Jason Sherwin, Paul Sajda

    Humans are extremely good at detecting anomalies in sensory input. For example, while listening to a piece of Western-style music, an anomalous key change or an out-of-key pitch is readily apparent, even to the non-musician. In this paper we investigate differences between musical experts and non-experts during musical anomaly detection. Specifically, we analyzed the electroencephalograms (EEG) of five expert cello players and five non-musicians while they listened to excerpts of J.S. Bach’s Prelude from Cello Suite No. 1. All subjects were familiar with the piece, though experts also had extensive experience playing the piece. Subjects were told that anomalous musical events (AMEs) could occur at random within the excerpts of the piece and were told to report the number of AMEs after each excerpt. Furthermore, subjects were instructed to remain still while listening to the excerpts and their lack of movement was verified via visual and EEG monitoring. Experts had significantly better behavioral performance (i.e. correctly reporting AME counts) than non-experts, though both groups had mean accuracies greater than 80%. These group differences were also reflected in the EEG correlates of key-change detection post-stimulus, with experts showing more significant, greater magnitude, longer periods of, and earlier peaks in condition-discriminating EEG activity than novices. Using the timing of the maximum discriminating neural correlates, we performed source reconstruction and compared significant differences between cellists and non-musicians. We found significant differences that included a slightly right lateralized motor and frontal source distribution. The right lateralized motor activation is consistent with the cortical representation of the left hand – i.e. the hand a cellist would use, while playing, to generate the anomalous key-changes. In general, these results suggest that sensory anomalies detected by experts may in fact be partially a result of an embodied cognition, with a model of the action for generating the anomaly playing a role in its detection.

  • Perceptual Decision Making “Through the Eyes” of a Large-scale Neural Model of V1

    December 16, 2016 | Jianing Shi, Jim Wielaard, R.T. Smith, Paul Sajda

    Sparse coding has been posited as an efficient information processing strategy employed by sensory systems, particularly visual cortex. Substantial theoretical and experimental work has focused on the issue of sparse encoding, namely how the early visual system maps the scene into a sparse representation. In this paper we investigate the complementary issue of sparse decoding, for example given activity generated by a realistic mapping of the visual scene to neuronal spike trains, how do downstream neurons best utilize this representation to generate a “decision.” Specifically we consider both sparse (L1-regularized) and non-sparse (L2 regularized) linear decoding for mapping the neural dynamics of a large-scale spiking neuron model of primary visual cortex (V1) to a two alternative forced choice (2-AFC) perceptual decision. We show that while both sparse and non-sparse linear decoding yield discrimination results quantitatively consistent with human psychophysics, sparse linear decoding is more efficient in terms of the number of selected informative dimension.

  • Post-stimulus endogenous and exogenous oscillations are differentially modulated by task difficulty.

    December 16, 2016 | Y. Li, Bin Lou, X. Gao, Paul Sajda

    We investigate the modulation of post-stimulus endogenous and exogenous oscillations when a visual discrimination is made more difficult. We use exogenous frequency tagging to induce steady-state visually evoked potentials (SSVEP) while subjects perform a face-car discrimination task, the difficulty of which varies on a trial-to-trial basis by varying the noise (phase coherence) in the image. We simultaneously analyze amplitude modulations of the SSVEP and endogenous alpha activity as a function of task difficulty. SSVEP modulation can be viewed as a neural marker of attention toward/away from the primary task, while modulation of post-stimulus alpha is closely related to cortical information processing. We find that as the task becomes more difficult, the amplitude of SSVEP decreases significantly, approximately 250-450 ms post-stimulus. Significant changes in endogenous alpha amplitude follow SSVEP modulation, occurring at approximately 400-700 ms post-stimulus and, unlike the SSVEP, the alpha amplitude is increasingly suppressed as the task becomes less difficult. Our results demonstrate simultaneous measurement of endogenous and exogenous oscillations that are modulated by task difficulty, and that the specific timing of these modulations likely reflects underlying information processing flow during perceptual decision-making.

  • Prospective active marker motion correction improves statistical power in BOLD fMRI

    December 16, 2016 | Jordan Muraskin, M.B. Ooi, Robin Goldman, S. Krueger, W.J. Thomas, Paul Sajda, Truman R. Brown

    Group level statistical maps of blood oxygenation level dependent (BOLD) signals acquired using functional magnetic resonance imaging (fMRI) have become a basic measurement for much of systems, cognitive and social neuroscience. A challenge in making inferences from these statistical maps is the noise and potential confounds that arise from the head motion that occurs within and between acquisition volumes. This motion results in the scan plane being misaligned during acquisition, ultimately leading to reduced statistical power when maps are constructed at the group level. In most cases, an attempt is made to correct for this motion through the use of retrospective analysis methods. In this paper, we use a prospective active marker motion correction (PRAMMO) system that uses radio frequency markers for real-time tracking of motion, enabling on-line slice plane correction. We show that the statistical power of the activation maps is substantially increased using PRAMMO compared to conventional retrospective correction. Analysis of our results indicates that the PRAMMO acquisition reduces the variance without decreasing the signal component of the BOLD (beta). Using PRAMMO could thus improve the overall statistical power of fMRI based BOLD measurements, leading to stronger inferences of the nature of processing in the human brain.

  • You Can’t Think and Hit at the Same Time: Neural Correlates of Baseball Pitch Classification

    Hitting a baseball is often described as the most difficult thing to do in sports. A key aptitude of a good hitter is the ability to determine which pitch is coming. This rapid decision requires the batter to make a judgment in a fraction of a second based largely on the trajectory and spin of the ball. When does this decision occur relative to the ball’s trajectory and is it possible to identify neural correlates that represent how the decision evolves over a split second? Using single-trial analysis of electroencephalography (EEG) we address this question within the context of subjects discriminating three types of pitches (fastball, curveball, slider) based on pitch trajectories. We find clear neural signatures of pitch classification and, using signal detection theory, we identify the times of discrimination on a trial-to-trial basis. Based on these neural signatures we estimate neural discrimination distributions as a function of the distance the ball is from the plate. We find all three pitches yield unique distributions, namely the timing of the discriminating neural signatures relative to the position of the ball in its trajectory. For instance, fastballs are discriminated at the earliest points in their trajectory, relative to the two other pitches, which is consistent with the need for some constant time to generate and execute the motor plan for the swing (or inhibition of the swing). We also find incorrect discrimination of a pitch (errors) yields neural sources in Brodmann Area 10, which has been implicated in prospective memory, recall, and task difficulty. In summary, we show that single-trial analysis of EEG yields informative distributions of the relative point in a baseball’s trajectory when the batter makes a decision on which pitch is coming.

  • Where Is the User in Multimedia Retrieval?

    December 16, 2016 | M. Worring, Paul Sajda, S. Santini, D.A. Shamma, A.F. Smeaton, Q. Yang

    This article summarizes a recent panel discussion at the ACM International Conference on Multimedia Retrieval, where a case was made for making the interacting user a first-class citizen again in multimedia retrieval research.

  • Neural correlates of the time course of the waxing and waning of attention

    December 16, 2016 | Paul Sajda

    A commentary on Trial-by-trial variations in subjective attentional state are reflected in ongoing prestimulus EEG alpha oscillations by Macdonald, J. S. P., Mathan, S., and Yeung, N. (2011). Front. Percept. Sci. 2:82. doi: 10.3389/fpsyg.2011.00082

  • Components of ongoing EEG with high correlation point to emotionally-laden attention — a possible marker of engagement?

    December 16, 2016 | Jacek Dmochowski, Paul Sajda, J. Dias Joao, L. Parra

    Recent evidence from functional magnetic resonance imaging suggests that cortical hemo- dynamic responses coincide in different subjects experiencing a common naturalistic stimulus. Here we utilize neural responses in the electroencephalogram (EEG) evoked by multiple presentations of short film clips to index brain states marked by high levels of corre- lation within and across subjects.We formulate a novel signal decomposition method which extracts maximally correlated signal components from multiple EEG records.The resulting components capture correlations down to a one-second time resolution, thus revealing that peak correlations of neural activity across viewings can occur in remarkable corre- spondence with arousing moments of the film. Moreover, a significant reduction in neural correlation occurs upon a second viewing of the film or when the narrative is disrupted by presenting its scenes scrambled in time. We also probe oscillatory brain activity during periods of heightened correlation, and observe during such times a significant increase in the theta band for a frontal component and reductions in the alpha and beta frequency bands for parietal and occipital components. Low-resolution EEG tomography of these components suggests that the correlated neural activity is consistent with sources in the cingulate and orbitofrontal cortices. Put together, these results suggest that the observed synchrony reflects attention- and emotion-modulated cortical processing which may be decoded with high temporal resolution by extracting maximally correlated components of neural activity.

  • Decoding of simulated neurodynamics predicts the perceptual consequences of age-related macular degeneration

    December 15, 2016 | Jim Wielaard, Paul Sajda, Jianing Shi

    Age-related macular degeneration (AMD) is the major cause of blindness in the developed world. Though substantial work has been done to characterize the disease, it is difficult to predict how the state of an individual’s retina will ultimately affect their high-level perceptual function. In this paper, we describe an approach that couples retinal imaging with computational neural modeling of early visual processing to generate quantitative predictions of an individual’s visual perception. Using a patient population with mild to moderate AMD, we show that we are able to accurately predict subject-specific psychometric performance by decoding simulated neurodynamics that are a function of scotomas derived from an individual’s fundus image. On the population level, we find that our approach maps the disease on the retina to a representation that is a substantially better predictor of high-level perceptual performance than traditional clinical metrics such as drusen density and coverage. In summary, our work identifies possible new metrics for evaluating the efficacy of treatments for AMD at the level of the expected changes in high-level visual perception and, in general, typifies how computational neural models can be used as a framework to characterize the perceptual consequences of early visual pathologies.

  • Single-trial analyses: why bother?

    December 15, 2016 | C.R. Pernet, Paul Sajda, G.A. Rousselet
  • Adaptive Thresholding for Improving Sensitivity in Single-Trial Simultaneous EEG/fMRI

    December 15, 2016 | Megan deBettencourt, Robin Goldman, Truman R. Brown, Paul Sajda

    A common approach used to fuse simultaneously recorded EEG and fMRI is to correlate trial-by-trial variability in the EEG, or variability of components derived therefrom, with the blood oxygenation level dependent response. When this correlation is done using the conventional univariate approach, for example with the general linear model, there is the usual problem of correcting the statistics for multiple comparisons. Cluster thresholding is often used as the correction of choice, though in many cases it is utilized in an ad hoc way, for example by employing the same cluster thresholds for both traditional regressors (stimulus or behaviorally derived) and EEG-derived regressors. In this paper we describe a resampling procedure that takes into account the a priori statistics of the trial-to-trial variability of the EEG-derived regressors in a way that trades off cluster size and maximum voxel Z-score to properly correct for multiple comparisons. We show that this data adaptive procedure improves sensitivity for smaller clusters of activation, without sacrificing the specificity of the results. Our results suggest that extra care is needed in correcting statistics when the regressor model is derived from noisy and/or uncertain measurements, as is the case for regressors constructed from single-trial variations in the EEG.

  • Closing the loop in cortically-coupled computer vision: a BCI for searching image databases

    December 15, 2016 | Paul Sajda, Eric Pohlmeyer

    We describe a closed-loop brain–computer interface that re-ranks an image database by iterating between user generated ‘interest’ scores and computer vision generated visual similarity measures. The interest scores are based on decoding the electroencephalographic (EEG) correlates of target detection, attentional shifts and self-monitoring processes, which result from the user paying attention to target images interspersed in rapid serial visual presentation (RSVP) sequences. The highest scored images are passed to a semi-supervised computer vision system that reorganizes the image database accordingly, using a graph-based representation that captures visual similarity between images. The system can either query the user for more information, by adaptively resampling the database to create additional RSVP sequences, or it can converge to a ‘done’ state. The done state includes a final ranking of the image database and also a ‘guess’ of the user’s chosen category of interest. We find that the closed-loop system’s re-rankings can substantially expedite database searches for target image categories chosen by the subjects. Furthermore, better reorganizations are achieved than by relying on EEG interest rankings alone, or if the system were simply run in an open loop format without adaptive resampling.

  • Weighted Maximum Likelihood is a Nearly Optimal Solution to the Cost-Sensitive Learning Problem

    December 15, 2016 | Paul Sajda, Jacek Dmochowski

    The presence of asymmetry in the misclassification costs or class prevalences is a common occurrence in the pattern classification domain. While much interest has been devoted to the study of cost-sensitive learning techniques, the relationship between cost-sensitive learning and the specification of the model set in a parametric estimation framework remains somewhat unclear. To that end, we differentiate between the case of the model including the true posterior, and that in which the model is misspecified. In the former case, it is shown that thresholding the maximum likelihood (ML) estimate is an asymptotically optimal solution to the risk minimization problem. On the other hand, under model misspecification, it is demonstrated that thresholded ML is suboptimal and that the risk-minimizing solution varies with the misclassification cost ratio. Moreover, we analytically show that the negative weighted log likelihood (Elkan, 2001) is a tight, convex upper bound of the empirical loss. Coupled with empirical results on several real-world data sets, we argue that weighted ML is the preferred cost-sensitive technique.

  • A Fast Hybrid Algorithm for Large Scale l1-Regularized Logistic Regression

    December 15, 2016 | Paul Sajda, Jianing Shi, W. Yin, S. Osher

    l1-regularized logistic regression, also known as sparse logistic regression, is widely used in machine learning, computer vision, data mining, bioinformatics and neural signal processing. The use of l1 regularization attributes attractive properties to the classifier, such as feature selection, robustness to noise, and as a result, classifier generality in the context of supervised learning. When a sparse logistic regression problem has large-scale data in high dimensions, it is computationally expensive to minimize the non-differentiable l1-norm in the objective function. Motivated by recent work (Koh et al., 2007; Hale et al., 2008), we propose a novel hybrid algorithm based on combining two types of optimization iterations: one being very fast and memory friendly while the other being slower but more accurate. Called hybrid iterative shrinkage (HIS), the resulting algorithm is comprised of a fixed point continuation phase and an interior point phase. The first phase is based completely on memory efficient operations such as matrix-vector multiplications, while the second phase is based on a truncated Newton’s method. Furthermore, we show that various optimization techniques, including line search and continuation, can significantly accelerate convergence. The algorithm has global convergence at a geometric rate (a Q-linear rate in optimization terminology). We present a numerical comparison with several existing algorithms, including an analysis using benchmark data from the UCI machine learning repository, and show our algorithm is the most computationally efficient without loss of accuracy.

  • In a Blink of an Eye and a Switch of a Transistor: Cortically-coupled Computer Vision

    December 15, 2016 | Paul Sajda, L. Parra, C. Christoforou, B. Hanna, C. Bahlmann, J. Wang, S-F Chang

    Our society’s information technology advancements have resulted in the increasingly problematic issue of information overload-i.e., we have more access to information than we can possibly process. This is nowhere more apparent than in the volume of imagery and video that we can access on a daily basis-for the general public, availability of YouTube video and Google Images, or for the image analysis professional tasked with searching security video or satellite reconnaissance. Which images to look at and how to ensure we see the images that are of most interest to us, begs the question of whether there are smart ways to triage this volume of imagery. Over the past decade, computer vision research has focused on the issue of ranking and indexing imagery. However, computer vision is limited in its ability to identify interesting imagery, particularly as ¿interesting¿ might be defined by an individual. In this paper we describe our efforts in developing brain-computer interfaces (BCIs) which synergistically integrate computer vision and human vision so as to construct a system for image triage. Our approach exploits machine learning for real-time decoding of brain signals which are recorded noninvasively via electroencephalography (EEG). The signals we decode are specific for events related to imagery attracting a user’s attention. We describe two architectures we have developed for this type of cortically coupled computer vision and discuss potential applications and challenges for the future

  • Second-Order Bilinear Discriminant Analysis

    December 15, 2016 | C. Christoforou, R. Haralick, Paul Sajda, L. Parra

    Traditional analysis methods for single-trial classification of electro-encephalography (EEG) focus on two types of paradigms: phase-locked methods, in which the amplitude of the signal is used as the feature for classification, that is, event related potentials; and second-order methods, in which the feature of interest is the power of the signal, that is, event related (de)synchronization. The process of deciding which paradigm to use is ad hoc and is driven by assumptions regarding the underlying neural generators. Here we propose a method that provides an unified framework for the analysis of EEG, combining first and second-order spatial and temporal features based on a bilinear model. Evaluation of the proposed method on simulated data shows that the technique outperforms state-of-the art techniques for single-trial classification for a broad range of signal-to-noise ratios. Evaluations on human EEG−including one benchmark data set from the Brain Computer Interface (BCI) competition−show statistically significant gains in classification accuracy, with a reduction in overall classification error from 26%-28% to 19%.

  • Single-trial Analysis of Neuroimaging Data: Inferring Neural Networks Underlying Perceptual, Decision Making in the Human Brain

    December 15, 2016 | Paul Sajda, L. Parra

    Advances in neural signal and image acquisition as well as in multivariate signal processing and machine learning are enabling a richer and more rigorous understanding of the neural basis of human decision-making. Decision-making is essentially characterized behaviorally by the variability of the decision across individual trials—e.g., error and response time distributions. To infer the neural processes that govern decision-making requires identifying neural correlates of such trial-to-trial behavioral variability. In this paper, we review efforts that utilize signal processing and machine learning to enable single-trial analysis of neural signals acquired while subjects perform simple decision-making tasks. Our focus is on neuroimaging data collected noninvasively via electroencephalograpy (EEG) and functional magnetic resonance imaging (fMRI). We review the specific frame-work for extracting decision-relevant neural components from the neuroimaging data, the goal being to analyze the trial-to-trial variability of the neural signal along these component directions and to relate them to elements of the decision-making process. We review results for perceptual decision-making and discrimination tasks, including paradigms in which EEG variability is used to inform an fMRI analysis. We discuss how single-trial analysis reveals aspects of the underlying decision-making networks that are unobservable using traditional trial-averaging methods.

  • Single-trial discrimination for integrating simultaneous EEG and fMRI: Identifying cortical areas contributing to trial-to-trial variability in the auditory oddball task

    December 15, 2016 | Robin Goldman, CY Wei, Adam Gerson, D. Friedman, Paul Sajda

    The auditory oddball task is a well-studied stimulus paradigm used to investigate the neural correlates of simple target detection. It elicits several classic event-related potentials (ERPs), the most prominent being the P300 which is seen as a neural correlate of subjects’ detection of rare (target) stimuli. Though trial-averaging is typically used to identify and characterize such ERPs, their latency and amplitude can vary on a trial-to-trial basis reflecting variability in the underlying neural information processing. Here we simultaneously recorded EEG and fMRI during an auditory oddball task and identified cortical areas correlated with the trial-to-trial variability of task-discriminating EEG components. Unique to our approach is a linear multivariate method for identifying task-discriminating components within specific stimulus- or response-locked time windows. We find fMRI activations indicative of distinct processes that contribute to the single-trial variability during target detection. These regions are different from those found using standard, including trial-averaged, regressors. Of particular note is the strong activation of the lateral occipital complex (LOC). The LOC was not seen when using traditional event-related regressors. Though LOC is typically associated with visual/spatial attention, its activation in an auditory oddball task, where attention can wax and wane from trial to trial, indicates that it may be part of a more general attention network involved in allocating resources for target detection and decision making. Our results show that trial-to-trial variability in EEG components, acquired simultaneously with fMRI, can yield task-relevant BOLD activations that are otherwise unobservable using traditional fMRI analysis.

  • Quality of Evidence for Perceptual Decision Making is Indexed by Trial-to-Trial Variability of the EEG

    December 15, 2016 | R. Ratcliff, Paul Sajda

    A fundamental feature of how we make decisions is that our responses are variable in the choices we make and the time it takes to make them. This makes it impossible to determine, for a single trial of an experiment, the quality of the evidence on which a decision is based. Even for stimuli from a single experimental condition, it is likely that stimulus and encoding differences lead to differences in the quality of evidence. In the research reported here, with a simple “face”/”car” perceptual discrimination task, we obtained late (decision-related) and early (stimulus-related) single-trial EEG component amplitudes that discriminated between faces and cars within and across conditions. We used the values of these amplitudes to sort the response time and choice within each experimental condition into more-face-like and less-face-like groups and then fit the diffusion model for simple decision making (a well-established model in cognitive psychology) to the data in each group separately. The results show that dividing the data on a trial-by-trial basis by using the late-component amplitude produces differences in the estimates of evidence used in the decision process. However, dividing the data on the basis of the early EEG component amplitude or the times of the peak amplitudes of either component did not index the information used in the decision process. The results we present show that a single-trial EEG neurophysiological measure for nominally identical stimuli can be used to sort behavioral response times and choices into those that index the quality of decision-relevant evidence.

  • Comparing neural correlates of visual target detection in serial visual presentations having different temporal correlations

    December 15, 2016 | An Luo, Paul Sajda

    Most visual stimuli we experience on a day-to-day basis are continuous sequences, with spatial structure highly correlated in time. During rapid serial visual presentation (RSVP), this correlation is absent. Here we study how subjects’ target detection responses, both behavioral and electrophysiological, differ between continuous serial visual sequences (CSVP), flashed serial visual presentation (FSVP) and RSVP. Behavioral results show longer reaction times for CSVP compared to the FSVP and RSVP conditions, as well as a difference in miss rate between RSVP and the other two conditions. Using mutual information, we measure electrophysiological differences in the electroencephalography (EEG) for these three conditions. We find two peaks in the mutual information between EEG and stimulus class (target vs. distractor), with the second peak occurring 30–40 ms earlier for the FSVP and RSVP conditions. In addition, we find differences in the persistence of the peak mutual information between FSVP and RSVP conditions. We further investigate these differences using a mutual information based functional connectivity analysis and find significant fronto-parietal functional coupling for RSVP and FSVP but no significant coupling for the CSVP condition. We discuss these findings within the context of attentional engagement, evidence accumulation and short-term visual memory.

  • Removal of BCG artifacts using a non-Kirchhoffian overcomplete representation

    December 15, 2016 | Mads Dyrholm, Robin Goldman, Paul Sajda

    We present a nonlinear unmixing approach for extracting the ballistocardiogram (BCG) from EEG recorded in an MR scanner during simultaneous acquisition of functional MRI (fMRI). First, an overcomplete basis is identified in the EEG based on a custom multipath EEG electrode cap. Next, the overcomplete basis is used to infer non-Kirchhoffian latent variables that are not consistent with a conservative electric field. Neural activity is strictly Kirchhoffian while the BCG artifact is not, and the representation can hence be used to remove the artifacts from the data in a way that does not attenuate the neural signals needed for optimal single-trial classification performance. We compare our method to more standard methods for BCG removal, namely independent component analysis and optimal basis sets, by looking at single-trial classification performance for an auditory oddball experiment. We show that our overcomplete representation method for removing BCG artifacts results in better single-trial classification performance compared to the conventional approaches, indicating that the derived neural activity in this representation retains the complex information in the trial-to-trial variability.

  • Spectrum Separation Resolves Partial Volume Effect of MRSI as Demonstrated on Brain Tumor

    December 15, 2016 | Y. Su, S. Thakur, K. Sasan, S. Du, Paul Sajda, W. Huang, L. Parra

    Magnetic resonance spectroscopic imaging (MRSI) is currently used clinically in conjunction with anatomical MRI to assess the presence and extent of brain tumors and to evaluate treatment response. Unfortunately, the clinical utility of MRSI is limited by significant variability of in vivo spectra. Spectral profiles show increased variability because of partial coverage of large voxel volumes, infiltration of normal brain tissue by tumors, innate tumor heterogeneity, and measurement noise. We address these problems directly by quantifying the abundance (i.e. volume fraction) within a voxel for each tissue type instead of the conventional estimation of metabolite concentrations from spectral resonance peaks. This ‘spectrum separation’ method uses the non-negative matrix factorization algorithm, which simultaneously decomposes the observed spectra of multiple voxels into abundance distributions and constituent spectra. The accuracy of the estimated abundances is validated on phantom data. The presented results on 20 clinical cases of brain tumor show reduced cross-subject variability. This is reflected in improved discrimination between high-grade and low-grade gliomas, which demonstrates the physiological relevance of the extracted spectra. These results show that the proposed spectral analysis method can improve the effectiveness of MRSI as a diagnostic tool.

  • Spatio-temporal linear decoding of brain state: Application to performance augmentation in high-throughput tasks

    December 15, 2016 | L. Parra, C. Christoforou, Adam Gerson, Mads Dyrholm, An Luo, M. Wagner, Paul Sajda

    This review summarizes linear spatiotemporal signal analysis methods that derive their power from careful consideration of spatial and temporal features of skull surface potentials. BCIs offer tremendous potential for improving the quality of life for those with severe neurological disabilities. At the same time, it is now possible to use noninvasive systems to improve performance for time-demanding tasks. Signal processing and machine learning are playing a fundamental role in enabling applications of BCI and in many respects, advances in signal processing and computation have helped to lead the way to real utility of noninvasive BCI.

  • EEG-Informed fMRI Reveals Spatiotemporal Characteristics of Perceptual Decision Making

    December 14, 2016 | Paul Sajda, Marios Philiastides

    Single-unit and multiunit recordings in primates have already established that decision making involves at least two general stages of neural processing: representation of evidence from early sensory areas and accumulation of evidence to a decision threshold from decision-related regions. However, the relay of information from early sensory to decision areas, such that the accumulation process is instigated, is not well understood. Using a cued paradigm and single-trial analysis of electroencephalography (EEG), we previously reported on temporally specific components related to perceptual decision making. Here, we use information derived from our previous EEG recordings to inform the analysis of fMRI data collected for the same behavioral task to ascertain the cortical origins of each of these EEG components. We demonstrate that a cascade of events associated with perceptual decision making takes place in a highly distributed neural network. Of particular importance is an activation in the lateral occipital complex implicating perceptual persistence as a mechanism by which object decision making in the human brain is instigated.

  • Dependence of response properties on sparse connectivity in a spiking neuron model of the lateral geniculate nucleus

    December 14, 2016 | Paul Sajda, Jim Wielaard

    We present a large-scale anatomically constrained spiking neuron model of the lateral geniculate nucleus (LGN), which operates solely with retinal input, relay cells, and interneurons. We show that interneuron inhibition and sparse connectivity between LGN cells could be key factors for explaining a number of observed classical and extraclassical response properties in LGN of monkey and cat. Among them are 1 ) weak orientation tuning, 2 ) contrast invariance of spatial frequency tuning in the absence of cortical feedback, 3 ) extraclassical surround suppression, and 4 ) orientation tuning of extraclassical surround suppression. The model also makes two surprising predictions: 1 ) a possible pinwheel-like spatial organization of orientation preference in the parvo layers of monkey LGN, much like what is seen in V1, and 2 ) a stimulus-induced trend (bias) in the orientation and phase preference of surround suppression, originating from the stimulus discontinuity between center and surround gratings rather than from specific circuitry.

  • Causal influences in the human brain during face discrimination: a short-window directed transfer function approach

    December 14, 2016 | Paul Sajda, Marios Philiastides

    In this letter, we considered the application of parametric spectral analysis, namely a short-window directed transfer function (DTF) approach, to multichannel electroencephalography (EEG) data during a face discrimination task. We identified causal influences between occipitoparietal and centrofrontal electrode sites, the timing of which corresponded to previously reported EEG face-selective components. More importantly we present evidence that there are both feedforward and feedback influences, a finding that is in direct contrast to current computational models of perceptual discrimination and decision making which tend to favor a purely feedforward processing scheme.

  • HiRes – A Tool for Comprehensive Assessment and Interpretation of Metabolomic Data

    December 14, 2016 | Q. Zhao, R. Stoyanova, S. Du, Paul Sajda

    The increasing role of metabolomics in system biology is driving the development of tools for comprehensive analysis of high-resolution NMR spectral datasets. This task is quite challenging since unlike the datasets resulting from other ‘omics’, a substantial preprocessing of the data is needed to allow successful identification of spectral patterns associated with relevant biological variability. HiRes is a unique stand-alone software tool that combines standard NMR spectral processing functionalities with techniques for multi-spectral dataset analysis, such as principal component analysis and non-negative matrix factorization. In addition, HiRes contains extensive abilities for data cleansing, such as baseline correction, solvent peak suppression, removal of frequency shifts owing to experimental conditions as well as auxiliary information management. Integration of these components together with multivariate analytical procedures makes HiRes very capable of addressing the challenges for assessment and interpretation of large metabolomic datasets, greatly simplifying this otherwise lengthy and difficult process and assuring optimal information retrieval.

  • Extraclassical receptive field phenomena & short-range connectivity in V1

    December 14, 2016 | Jim Wielaard, Paul Sajda

    Extraclassical receptive field phenomena in V1 are commonly attributed to long-range lateral connections and/or extrastriate feedback. We address 2 such phenomena: surround suppression and receptive field expansion at low contrast. We present rigorous computational support for the hypothesis that the phenomena largely result from local short-range (

  • Circuitry and the classification of simple and complex cells in V1

    December 14, 2016 | Paul Sajda, Jim Wielaard

    Based on a large-scale neural network model of striate cortex (V1), we present a simulation study of extra- and intracellular response modulations for drifting and contrast reversal grating stimuli. Specifically, we study the dependency of these modulations on the neural circuitry. We find that the frequently used ratio of the first harmonic to the mean response to classify simple and complex cells is highly insensitive to circuitry. Limited experimental sample size for the distribution of this measure makes it unsuitable for distinguishing whether the dichotomy of simple and complex cells originates from distinct LGN axon connectivity and/or local circuitry in V1. We show that a possible useful measure in this respect is the ratio of the intracellular second- to first-harmonic response for contrast reversal gratings. This measure is highly sensitive to neural circuitry and its distribution can be sampled with sufficient accuracy from a limited amount of experimental data. Further, the distribution of this measure is qualitatively similar to that of the subfield correlation coefficient, although it is more robust and easier to obtain experimentally.

  • Cortically-coupled computer vision for rapid image search

    December 14, 2016 | Adam Gerson, L. Parra, Paul Sajda

    We describe a real-time electroencephalography (EEG)-based brain-computer interface system for triaging imagery presented using rapid serial visual presentation. A target image in a sequence of nontarget distractor images elicits in the EEG a stereotypical spatiotemporal response, which can be detected. A pattern classifier uses this response to reprioritize the image sequence, placing detected targets in the front of an image stack. We use single-trial analysis based on linear discrimination to recover spatial components that reflect differences in EEG activity evoked by target versus nontarget images. We find an optimal set of spatial weights for 59 EEG sensors within a sliding 50-ms time window. Using this simple classifier allows us to process EEG in real time. The detection accuracy across five subjects is on average 92%, i.e., in a sequence of 2500 images, resorting images based on detector output results in 92% of target images being moved from a random position in the sequence to one of the first 250 images (first 10% of the sequence). The approach leverages the highly robust and invariant object recognition capabilities of the human visual system, using single-trial EEG analysis to efficiently detect neural signatures correlated with the recognition event.

  • Machine learning for detection and diagnosis of disease

    December 14, 2016 | Paul Sajda

    Machine learning offers a principled approach for developing sophisticated, automatic, and objective algorithms for analysis of high-dimensional and multimodal biomedical data. This review focuses on several advances in the state of the art that have shown promise in improving detection, diagnosis, and therapeutic monitoring of disease. Key in the advancement has been the development of a more in-depth understanding and theoretical analysis of critical issues related to algorithmic construction and learning theory. These include trade-offs for maximizing generalization performance, use of physically realistic constraints, and incorporation of prior knowledge and uncertainty. The review describes recent developments in machine learning, focusing on supervised and unsupervised linear methods and Bayesian inference, which have made significant impacts in the detection and diagnosis of disease in biomedicine. We describe the different methodologies and, for each, provide examples of their application to specific domains in biomedical diagnostics.

  • Neural representation of task difficulty and decision making during perceptual categorization: a timing diagram

    December 14, 2016 | Paul Sajda, Marios Philiastides

    When does the brain know that a decision is difficult to make? How does decision difficulty affect the allocation of neural resources and timing of constituent cortical processing? Here, we use single-trial analysis of electroencephalography (EEG) to identify neural correlates of decision difficulty and relate these to neural correlates of decision accuracy. Using a cued paradigm, we show that we can identify a component in the EEG that reflects the inherent task difficulty and not simply a correlation with the stimulus. We find that this decision difficulty component arises ≈220 ms after stimulus presentation, between two EEG components that are predictive of decision accuracy [an “early” (170 ms) and a “late” (≈300 ms) component]. We use these results to develop a timing diagram for perceptual decision making and relate the component activities to parameters of a diffusion model for decision making.

  • Learning discrimination trajectories in EEG sensor space: Application to inferring task difficulty

    December 14, 2016 | An Luo, Paul Sajda

    We describe a spatio-temporal linear discriminator for single-trial classification of multi-channel electroencephalography (EEG). No prior information about the characteristics of the neural activity is required, i.e., the algorithm requires no knowledge about the timing and spatial distribution of the evoked responses. The algorithm finds a temporal delay/window onset time for each EEG channel and then spatially integrates the channels for each channel-specific onset time. The algorithm can be seen as learning discrimination trajectories defined within the space of EEG channels. We demonstrate the method for detecting auditory-evoked neural activity and discrimination of task difficulty in a complex visual-auditory environment.

  • Varying complexity in tree structured distribution models

    December 14, 2016 | C. Spence, L. Parra, Paul Sajda

    Probabilistic models of image statistics underlie many approaches in image analysis and processing. An important class of such models have variables whose dependency graph is a tree. If the hidden variables take values on a finite set, most computations with the model can be performed exactly, including the likelihood calculation, training with the EM algorithm, etc. Crouse et al. developed one such model, the hidden Markov tree ( HMT). They took particular care to limit the complexity of their model. We argue that it is beneficial to allow more complex tree-structured models, describe the use of information theoretic penalties to choose the model complexity, and present experimental results to support these proposals. For these experiments, we use what we call the hierarchical image probability (HIP) model. The differences between the HIP and the HMT models include the use of multivariate Gaussians to model the distributions of local vectors of wavelet coefficients and the use of different numbers of hidden states at each resolution. We demonstrate the broad utility of image distributions by applying the HIP model to classification, synthesis, and compression, across a variety of image types, namely, electrooptical, synthetic aperture radar, and mammograms (digitized X-rays). In all cases, we compare with the HMT.

  • Temporal characterization of the neural correlates of perceptual decision making in the human brain

    December 14, 2016 | Paul Sajda, Marios Philiastides

    Single and multi-unit recordings in primates have identified spatially localized neuronal activity correlating with an animal’s behavioral performance. Due to the invasive nature of these experiments, it has been difficult to identify such correlates in humans. We report the first non-invasive neural measurements of perceptual decision making, via single-trial EEG analysis, that lead to neurometric functions predictive of psychophysical performance for a face versus car categorization task. We identified two major discriminating components. The earliest correlating with psychophysical performance was consistent with the well-known face-selective N170. The second component, which was a better match to the psychometric function, did not occur until at least 130 ms later. As evidence for faces versus cars decreased, onset of the later, but not the earlier, component systematically shifted forward in time. In addition, a choice probability analysis indicated strong correlation between the neural responses of the later component and our subjects’ behavioral judgements. These findings demonstrate a temporal evolution of component activity indicative of an evidence accumulation process which begins after early visual perception and has a processing time that depends on the strength of the evidence.

  • Recipes for the linear analysis of EEG

    October 28, 2016 | L. Parra, C. Spence, Adam Gerson, Paul Sajda

    In this paper, we describe a simple set of “recipes” for the analysis of high spatial density EEG. We focus on a linear integration of multiple channels for extracting individual components without making any spatial or anatomical modeling assumptions, instead requiring particular statistical properties such as maximum difference, maximum power, or statistical independence. We demonstrate how corresponding algorithms, for example, linear discriminant analysis, principal component analysis and independent component analysis, can be used to remove eye-motion artifacts, extract strong evoked responses, and decompose temporally overlapping components. The general approach is shown to be consistent with the underlying physics of EEG, which specifies a linear mixing model of the underlying neural and non-neural current sources.

  • Inferring figure-ground using a recurrent integrate-and-fire neural circuit

    October 28, 2016 | Kay Baek, Paul Sajda

    Several theories of early visual perception hypothesize neural circuits that are responsible for assigning ownership of an object’s occluding contour to a region which represents the “figure.” Previously, we have presented a Bayesian network model which integrates multiple cues and uses belief propagation to infer local figure-ground relationships along an object’s occluding contour. In this paper, we use a linear integrate-and-fire model to demonstrate how such inference mechanisms could be carried out in a biologically realistic neural circuit. The circuit maps the membrane potentials of individual neurons to log probabilities and uses recurrent connections to represent transition probabilities. The network’s “perception” of figure-ground is demonstrated for several examples, including perceptually ambiguous figures, and compared qualitatively and quantitatively with human psychophysics.

  • Non-negative matrix factorization for rapid recovery of constituent spectra in magnetic resonance chemical shift imaging of the brain

    October 28, 2016 | Paul Sajda, S. Du, Truman R. Brown, R. Stoyanova, D.C. Shungu, X. Mao, L. Parra

    We present an algorithm for blindly recovering constituent source spectra from magnetic resonance (MR) chemical shift imaging (CSI) of the human brain. The algorithm, which we call constrained nonnegative matrix factorization (cNMF), does not enforce independence or sparsity, instead only requiring the source and mixing matrices to be nonnegative. It is based on the nonnegative matrix factorization (NMF) algorithm, extending it to include a constraint on the positivity of the amplitudes of the recovered spectra. This constraint enables recovery of physically meaningful spectra even in the presence of noise that causes a significant number of the observation amplitudes to be negative. We demonstrate and characterize the algorithm’s performance using 31P volumetric brain data, comparing the results with two different blind source separation methods: Bayesian spectral decomposition (BSD) and nonnegative sparse coding (NNSC). We then incorporate the cNMF algorithm into a hierarchical decomposition framework, showing that it can be used to recover tissue-specific spectra given a processing hierarchy that proceeds coarse-to-fine. We demonstrate the hierarchical procedure on 1H brain data and conclude that the computational efficiency of the algorithm makes it well-suited for use in diagnostic work-up.

  • Integration of form and motion within a generative model of visual cortex

    October 28, 2016 | Paul Sajda, Kay Baek

    One of the challenges faced by the visual system is integrating cues within and across processing streams for inferring scene properties and structure. This is particularly apparent in the inference of object motion, where psychophysical experiments have shown that integration of motion signals, distributed across space, must also be integrated with form cues. This has led several to conclude that there exist mechanisms which enable form cues to ‘veto’ or completely suppress ambiguous motion signals. We describe a probabilistic approach which uses a generative network model for integrating form and motion cues using the machinery of belief propagation and Bayesian inference. We show, using computer simulations, that motion integration can be mediated via a local, probabilistic representation of contour ownership, which we have previously termed ‘direction of figure’. The uncertainty of this inferred form cue is used to modulate the covariance matrix of network nodes representing local motion estimates in the motion stream. We show with results for two sets of stimuli that the model does not completely suppress ambiguous cues, but instead integrates them in a way that is a function of their underlying uncertainty. The result is that the model can account for the continuum of bias seen for motion coherence and perceived object motion in psychophysical experiments.

  • A multi-scale probabilistic network model for detection, synthesis and compression in mammographic image analysis

    October 28, 2016 | Paul Sajda, C. Spence, L. Parra

    We develop a probabilistic network model over image spaces and demonstrate its broad utility in mammographic image analysis, particularly with respect to computer-aided diagnosis. The model employs a multi-scale pyramid decomposition to factor images across scale and a network of tree-structured hidden variables to capture long-range spatial dependencies. This factoring makes the computation of the density functions local and tractable. The result is a hierarchical mixture of conditional probabilities, similar to a hidden Markov model on a tree. The model parameters are found with maximum likelihood estimation using the expectation-maximization algorithm. The utility of the model is demonstrated for three applications: (1) detection of mammographic masses for computer-aided diagnosis; (2) qualitative assessment of model structure through mammographic synthesis; and (3) compression of mammographic regions of interest.

  • Single-trial detection in EEG and MEG: Keeping it linear

    October 28, 2016 | L. Parra, C. Alvino, A. Tang, B. Pearlmutter, N. Yeung, A. Osman, Paul Sajda

    Conventional electroencephalography (EEG) and magnetoencephalography (MEG) analysis often rely on averaging over multiple trials to extract statistically relevant differences between two or more experimental conditions. We demonstrate that by linearly integrating information over multiple spatially distributed sensors within a predefined time window, one can discriminate conditions on a trial-by-trial basis with high accuracy. We restrict ourselves to a linear integration as it allows the computation of a spatial distribution of the discriminating source activity. In the present set of experiments the resulting source activity distributions correspond to functional neuroanatomy consistent with the task (e.g. contralateral sensory-motor cortex and anterior cingulate).

  • A data analysis competition to evaluate machine learning algorithms for use in brain-computer interfaces

    October 28, 2016 | Paul Sajda, Adam Gerson, K-R Mueller, B. Blankertz, L. Parra

    We present three datasets that were used to conduct an open competition for evaluating the performance of various machine-learning algorithms used in brain-computer interfaces. The datasets were collected for tasks that included: 1) detecting explicit left/right (L/R) button press; 2) predicting imagined L/R button press; and 3) vertical cursor control. A total of ten entries were submitted to the competition, with winning results reported for two of the three datasets.

  • Response error correction: A demonstration of improved human-machine performance using real-time EEG monitoring

    October 28, 2016 | L. Parra, C. Spence, Adam Gerson, Paul Sajda

    We describe a brain-computer interface (BCI) system, which uses a set of adaptive linear preprocessing and classification algorithms for single-trial detection of error related negativity (ERN). We use the detected ERN as an estimate of a subject’s perceived error during an alternative forced choice visual discrimination task. The detected ERN is used to correct subject errors. Our initial results show average improvement in subject performance of 21% when errors are automatically corrected via the BCI. We are currently investigating the generalization of the overall approach to other tasks and stimulus paradigms.

  • Blind Source Separation via Generalized Eigenvalue Decomposition

    October 28, 2016 | L. Parra, Paul Sajda

    In this short note we highlight the fact that linear blind source separation can be formulated as a generalized eigenvalue decomposition under the assumptions of non-Gaussian, non-stationary, or non-white independent sources. The solution for the unmixing matrix is given by the generalized eigenvectors that simultaneously diagonalize the covariance matrix of the observations and an additional symmetric matrix whose form depends upon the particular assumptions. The method critically determines the mixture coefficients and is therefore not robust to estimation errors. However it provides a rather general and unified solution that summarizes the conditions for successful blind source separation. To demonstrate the method, which can be implemented in two lines of matlab code, we present results for artificial mixtures of speech and real mixtures of electroencephalography (EEG) data, showing that the same sources are recovered under the various assumptions.

  • Multi-resolution and wavelet representations for identifying signatures of disease

    October 28, 2016 | Paul Sajda, A. Laine, Y. Zeevi

    Identifying physiological and anatomical signatures of disease in signals and images is one of the fundamental challenges in biomedical engineering. The challenge is most apparent given that such signatures must be identified in spite of tremendous inter and intra-subject variability and noise. Crucial for uncovering these signatures has been the development of methods that exploit general statistical properties of natural signals. The signal processing and applied mathematics communities have developed, in recent years, signal representations which take advantage of Gabor-type and wavelet-type functions that localize signal energy in a joint time-frequency and/or space-frequency domain. These techniques can be expressed as multi-resolution transformations, of which perhaps the best known is the wavelet transform. In this paper we review wavelets, and other related multi-resolution transforms, within the context of identifying signatures for disease. These transforms construct a general representation of signals which can be used in detection, diagnosis and treatment monitoring. We present several examples where these transforms are applied to biomedical signal and imaging processing. These include computer-aided diagnosis in mammography, real-time mosaicking of ophthalmic slit-lamp imagery, characterization of heart disease via ultrasound, predicting epileptic seizures and signature analysis of the electroencephalogram, and reconstruction of positron emission tomography data.

  • Learning contextual relationships in mammograms using a hierarchical pyramid neural network

    October 28, 2016 | Paul Sajda, C. Spence, J. Pearson

    This paper describes a pattern recognition architecture, which we term hierarchical pyramid/neural network (HPNN), that learns to exploit image structure at multiple resolutions for detecting clinically significant features in digital/digitized mammograms. The HPNN architecture consists of a hierarchy of neural networks, each network receiving feature inputs at a given scale as well as features constructed by networks lower in the hierarchy. Networks are trained using a novel error function for the supervised learning of image search/detection tasks when the position of the objects to be found is uncertain or ill defined. We have evaluated the HPNN’s ability to eliminate false positive (FP) regions of interest generated by the University of Chicago’s (UofC) Computer-aided diagnosis (CAD) systems for microcalcification and mass detection. Results show that the HPNN architecture, trained using the uncertain object position (UOP) error function, reduces the FP rate of a mammographic CAD system by approximately 50% without significant loss in sensitivity. Investigation into the types of FPs that the HPNN eliminates suggests that the pattern recognizer is automatically learning and exploiting contextual information. Clinical utility is demonstrated through the evaluation of an integrated system in a clinical reader study. We conclude that the HPNN architecture learns contextual relationships between features at multiple scales and integrates these features for detecting microcalcifications and breast masses.

  • Linear Spatial Integration for Single-Trial Detection in Encephalography

    October 28, 2016 | L. Parra, C. Alvino, A. Tang, B. Pearlmutter, N. Yeung, A. Osman, Paul Sajda

    Conventional analysis of electroencephalography (EEG) and magnetoencephalography (MEG) often relies on averaging over multiple trials to extract statistically relevant differences between two or more experimental conditions. In this article we demonstrate single-trial detection by linearly integrating information over multiple spatially distributed sensors within a predefined time window. We report an average, single-trial discrimination performance of Az ≈ 0.80 and fraction correct between 0.70 and 0.80, across three distinct encephalographic data sets. We restrict our approach to linear integration, as it allows the computation of a spatial distribution of the discriminating component activity. In the present set of experiments the resulting component activity distributions are shown to correspond to the functional neuroanatomy consistent with the task (e.g., contralateral sensory–motor cortex and anterior cingulate). Our work demonstrates how a purely data-driven method for learning an optimal spatial weighting of encephalographic activity can be validated against the functional neuroanatomy.

  • Mosaicking and enhancement of slit lamp biomicroscopic fundus images

    October 28, 2016 | J. Asmuth, B. Madjarov, Paul Sajda, J. Berger

    AIMS—To process video slit lamp biomicroscopic fundus image sequences in order to generate wide field, high quality fundus image montages which might be suitable for photodocumentation. METHODS—Slit lamp biomicroscopic fundus examination was performed on human volunteers with a contact or non-contact lens. A stock, charge coupled device camera permitted image capture and storage of the image sequence at 30 frames per second. Acquisition time was approximately 30 seconds. Individual slit lamp biomicroscope fundus image frames were aligned and blended with custom developed software. RESULTS—The developed algorithms allowed for highly accurate alignment and blending of partially overlapping slit lamp biomicroscopic fundus images to generate a seamless, high quality, wide field montage. CONCLUSIONS—Video image acquisition and processing algorithms allow for mosaicking and enhancement of slit lamp biomicroscopic fundus images. The improved quality and wide field of view may confer suitability for inexpensive, real time photodocumentation of disc and macular abnormalities.

  • Coarse-grain parallel computing for very large scale neural simulations in the NEXUS simulation environment

    October 28, 2016 | K. Sakai, Paul Sajda, S.C. Yen, L. H. Finkel

    We describe a neural simulator designed for simulating very large scale models of cortical architectures. This simulator, NEXUS, uses coarse-grain parallel computing by distributing computation and data onto multiple conventional workstations connected via a local area network. Coarse-grain parallel computing offers natural advantages in simulating functionally segregated neural processes. We partition a complete model into modules with locally dense connections–a module may represent a cortical area, column, layer, or functional entity. Asynchronous data communications among workstations are established through the Network File System, which, together with the implicit modularity, decreases communications overhead, and increases overall performance. Coarse-grain parallelism also benefits from the standardization of conventional workstations and LAN, including portability between generations and vendors.

  • Integrating neural networks with image pyramids to learn target context

    October 28, 2016 | Paul Sajda, C. Spence, H. Hsu, J. Pearson

    The utility of combining neural networks with pyramid representations for target detection in aerial imagery is explored. First, it is shown that a neural network constructed using relatively simple pyramid features is a more effective detector, in terms of its sensitivity, than a network which utilizes more complex object-tuned features. Next, an architecture that supports coarse-to-fine search, context learning and data fusion is tested. The accuracy of this architecture is comparable to a more computationally expensive non-hierarchical neural network architecture, and is more accurate than a comparable conventional approach using a Fisher discriminant. Contextual relationships derived both from low-resolution imagery and supplemental data can be learned and used to improve the accuracy of detection. Such neural network/pyramid target detectors should be useful components in both user assisted search and fully automatic target recognition and monitoring systems

  • Intermediate-level visual representations and the construction of surface perception

    October 28, 2016 | Paul Sajda, L. H. Finkel

    Visual processing has often been divided into three stages—early, intermediate, and high level vision, which roughly correspond to the sensation, perception, and cognition of the visual world. In this paper, we present a network-based model of intermediate-level vision that focuses on how surfaces might be represented in visual cortex. We propose a mechanism for representing surfaces through the establishment of “ownership”—a selective binding of contours and regions. The representation of ownership provides a central locus for visual integration. Our simulations show the ability to segment real and illusory images in a manner consistent with human perception. In addition, through ownership, other processes such as depth, transparency, and surface completion can interact with one another to organize an image into a perceptual scene.

  • Constructing visual perception

    October 28, 2016 | L. H. Finkel, Paul Sajda
  • Object discrimination based on depth-from-occlusion

    October 28, 2016 | L. H. Finkel, Paul Sajda

    We present a model of how objects can be visually discriminated based on the extraction of depth-from-occlusion. Object discrimination requires consideration of both the binding problem and the problem of segmentation. We propose that the visual system binds contours and surfaces by identifying “proto-objects”-compact regions bounded by contours. Proto-objects can then be linked into larger structures. The model is simulated by a system of interconnected neural networks. The networks have biologically motivated architectures and utilize a distributed representation of depth. We present simulations that demonstrate three robust psychophysical properties of the system. The networks are able to stratify multiple occluding objects in a complex scene into separate depth planes. They bind the contours and surfaces of occluded objects (for example, if a tree branch partially occludes the moon, the two “half-moons” are bound into a single object). Finally, the model accounts for human perceptions of illusory contour stimuli.

  • Pre-stimulus functional networks modulate task performance in time-pressured evidence gathering and decision-making

    September 19, 2016 | Jason Sherwin, Jordan Muraskin, Paul Sajda

    Rapid perceptual decision-making is believed to depend upon efficient allocation of neural resources to the processing of transient stimuli within task-relevant contexts…

  • Human Scalp Potentials Reflect a Mixture of Decision-Related Signals during Perceptual Choices

    September 14, 2016 | Marios Philiastides, Hauke R. Heekeren, Paul Sajda

    Single-unit animal studies have consistently reported decision-related activity mirroring a process of temporal accumulation of sensory evidence to a fixed internal decision boundary…

  • Prestimulus EEG alpha oscillations modulate task-related fMRI BOLD responses to auditory stimuli

    September 14, 2016 | Jennifer Walz, Robin Goldman, M. Carapezza, Jordan Muraskin, Truman R. Brown, Paul Sajda

    EEG alpha-band activity is generally thought to represent an inhibitory state related to decreased attention and play a role in suppression of task-irrelevant stimulus processing, but a competing hypothesis suggests an active role in processing task-relevant information…

  • Perceptual Salience and Reward Both Influence Feedback-Related Neural Activity Arising from Choice

    September 13, 2016 | Bin Lou, Wha-Yin Hsu, Paul Sajda

    For day-to-day decisions, multiple factors influence our choice between alternatives. Two dimensions of decision making that substantially affect choice are the objective perceptual properties of the stimulus (e.g., salience) and its subjective value….

  • Knowing when not to swing: EEG evidence that enhanced perception–action coupling underlies baseball batter expertise

    September 13, 2016 | Jordan Muraskin, Jason Sherwin, Paul Sajda

    Given a decision that requires less than half a second for evaluating the characteristics of the incoming pitch and generating a motor response,…

  • Automated Tissue Segmentation and Blind Recovery of 1H MRSI Spectral Patterns of Normal and Diseased Human Brain

    December 14, 2008 | S. Du, X. Mao, Paul Sajda, D.C. Shungu

    Constrained non-negative matrix factorization (cNMF) with iterative data selection is described and demonstrated as a data analysis method for fast and automatic recovery of biochemically meaningful and diagnostically specific spectral patterns of the human brain from 1H MRS imaging (1H MRSI) data. To achieve this goal, cNMF decomposes in vivo multidimensional 1H MRSI data into two non-negative matrices representing (a) the underlying tissue-specific spectral patterns and (b) the spatial distribution of the corresponding metabolite concentrations. Central to the proposed approach is automatic iterative data selection which uses prior knowledge about the spatial distribution of the spectra to remove voxels that are due to artifacts and undesired metabolites/tissues such as the strong lipid and water components. The automatic recovery of diagnostic spectral patterns is demonstrated for long-TE1H MRSI data on normal human brain, multiple sclerosis, and serial brain tumor. The results show the ability of cNMF with iterative data selection to automatically and simultaneously recover tissue-specific spectral patterns and achieve segmentation of normal and diseased human brain tissue, concomitant with simplification of information content. These features of cNMF, which permit rapid recovery, reduction and interpretation of the complex diagnostic information content of large multi-dimensional spectroscopic imaging data sets, have the potential to enhance the clinical utility of in vivo1H MRSI.

  • Cortical origins of response time variability during rapid discrimination of visual objects

    January 1, 2005 | Adam Gerson, L. Parra, Paul Sajda

    In this paper, we use single-trial analysis of electroencephalography (EEG) to ascertain the cortical origins of response time variability in a rapid serial visual presentation (RSVP) task. We extract spatial components that maximally discriminate between target and distractor stimulus conditions over specific time windows between stimulus onset and the time of a motor response. We then compute the peak latency of this differential activity on a trial-by-trial basis, and correlate this with response time. We find, for our nine participants, that the majority of the latency is introduced by component activity which begins far-frontally 200 ms prior to the response and proceeds to become parietally distributed near the time of response. This activity is consistent with the hypothesis that cortical networks involved in generating the late positive complexes may be the origins of the observed response time variability in rapid discrimination of visual objects.

  • NEXUS: A simulation environment for large-scale neural systems

    December 1, 1992 | Paul Sajda, L. H. Finkel

    NEXUS is a novel simulation environment designed for modeling large neural systems. The simulator allows the user to incorporate complex functional properties and symbolic processing at the level of the individual unit. In addition, NEXUS takes advantage of the principle of topographic map organization, found throughout the mammalian nervous system, to facilitate the modeling and design process. An easy-to-use graphical interface allows the user to interactively build and test models. This paper describes the principles underlying the NEXUS design and its advantages over other current simulation approaches.

  • Simulating biological vision with hybrid neural networks.

    July 1, 1992 | Paul Sajda, L. H. Finkel

    We present an example of how vision systems can be modeled and designed by integrating a top-down computationally- based approach with a bottom-up biologi cally-motivated architecture. The specific visual processing task we address is occlusion-based object segmentation — the discrimination of objects using cues derived from object interposition. We construct a model of object segmentation using hybrid neural networks—distributed parallel systems consisting of neural units modeled at different levels af abstraction. We show that such networks are particularly useful for systems which can be modeled using the combined top-down/bottom-up approach. Our hybrid model is capable of discriminat ing objects and stratifying them in relative depth. In addition, our system can account for several classes of human perceptual phenomena, such as illusory contours. We conclude that hybrid systems serve as a powerful paradigm for understanding the information processing strategies of biological vision and for constructing artificial vision-based applications.