Micro magnetic resonance imaging (µMRI) is an in vivo imaging method that permits 3D quantification of cortical and trabecular bone microstructure. µMR images can also be used for building microstructural finite element (µFE) models to assess bone stiffness, which highly correlates with bone’s resistance to fractures. In order for µMRI-based microstructural and µFE analyses to become standard clinical tools for assessing bone quality, validation with a current gold standard, namely, high-resolution micro computed tomography (µCT), is required. Microstructural measurements of 25 human cadaveric distal tibias were performed for the registered µMR and µCT images, respectively. Next, whole bone stiffness, trabecular bone stiffness, and elastic moduli of cubic subvolumes of trabecular bone in both µMR and µCT images were determined by voxel-based µFE analysis. The bone volume fraction (BV/TV), trabecular number (Tb.N*), trabecular spacing (Tb.Sp*), cortical thickness (Ct.Th), and structure model index (SMI) based on µMRI showed strong correlations with µCT measurements (r2 = 0.67 to 0.97), and bone surface-to-volume ratio (BS/BV), connectivity density (Conn.D), and degree of anisotropy (DA) had significant but moderate correlations (r2 = 0.33 to 0.51). Each of these measurements also contributed to one or many of the µFE-predicted mechanical properties. However, model-independent trabecular thickness (Tb.Th*) based on µMRI had no correlation with the µCT measurement and did not contribute to any mechanical measurement. Furthermore, the whole bone and trabecular bone stiffness based on µMRI were highly correlated with those of µCT images (r2 = 0.86 and 0.96), suggesting that µMRI-based µFE analyses can directly and accurately quantify whole bone mechanical competence. In contrast, the elastic moduli of the µMRI trabecular bone subvolume had significant but only moderate correlations with their gold standards (r2 = 0.40 to 0.58). We conclude that most microstructural and mechanical properties of the distal tibia can be derived efficiently from µMR images and can provide additional information regarding bone quality. © 2010 American Society for Bone and Mineral Research.
The 19 papers in this special issue cover a variety of brain-machine/computer interfaces and include: 1) monitoring and data recording methods, 2) spike-based detection techniques, 3) adaptive/automatic biosignal processing approaches, 4) bioamplifiers implementation and validation, 5) ambulatory/implantable electronics, 6) electronics for brain science, and vision-based interfaces, 7) electrical stimulation, and 8) experimental and clinical case studies.
Mechanical stimuli can trigger intracellular calcium (Ca2 +) responses in osteocytes and osteoblasts. Successful construction of bone cell networks necessitates more elaborate and systematic analysis for the spatiotemporal properties of Ca2 + signaling in the networks. In the present study, an unsupervised algorithm based on independent component analysis (ICA) was employed to extract the Ca2 + signals of bone cells in the network. We demonstrated that the ICA-based technology could yield higher signal fidelity than the manual region of interest (ROI) method. Second, the spatiotemporal properties of Ca2 + signaling in osteocyte-like MLO-Y4 and osteoblast-like MC3T3-E1 cell networks under laminar and steady fluid flow stimulation were systematically analyzed and compared. MLO-Y4 cells exhibited much more active Ca2 + transients than MC3T3-E1 cells, evidenced by more Ca2 + peaks, less time to the 1st peak and less time between the 1st and 2nd peaks. With respect to temporal properties, MLO-Y4 cells demonstrated higher spike rate and Ca2 + oscillating frequency. The spatial intercellular synchronous activities of Ca2 + signaling in MLO-Y4 cell networks were higher than those in MC3T3-E1 cell networks and also negatively correlated with the intercellular distance, revealing faster Ca2 + wave propagation in MLO-Y4 cell networks. Our findings show that the unsupervised ICA-based technique results in more sensitive and quantitative signal extraction than traditional ROI analysis, with the potential to be widely employed in Ca2 + signaling extraction in the cell networks. The present study also revealed a dramatic spatiotemporal difference in Ca2 + signaling for osteocytic and osteoblastic cell networks in processing the mechanical stimulus. The higher intracellular Ca2 + oscillatory behaviors and intercellular coordination of MLO-Y4 cells provided further evidences that osteocytes may behave as the major mechanical sensor in bone modeling and remodeling processes.
Brain-computer interface (BCI) technologies, or technologies that use online brain signal processing, have a great promise to improve human interactions with computers, their environment, and even other humans. Despite this promise, there are no current serious BCI technologies in widespread use, due to the lack of robustness in BCI technologies. The key neural aspect of this lack of robustness is human variability, which has two main components: (1) individual differences in neural signals and (2) intraindividual variability over time. In order to develop widespread BCI technologies, it will be necessary to address this lack of robustness. However, it is currently unknown how neural variability affects BCI performance. To accomplish these goals, it is essential to obtain data from large numbers of individuals using BCI technologies over considerable lengths of time. One promising method for this is through the use of BCI technologies embedded into games with a purpose (GWAP). GWAP are a game-based form of crowdsourcing which players choose to play for enjoyment and during which the player performs key tasks which cannot be automated but that are required to solve research questions. By embedding BCI paradigms in GWAP and recording neural and behavioral data, it should be possible to much more clearly understand the differences in neural signals between individuals and across different time scales, enabling the development of novel and increasingly robust adaptive BCI algorithms.
Purpose We have shown how hyperspectral imaging can extract individual signals of abundant fluorophores ex-vivo in normal RPE . Herein we analyze donor eyes with AMD, hypothesizing that the spectral signatures would vary with perturbations in RPE physiology. Methods Hyperspectral AF images were captured from 7 locations in 5 RPE/Bruch’s-membrane (BrM) flat-mounts from donor eyes with early to late AMD. Imaging was performed at 2 excitation bands, 436-460 and 480-510 nm, with emission captured between 420 and 720 nm in 10 nm intervals using a Nuance FX camera. Results Gaussian mixture modeling and mathematical factorization were applied to extract 1 BrM spectrum and 4 abundant emission spectra from RPE organelles, the latter peaking at mean wavelengths of 519±7, 574±8, 599±4, and 644±10 nm (436 nm excitation). The 519 nm peak was blue-shifted ~50 nm relative to the corresponding signal from normal eyes . Spatial abundance images showed unique signals localized to RPE granule aggregates, melanosomes, and basal laminar deposits (BLamD) (Figure). Conclusions In AMD, some AF signals from RPE become spatially discrete , and a morphological diversity of RPE granule populations is well demonstrated using hyperspectral imaging. Differences in spectra and their spatial distributions between normal and AMD eyes may extend our knowledge of RPE pathophysiology in AMD. 1. Smith, Post, Johri, Lee, Ablonczy, Curcio, Ach, Sajda: Hyperspectral signal recovery of unknown fluorophores in the RPE. Biomed Opt Express 2014. 2. Ach, ARVO 2014
Purpose Drusen are one of the hallmark lesions of AMD. Our purpose is to characterize drusen by their spectral AF signatures, for a better molecular understanding of AMD. Methods Retinal pigment epithelium (RPE)/Bruch’s membrane (BrM)-flatmounts were prepared from 5 donors with AMD. Hyperspectral AF imaging was performed at 11 locations containing drusen at 2 excitation bands, 436-460 and 480-510 nm, with emissions captured between 420 and 700 nm in 10 nm intervals, with the Nuance FX camera. Abundant individual spectra with corresponding spatial localizations were recovered with custom software based on a nonnegative tensor factorization algorithm . Results In a typical example the original AF image (RGB) is presented with 5 spatial abundances (Fig. 1) of 5 recovered individual spectra (Fig. 2). In particular, spectrum C4 (Fig. 2, cyan line) localizes precisely and specifically to the drusen and also a diffuse region (upper left Fig. 1, abundance C4) with a peak wavelength of 510 nm. A spectrum with this characteristic peak and shape was recovered from all drusen and variably from diffuse regions in 10 of the 11 locations, with one spectrum peaking at 490 nm. Further, the dominant fluorophore (spectrum C1 and abundance C1 herein) consistently shows moderate co-localization with drusen, indicating this fluorophore overlying as granules and/or within drusen. This fluorophore spectrum is remarkably blue-shifted about 50 nm from the corresponding dominant fluorophore spectrum recovered from previously reported normal donors . Conclusions With hyperspectral AF image analysis, we found a single spectral AF signature for drusen and diffuse sub-RPE deposits in AMD that appears to be highly sensitive for drusen. The possibility that this signal also localizes to basal linear deposit is being addressed. A second drusen-associated AF spectral signature shares commonalities with RPE lipofuscin. These hyperspectral AF characterizations of drusen may aid the molecular understanding of AMD.
Perceptual decisions depend on coordinated patterns of neural activity cascading across the brain, running in time from stimulus to response and in space from primary sensory regions to the frontal lobe. Measuring this cascade and how it flows through the brain is key to developing an understanding of how our brains function. However observing, let alone understanding, this cascade, particularly in humans, is challenging. Here, we report a significant methodological advance allowing this observation in humans at unprecedented spatiotemporal resolution. We use a novel encoding model to link simultaneously measured electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) signals to infer the high-resolution spatiotemporal brain dynamics taking place during rapid visual perceptual decision-making. After demonstrating the methodology replicates past results, we show that it uncovers a previously unobserved sequential reactivation of a substantial fraction of the pre-response network whose magnitude correlates with decision confidence. Our results illustrate that a temporally coordinated and spatially distributed neural cascade underlies perceptual decision-making, with our methodology illuminating complex brain dynamics that would otherwise be unobservable using conventional fMRI or EEG separately. We expect this methodology to be useful in observing brain dynamics in a wide range of other mental processes.
In neuroscience, stimulus-response relationships have traditionally been analyzed using either encoding or decoding models. Here we combined both techniques by decomposing neural activity into multiple components, each representing a portion of the stimulus. We tested this hybrid approach on encephalographic responses to auditory and audiovisual narratives identically experienced across subjects, as well as uniquely experienced video game play. The highest stimulus-response correlations (SRC) were detected for dynamic visual features. During narratives both auditory and visual SRC were modulated by attention and tracked correlations between subjects. During video game play, SRC was modulated by task difficulty and attentional state. Importantly, the strongest component extracted for visual and auditory features had nearly identical spatial distributions, suggesting that the predominant encephalographic response to naturalistic stimuli is supramodal. The variety of novel findings demonstrates the utility of measuring multidimensional stimulus-response correlations.
Objective. We investigated the neural correlates of workload buildup in a fine visuomotor task called the boundary avoidance task (BAT). The BAT has been known to induce naturally occurring failures of human–machine coupling in high performance aircraft that can potentially lead to a crash—these failures are termed pilot induced oscillations (PIOs). Approach. We recorded EEG and pupillometry data from human subjects engaged in a flight BAT simulated within a virtual 3D environment. Main results. We find that workload buildup in a BAT can be successfully decoded from oscillatory features in the electroencephalogram (EEG). Information in delta, theta, alpha, beta, and gamma spectral bands of the EEG all contribute to successful decoding, however gamma band activity with a lateralized somatosensory topography has the highest contribution, while theta band activity with a fronto-central topography has the most robust contribution in terms of real-world usability. We show that the output of the spectral decoder can be used to predict PIO susceptibility. We also find that workload buildup in the task induces pupil dilation, the magnitude of which is significantly correlated with the magnitude of the decoded EEG signals. These results suggest that PIOs may result from the dysregulation of cortical networks such as the locus coeruleus (LC)—anterior cingulate cortex (ACC) circuit. Significance. Our findings may generalize to similar control failures in other cases of tight man-machine coupling where gains and latencies in the control system must be inferred and compensated for by the human operators. A closed-loop intervention using neurophysiological decoding of workload buildup that targets the LC-ACC circuit may positively impact operator performance in such situations.
Post-task resting state dynamics can be viewed as a task-driven state where behavioral performance is improved through endogenous, non-explicit learning. Tasks that have intrinsic value for individuals are hypothesized to produce post-task resting state dynamics that promote learning. We measured simultaneous fMRI/EEG and DTI in Division-1 collegiate baseball players and compared to a group of controls, examining differences in both functional and structural connectivity. Participants performed a surrogate baseball pitch Go/No-Go task before a resting state scan, and we compared post-task resting state connectivity using a seed-based analysis from the supplementary motor area (SMA), an area whose activity discriminated players and controls in our previous results using this task. Although both groups were equally trained on the task, the experts showed differential activity in their post-task resting state consistent with motor learning. Specifically, we found (1) differences in bilateral SMA–L Insula functional connectivity between experts and controls that may reflect group differences in motor learning, (2) differences in BOLD-alpha oscillation correlations between groups suggests variability in modulatory attention in the post-task state, and (3) group differences between BOLD-beta oscillations that may indicate cognitive processing of motor inhibition. Structural connectivity analysis identified group differences in portions of the functionally derived network, suggesting that functional differences may also partially arise from variability in the underlying white matter pathways. Generally, we find that brain dynamics in the post-task resting state differ as a function of subject expertise and potentially result from differences in both functional and structural connectivity.
In the last few decades, noninvasive neuroimaging has revealed macroscale brain dynamics that underlie perception, cognition, and action. Advances in noninvasive neuroimaging target two capabilities: 1) increased spatial and temporal resolution of measured neural activity; and 2) innovative methodologies to extract brain–behavior relationships from evolving neuroimaging technology. We target the second. Our novel methodology integrated three neuroimaging methodologies and elucidated expertise-dependent differences in functional (fused EEG-fMRI) and structural (dMRI) brain networks for a perception–action coupling task. A set of baseball players and controls performed a Go/No-Go task designed to mimic the situation of hitting a baseball. In the functional analysis, our novel fusion methodology identifies 50-ms windows with predictive EEG neural correlates of expertise and fuses these temporal windows with fMRI activity in a whole-brain 2-mm voxel analysis, revealing time-localized correlations of expertise at a spatial scale of millimeters. The spatiotemporal cascade of brain activity reflecting expertise differences begins as early as 200 ms after the pitch starts and lasts up to 700 ms afterwards. Network differences are spatially localized to include motor and visual processing areas, providing evidence for differences in perception–action coupling between the groups. Furthermore, an analysis of structural connectivity reveals that the players have significantly more connections between cerebellar and left frontal/motor regions, and many of the functional activation differences between the groups are located within structurally defined network modules that differentiate expertise. In short, our novel method illustrates how multimodal neuroimaging can provide specific macroscale insights into the functional and structural correlates of expertise development.
Recent advances in signal processing and machine learning techniques have enabled the application of Brain-Computer Interface (BCI) technologies to fields such as medicine, industry, and recreation; however, BCIs still suffer from the requirement of frequent calibration sessions due to the intra- and inter-individual variability of brain-signals, which makes calibration suppression through transfer learning an area of increasing interest for the development of practical BCI systems. In this paper, we present an unsupervised transfer method (spectral transfer using information geometry, STIG), which ranks and combines unlabeled predictions from an ensemble of information geometry classifiers built on data from individual training subjects. The STIG method is validated in both off-line and real-time feedback analysis during a rapid serial visual presentation task (RSVP). For detection of single-trial, event-related potentials (ERPs), the proposed method can significantly outperform existing calibration-free techniques as well as outperform traditional within-subject calibration techniques when limited data is available. This method demonstrates that unsupervised transfer learning for single-trial detection in ERP-based BCIs can be achieved without the requirement of costly training data, representing a step-forward in the overall goal of achieving a practical user-independent BCI system.
Upon excitation with different wavelengths of light, biological tissues emit distinct but related autofluorescence signals. We used non-negative matrix factorization (NMF) to simultaneously decompose co-registered hyperspectral emission data from human retinal pigment epithelium/Bruch’s membrane specimens illuminated with 436 and 480 nm light. NMF analysis was initialized with Gaussian mixture model fits and constrained to provide identical abundance images for the two excitation wavelengths. Spectra recovered this way were smoother than those obtained separately; fluorophore abundances more clearly localized within tissue compartments. These studies provide evidence that leveraging multiple co-registered hyperspectral emission data sets is preferential for identifying biologically relevant fluorophore information.
Background As neuroscientists endeavor to understand the brain’s response to ecologically valid scenarios, many are leaving behind hyper-controlled paradigms in favor of more realistic ones. This movement has made the use of 3D rendering software an increasingly compelling option. However, mastering such software and scripting rigorous experiments requires a daunting amount of time and effort. New method To reduce these startup costs and make virtual environment studies more accessible to researchers, we demonstrate a naturalistic experimental design environment (NEDE) that allows experimenters to present realistic virtual stimuli while still providing tight control over the subject’s experience. NEDE is a suite of open-source scripts built on the widely used Unity3D game development software, giving experimenters access to powerful rendering tools while interfacing with eye tracking and EEG, randomizing stimuli, and providing custom task prompts. Results Researchers using NEDE can present a dynamic 3D virtual environment in which randomized stimulus objects can be placed, allowing subjects to explore in search of these objects. NEDE interfaces with a research-grade eye tracker in real-time to maintain precise timing records and sync with EEG or other recording modalities. Comparison with existing methods Python offers an alternative for experienced programmers who feel comfortable mastering and integrating the various toolboxes available. NEDE combines many of these capabilities with an easy-to-use interface and, through Unity’s extensive user base, a much more substantial body of assets and tutorials. Conclusions Our flexible, open-source experimental design system lowers the barrier to entry for neuroscientists interested in developing experiments in realistic virtual environments.
In contrast to static imagery, detection of events of interest in video involves evidence accumulation across space and time; the observer is required to integrate features from both motion and form to decide whether a behavior constituents a target event. Do such events that extend in time elicit evoked responses of similar strength as evoked responses associated with instantaneous events such as the presentation of a static target image? Using a set of simulated scenarios, with avatars/actors having different behaviors, we identified evoked neural activity discriminative of target vs. distractor events (behaviors) at discrimination levels that are comparable to static imagery. EEG discriminative activity was largely in the time-locked evoked response and not in oscillatory activity, with the exception of very low EEG frequency bands such as delta and theta, which simply represent bands dominating the event related potential (ERP). The discriminative evoked response activity we see is observed in all target/distractor conditions and is robust across different recordings from the same subjects. The results suggest that we have identified a robust neural correlate of target detection in video, at least in terms of the stimulus set we used—i.e., dynamic behavior of an individual in a low clutter environment. Additional work is needed to test a larger variety of behaviors and more diverse environments.
OBJECTIVE: As we move through an environment, we are constantly making assessments, judgments and decisions about the things we encounter. Some are acted upon immediately, but many more become mental notes or fleeting impressions-our implicit ‘labeling’ of the world. In this paper, we use physiological correlates of this labeling to construct a hybrid brain-computer interface (hBCI) system for efficient navigation of a 3D environment. APPROACH: First, we record electroencephalographic (EEG), saccadic and pupillary data from subjects as they move through a small part of a 3D virtual city under free-viewing conditions. Using machine learning, we integrate the neural and ocular signals evoked by the objects they encounter to infer which ones are of subjective interest to them. These inferred labels are propagated through a large computer vision graph of objects in the city, using semi-supervised learning to identify other, unseen objects that are visually similar to the labeled ones. Finally, the system plots an efficient route to help the subjects visit the ‘similar’ objects it identifies. MAIN RESULTS: We show that by exploiting the subjects’ implicit labeling to find objects of interest instead of exploring naively, the median search precision is increased from 25% to 97%, and the median subject need only travel 40% of the distance to see 84% of the objects of interest. We also find that the neural and ocular signals contribute in a complementary fashion to the classifiers’ inference of subjects’ implicit labeling. SIGNIFICANCE: In summary, we show that neural and ocular signals reflecting subjective assessment of objects in a 3D environment can be used to inform a graph-based learning model of that environment, resulting in an hBCI system that improves navigation and information delivery specific to the user’s interests.
Pupillary measures have been linked to arousal and attention as well as activity in the brainstem’s locus coeruleus norepinephrine (LC-NE) system. Similarly, there is evidence that evoked EEG responses, such as the P3, might have LC-NE activity as their basis. Since it is not feasible to record electrophysiological data directly from the LC in humans due to its location in the brainstem, an open question has been whether pupillary measures and EEG variability can be linked in a meaningful way to shed light on the nature of the LC-NE role in attention and arousal. We used an auditory oddball task with a data-driven approach to learn task-relevant projections of the EEG, for windows of data spanning the entire trial. We investigated linear and quadratic relationships between the evoked EEG along these projections and both prestimulus (baseline) and poststimulus (evoked dilation) pupil diameter measurements. We found that baseline pupil diameter correlates with early (175–200 ms) and late (350–400 ms) EEG component variability, suggesting a linear relationship between baseline (tonic) LC-NE activity and evoked EEG. We found no relationships between evoked EEG and evoked pupil dilation, which is often associated with evoked (phasic) LC activity. After regressing out reaction time (RT), the correlation between EEG variability and baseline pupil diameter remained, suggesting that such correlation is not explainable by RT variability. We also investigated the relationship between these pupil measures and prestimulus EEG alpha activity, which has been reported as a marker of attentional state, and found a negative linear relationship with evoked pupil dilation. In summary, our results demonstrate significant relationships between prestimulus and poststimulus neural and pupillary measures, and they provide further evidence for tight coupling between attentional state and evoked neural activity and for the role of cortical and subcortical networks underlying the process of target detection.
Pre-stimulus α power has been shown to correlate with the behavioral accuracy of perceptual decisions. In most cases, these correlations have been observed by comparing α power for different behavioral outcomes (e.g. correct vs incorrect trials). In this paper we investigate such covariation within the context of behaviorally-latent fluctuations in task-relevant post-stimulus neural activity. Specially we consider variations of pre-stimulus α power with post-stimulus EEG components in a two alternative forced choice visual discrimination task. EEG components, discriminative of stimulus class, are identified using a linear multivariate classifier and only the variability of the components for correct trials (regardless of stimulus class, and for nominally identical stimuli) are correlated with the corresponding pre-stimulus α power. We find a significant relationship between the mean and variance of the pre-stimulus α power and the variation of the trial-to-trial magnitude of an early post-stimulus EEG component. This relationship is not seen for a later EEG component that is also discriminative of stimulus class and which has been previously linked to the quality of evidence driving the decision process. Our results suggest that early perceptual representations, rather than temporally later neural correlates of the perceptual decision, are modulated by pre-stimulus state.
Focused attention continuously and inevitably fluctuates, and to completely understand the mechanisms responsible for these modulations it is necessary to localize the brain regions involved. During a simple visual oddball task, neural responses measured by electroencephalography (EEG) modulate primarily with attention, but source localization of the correlates is a challenge. In this study we use single-trial analysis of simultaneously-acquired scalp EEG and functional magnetic resonance image (fMRI) data to investigate the blood oxygen level dependent (BOLD) correlates of modulations in task-related attention, and we unravel the temporal cascade of these transient activations. We hypothesize that activity in brain regions associated with various task-related cognitive processes modulates with attention, and that their involvements occur transiently in a specific order. We analyze the fMRI BOLD signal by first regressing out the variance linked to observed stimulus and behavioral events. We then correlate the residual variance with the trial-to-trial variation of EEG discriminating components for identical stimuli, estimated at a sequence of times during a trial. Post-stimulus and early in the trial, we find activations in right-lateralized frontal regions and lateral occipital cortex, areas that are often linked to task-dependent processes, such as attentional orienting, and decision certainty. After the behavioral response we see correlates in areas often associated with the default-mode network and introspective processing, including precuneus, angular gyri, and posterior cingulate cortex. Our results demonstrate that during simple tasks both task-dependent and default-mode networks are transiently engaged, with a distinct temporal ordering and millisecond timescale.
This report summarizes the outcomes of the NSF Workshop on Mapping and Engineering the Brain, held at Arlington, VA, during August 13-14, 2013. Three grand challenges were identified, including high spatiotemporal resolution neuroimaging, perturbation-based neuroimaging, and neuroimaging in naturalistic environments. It was highlighted that each grand challenge requires groundbreaking discoveries, enabling technologies, appropriate knowledge transfer, and multi- and transdisciplinary education and training for success.
When scanning a scene, the target of our search may be in plain sight and yet remain unperceived. Conversely, at other times the target may be perceived in the periphery prior to fixation. There is ample behavioral and neurophysiological evidence to suggest that in some constrained visual-search tasks, targets are detected prior to fixational eye movements. However, limited human data are available during unconstrained search to determine the time course of detection, the brain areas involved, and the neural correlates of failures to detect a foveated target. Here, we recorded and analyzed electroencephalographic (EEG) activity during free-viewing visual search, varying the task difficulty to compare neural signatures for detected and unreported (“missed”) targets. When carefully controlled to remove eye-movement-related potentials, saccade-locked EEG shows that: (a) “Easy” targets may be detected as early as 150 ms prior to foveation, as indicated by a premotor potential associated with a button response; (b) object-discriminating occipital activity emerges during the saccade to target; and (c) success and failures to detect a target are accompanied by a modulation in alpha-band power over fronto-central areas as well as altered saccade dynamics. Taken together, these data suggest that target detection during free viewing can begin prior to and continue during a saccade, with failure or success in reporting a target possibly resulting from inhibition or activation of fronto-central processing areas associated with saccade control.
Multivariate decoding models are increasingly being applied to functional magnetic imaging (fMRI) data to interpret the distributed neural activity in the human brain. These models are typically formulated to optimize an objective function that maximizes decoding accuracy. For decoding models trained on full-brain data, this can result in multiple models that yield the same classification accuracy, though some may be more reproducible than others—i.e. small changes to the training set may result in very different voxels being selected. This issue of reproducibility can be partially controlled by regularizing the decoding model. Regularization, along with the cross-validation used to estimate decoding accuracy, typically requires retraining many (often on the order of thousands) of related decoding models. In this paper we describe an approach that uses a combination of bootstrapping and permutation testing to construct both a measure of cross-validated prediction accuracy and model reproducibility of the learned brain maps. This requires re-training our classification method on many re-sampled versions of the fMRI data. Given the size of fMRI datasets, this is normally a time-consuming process. Our approach leverages an algorithm called fast simultaneous training of generalized linear models (FaSTGLZ) to create a family of classifiers in the space of accuracy vs. reproducibility. The convex hull of this family of classifiers can be used to identify a subset of Pareto optimal classifiers, with a single-optimal classifier selectable based on the relative cost of accuracy vs. reproducibility. We demonstrate our approach using full-brain analysis of elastic-net classifiers trained to discriminate stimulus type in an auditory and visual oddball event-related fMRI design. Our approach and results argue for a computational approach to fMRI decoding models in which the value of the interpretation of the decoding model ultimately depends upon optimizing a joint space of accuracy and reproducibility.
Cortical and subcortical networks have been identified that are commonly associated with attention and task engagement, along with theories regarding their functional interaction. However, a link between these systems has not yet been demonstrated in healthy humans, primarily because of data acquisition and analysis limitations. We recorded simultaneous EEG–fMRI while subjects performed auditory and visual oddball tasks and used these data to investigate the BOLD correlates of single-trial EEG variability at latencies spanning the trial. We focused on variability along task-relevant dimensions in the EEG for identical stimuli and then combined auditory and visual data at the subject level to spatially and temporally localize brain regions involved in endogenous attentional modulations. Specifically, we found that anterior cingulate cortex (ACC) correlates strongly with both early and late EEG components, whereas brainstem, right middle frontal gyrus (rMFG), and right orbitofrontal cortex (rOFC) correlate significantly only with late components. By orthogonalizing with respect to event-related activity, we found that variability in insula and temporoparietal junction is reflected in reaction time variability, rOFC and brainstem correlate with residual EEG variability, and ACC and rMFG are significantly correlated with both. To investigate interactions between these correlates of temporally specific EEG variability, we performed dynamic causal modeling (DCM) on the fMRI data. We found strong evidence for reciprocal effective connections between the brainstem and cortical regions. Our results support the adaptive gain theory of locus ceruleus–norepinephrine (LC–NE) function and the proposed functional relationship between the LC–NE system, right-hemisphere ventral attention network, and P300 EEG response.
Humans are extremely good at detecting anomalies in sensory input. For example, while listening to a piece of Western-style music, an anomalous key change or an out-of-key pitch is readily apparent, even to the non-musician. In this paper we investigate differences between musical experts and non-experts during musical anomaly detection. Specifically, we analyzed the electroencephalograms (EEG) of five expert cello players and five non-musicians while they listened to excerpts of J.S. Bach’s Prelude from Cello Suite No. 1. All subjects were familiar with the piece, though experts also had extensive experience playing the piece. Subjects were told that anomalous musical events (AMEs) could occur at random within the excerpts of the piece and were told to report the number of AMEs after each excerpt. Furthermore, subjects were instructed to remain still while listening to the excerpts and their lack of movement was verified via visual and EEG monitoring. Experts had significantly better behavioral performance (i.e. correctly reporting AME counts) than non-experts, though both groups had mean accuracies greater than 80%. These group differences were also reflected in the EEG correlates of key-change detection post-stimulus, with experts showing more significant, greater magnitude, longer periods of, and earlier peaks in condition-discriminating EEG activity than novices. Using the timing of the maximum discriminating neural correlates, we performed source reconstruction and compared significant differences between cellists and non-musicians. We found significant differences that included a slightly right lateralized motor and frontal source distribution. The right lateralized motor activation is consistent with the cortical representation of the left hand – i.e. the hand a cellist would use, while playing, to generate the anomalous key-changes. In general, these results suggest that sensory anomalies detected by experts may in fact be partially a result of an embodied cognition, with a model of the action for generating the anomaly playing a role in its detection.
Sparse coding has been posited as an efficient information processing strategy employed by sensory systems, particularly visual cortex. Substantial theoretical and experimental work has focused on the issue of sparse encoding, namely how the early visual system maps the scene into a sparse representation. In this paper we investigate the complementary issue of sparse decoding, for example given activity generated by a realistic mapping of the visual scene to neuronal spike trains, how do downstream neurons best utilize this representation to generate a “decision.” Specifically we consider both sparse (L1-regularized) and non-sparse (L2 regularized) linear decoding for mapping the neural dynamics of a large-scale spiking neuron model of primary visual cortex (V1) to a two alternative forced choice (2-AFC) perceptual decision. We show that while both sparse and non-sparse linear decoding yield discrimination results quantitatively consistent with human psychophysics, sparse linear decoding is more efficient in terms of the number of selected informative dimension.
We investigate the modulation of post-stimulus endogenous and exogenous oscillations when a visual discrimination is made more difficult. We use exogenous frequency tagging to induce steady-state visually evoked potentials (SSVEP) while subjects perform a face-car discrimination task, the difficulty of which varies on a trial-to-trial basis by varying the noise (phase coherence) in the image. We simultaneously analyze amplitude modulations of the SSVEP and endogenous alpha activity as a function of task difficulty. SSVEP modulation can be viewed as a neural marker of attention toward/away from the primary task, while modulation of post-stimulus alpha is closely related to cortical information processing. We find that as the task becomes more difficult, the amplitude of SSVEP decreases significantly, approximately 250-450 ms post-stimulus. Significant changes in endogenous alpha amplitude follow SSVEP modulation, occurring at approximately 400-700 ms post-stimulus and, unlike the SSVEP, the alpha amplitude is increasingly suppressed as the task becomes less difficult. Our results demonstrate simultaneous measurement of endogenous and exogenous oscillations that are modulated by task difficulty, and that the specific timing of these modulations likely reflects underlying information processing flow during perceptual decision-making.
Group level statistical maps of blood oxygenation level dependent (BOLD) signals acquired using functional magnetic resonance imaging (fMRI) have become a basic measurement for much of systems, cognitive and social neuroscience. A challenge in making inferences from these statistical maps is the noise and potential confounds that arise from the head motion that occurs within and between acquisition volumes. This motion results in the scan plane being misaligned during acquisition, ultimately leading to reduced statistical power when maps are constructed at the group level. In most cases, an attempt is made to correct for this motion through the use of retrospective analysis methods. In this paper, we use a prospective active marker motion correction (PRAMMO) system that uses radio frequency markers for real-time tracking of motion, enabling on-line slice plane correction. We show that the statistical power of the activation maps is substantially increased using PRAMMO compared to conventional retrospective correction. Analysis of our results indicates that the PRAMMO acquisition reduces the variance without decreasing the signal component of the BOLD (beta). Using PRAMMO could thus improve the overall statistical power of fMRI based BOLD measurements, leading to stronger inferences of the nature of processing in the human brain.
Hitting a baseball is often described as the most difficult thing to do in sports. A key aptitude of a good hitter is the ability to determine which pitch is coming. This rapid decision requires the batter to make a judgment in a fraction of a second based largely on the trajectory and spin of the ball. When does this decision occur relative to the ball’s trajectory and is it possible to identify neural correlates that represent how the decision evolves over a split second? Using single-trial analysis of electroencephalography (EEG) we address this question within the context of subjects discriminating three types of pitches (fastball, curveball, slider) based on pitch trajectories. We find clear neural signatures of pitch classification and, using signal detection theory, we identify the times of discrimination on a trial-to-trial basis. Based on these neural signatures we estimate neural discrimination distributions as a function of the distance the ball is from the plate. We find all three pitches yield unique distributions, namely the timing of the discriminating neural signatures relative to the position of the ball in its trajectory. For instance, fastballs are discriminated at the earliest points in their trajectory, relative to the two other pitches, which is consistent with the need for some constant time to generate and execute the motor plan for the swing (or inhibition of the swing). We also find incorrect discrimination of a pitch (errors) yields neural sources in Brodmann Area 10, which has been implicated in prospective memory, recall, and task difficulty. In summary, we show that single-trial analysis of EEG yields informative distributions of the relative point in a baseball’s trajectory when the batter makes a decision on which pitch is coming.
This article summarizes a recent panel discussion at the ACM International Conference on Multimedia Retrieval, where a case was made for making the interacting user a first-class citizen again in multimedia retrieval research.
A commentary on Trial-by-trial variations in subjective attentional state are reflected in ongoing prestimulus EEG alpha oscillations by Macdonald, J. S. P., Mathan, S., and Yeung, N. (2011). Front. Percept. Sci. 2:82. doi: 10.3389/fpsyg.2011.00082
Recent evidence from functional magnetic resonance imaging suggests that cortical hemo- dynamic responses coincide in different subjects experiencing a common naturalistic stimulus. Here we utilize neural responses in the electroencephalogram (EEG) evoked by multiple presentations of short film clips to index brain states marked by high levels of corre- lation within and across subjects.We formulate a novel signal decomposition method which extracts maximally correlated signal components from multiple EEG records.The resulting components capture correlations down to a one-second time resolution, thus revealing that peak correlations of neural activity across viewings can occur in remarkable corre- spondence with arousing moments of the film. Moreover, a significant reduction in neural correlation occurs upon a second viewing of the film or when the narrative is disrupted by presenting its scenes scrambled in time. We also probe oscillatory brain activity during periods of heightened correlation, and observe during such times a significant increase in the theta band for a frontal component and reductions in the alpha and beta frequency bands for parietal and occipital components. Low-resolution EEG tomography of these components suggests that the correlated neural activity is consistent with sources in the cingulate and orbitofrontal cortices. Put together, these results suggest that the observed synchrony reflects attention- and emotion-modulated cortical processing which may be decoded with high temporal resolution by extracting maximally correlated components of neural activity.
Age-related macular degeneration (AMD) is the major cause of blindness in the developed world. Though substantial work has been done to characterize the disease, it is difficult to predict how the state of an individual’s retina will ultimately affect their high-level perceptual function. In this paper, we describe an approach that couples retinal imaging with computational neural modeling of early visual processing to generate quantitative predictions of an individual’s visual perception. Using a patient population with mild to moderate AMD, we show that we are able to accurately predict subject-specific psychometric performance by decoding simulated neurodynamics that are a function of scotomas derived from an individual’s fundus image. On the population level, we find that our approach maps the disease on the retina to a representation that is a substantially better predictor of high-level perceptual performance than traditional clinical metrics such as drusen density and coverage. In summary, our work identifies possible new metrics for evaluating the efficacy of treatments for AMD at the level of the expected changes in high-level visual perception and, in general, typifies how computational neural models can be used as a framework to characterize the perceptual consequences of early visual pathologies.
A common approach used to fuse simultaneously recorded EEG and fMRI is to correlate trial-by-trial variability in the EEG, or variability of components derived therefrom, with the blood oxygenation level dependent response. When this correlation is done using the conventional univariate approach, for example with the general linear model, there is the usual problem of correcting the statistics for multiple comparisons. Cluster thresholding is often used as the correction of choice, though in many cases it is utilized in an ad hoc way, for example by employing the same cluster thresholds for both traditional regressors (stimulus or behaviorally derived) and EEG-derived regressors. In this paper we describe a resampling procedure that takes into account the a priori statistics of the trial-to-trial variability of the EEG-derived regressors in a way that trades off cluster size and maximum voxel Z-score to properly correct for multiple comparisons. We show that this data adaptive procedure improves sensitivity for smaller clusters of activation, without sacrificing the specificity of the results. Our results suggest that extra care is needed in correcting statistics when the regressor model is derived from noisy and/or uncertain measurements, as is the case for regressors constructed from single-trial variations in the EEG.
We describe a closed-loop brain–computer interface that re-ranks an image database by iterating between user generated ‘interest’ scores and computer vision generated visual similarity measures. The interest scores are based on decoding the electroencephalographic (EEG) correlates of target detection, attentional shifts and self-monitoring processes, which result from the user paying attention to target images interspersed in rapid serial visual presentation (RSVP) sequences. The highest scored images are passed to a semi-supervised computer vision system that reorganizes the image database accordingly, using a graph-based representation that captures visual similarity between images. The system can either query the user for more information, by adaptively resampling the database to create additional RSVP sequences, or it can converge to a ‘done’ state. The done state includes a final ranking of the image database and also a ‘guess’ of the user’s chosen category of interest. We find that the closed-loop system’s re-rankings can substantially expedite database searches for target image categories chosen by the subjects. Furthermore, better reorganizations are achieved than by relying on EEG interest rankings alone, or if the system were simply run in an open loop format without adaptive resampling.
The presence of asymmetry in the misclassification costs or class prevalences is a common occurrence in the pattern classification domain. While much interest has been devoted to the study of cost-sensitive learning techniques, the relationship between cost-sensitive learning and the specification of the model set in a parametric estimation framework remains somewhat unclear. To that end, we differentiate between the case of the model including the true posterior, and that in which the model is misspecified. In the former case, it is shown that thresholding the maximum likelihood (ML) estimate is an asymptotically optimal solution to the risk minimization problem. On the other hand, under model misspecification, it is demonstrated that thresholded ML is suboptimal and that the risk-minimizing solution varies with the misclassification cost ratio. Moreover, we analytically show that the negative weighted log likelihood (Elkan, 2001) is a tight, convex upper bound of the empirical loss. Coupled with empirical results on several real-world data sets, we argue that weighted ML is the preferred cost-sensitive technique.
l1-regularized logistic regression, also known as sparse logistic regression, is widely used in machine learning, computer vision, data mining, bioinformatics and neural signal processing. The use of l1 regularization attributes attractive properties to the classifier, such as feature selection, robustness to noise, and as a result, classifier generality in the context of supervised learning. When a sparse logistic regression problem has large-scale data in high dimensions, it is computationally expensive to minimize the non-differentiable l1-norm in the objective function. Motivated by recent work (Koh et al., 2007; Hale et al., 2008), we propose a novel hybrid algorithm based on combining two types of optimization iterations: one being very fast and memory friendly while the other being slower but more accurate. Called hybrid iterative shrinkage (HIS), the resulting algorithm is comprised of a fixed point continuation phase and an interior point phase. The first phase is based completely on memory efficient operations such as matrix-vector multiplications, while the second phase is based on a truncated Newton’s method. Furthermore, we show that various optimization techniques, including line search and continuation, can significantly accelerate convergence. The algorithm has global convergence at a geometric rate (a Q-linear rate in optimization terminology). We present a numerical comparison with several existing algorithms, including an analysis using benchmark data from the UCI machine learning repository, and show our algorithm is the most computationally efficient without loss of accuracy.
Our society’s information technology advancements have resulted in the increasingly problematic issue of information overload-i.e., we have more access to information than we can possibly process. This is nowhere more apparent than in the volume of imagery and video that we can access on a daily basis-for the general public, availability of YouTube video and Google Images, or for the image analysis professional tasked with searching security video or satellite reconnaissance. Which images to look at and how to ensure we see the images that are of most interest to us, begs the question of whether there are smart ways to triage this volume of imagery. Over the past decade, computer vision research has focused on the issue of ranking and indexing imagery. However, computer vision is limited in its ability to identify interesting imagery, particularly as Â¿interestingÂ¿ might be defined by an individual. In this paper we describe our efforts in developing brain-computer interfaces (BCIs) which synergistically integrate computer vision and human vision so as to construct a system for image triage. Our approach exploits machine learning for real-time decoding of brain signals which are recorded noninvasively via electroencephalography (EEG). The signals we decode are specific for events related to imagery attracting a user’s attention. We describe two architectures we have developed for this type of cortically coupled computer vision and discuss potential applications and challenges for the future
Traditional analysis methods for single-trial classification of electro-encephalography (EEG) focus on two types of paradigms: phase-locked methods, in which the amplitude of the signal is used as the feature for classification, that is, event related potentials; and second-order methods, in which the feature of interest is the power of the signal, that is, event related (de)synchronization. The process of deciding which paradigm to use is ad hoc and is driven by assumptions regarding the underlying neural generators. Here we propose a method that provides an unified framework for the analysis of EEG, combining first and second-order spatial and temporal features based on a bilinear model. Evaluation of the proposed method on simulated data shows that the technique outperforms state-of-the art techniques for single-trial classification for a broad range of signal-to-noise ratios. Evaluations on human EEG−including one benchmark data set from the Brain Computer Interface (BCI) competition−show statistically significant gains in classification accuracy, with a reduction in overall classification error from 26%-28% to 19%.
Advances in neural signal and image acquisition as well as in multivariate signal processing and machine learning are enabling a richer and more rigorous understanding of the neural basis of human decision-making. Decision-making is essentially characterized behaviorally by the variability of the decision across individual trials—e.g., error and response time distributions. To infer the neural processes that govern decision-making requires identifying neural correlates of such trial-to-trial behavioral variability. In this paper, we review efforts that utilize signal processing and machine learning to enable single-trial analysis of neural signals acquired while subjects perform simple decision-making tasks. Our focus is on neuroimaging data collected noninvasively via electroencephalograpy (EEG) and functional magnetic resonance imaging (fMRI). We review the specific frame-work for extracting decision-relevant neural components from the neuroimaging data, the goal being to analyze the trial-to-trial variability of the neural signal along these component directions and to relate them to elements of the decision-making process. We review results for perceptual decision-making and discrimination tasks, including paradigms in which EEG variability is used to inform an fMRI analysis. We discuss how single-trial analysis reveals aspects of the underlying decision-making networks that are unobservable using traditional trial-averaging methods.
The auditory oddball task is a well-studied stimulus paradigm used to investigate the neural correlates of simple target detection. It elicits several classic event-related potentials (ERPs), the most prominent being the P300 which is seen as a neural correlate of subjects’ detection of rare (target) stimuli. Though trial-averaging is typically used to identify and characterize such ERPs, their latency and amplitude can vary on a trial-to-trial basis reflecting variability in the underlying neural information processing. Here we simultaneously recorded EEG and fMRI during an auditory oddball task and identified cortical areas correlated with the trial-to-trial variability of task-discriminating EEG components. Unique to our approach is a linear multivariate method for identifying task-discriminating components within specific stimulus- or response-locked time windows. We find fMRI activations indicative of distinct processes that contribute to the single-trial variability during target detection. These regions are different from those found using standard, including trial-averaged, regressors. Of particular note is the strong activation of the lateral occipital complex (LOC). The LOC was not seen when using traditional event-related regressors. Though LOC is typically associated with visual/spatial attention, its activation in an auditory oddball task, where attention can wax and wane from trial to trial, indicates that it may be part of a more general attention network involved in allocating resources for target detection and decision making. Our results show that trial-to-trial variability in EEG components, acquired simultaneously with fMRI, can yield task-relevant BOLD activations that are otherwise unobservable using traditional fMRI analysis.
A fundamental feature of how we make decisions is that our responses are variable in the choices we make and the time it takes to make them. This makes it impossible to determine, for a single trial of an experiment, the quality of the evidence on which a decision is based. Even for stimuli from a single experimental condition, it is likely that stimulus and encoding differences lead to differences in the quality of evidence. In the research reported here, with a simple “face”/”car” perceptual discrimination task, we obtained late (decision-related) and early (stimulus-related) single-trial EEG component amplitudes that discriminated between faces and cars within and across conditions. We used the values of these amplitudes to sort the response time and choice within each experimental condition into more-face-like and less-face-like groups and then fit the diffusion model for simple decision making (a well-established model in cognitive psychology) to the data in each group separately. The results show that dividing the data on a trial-by-trial basis by using the late-component amplitude produces differences in the estimates of evidence used in the decision process. However, dividing the data on the basis of the early EEG component amplitude or the times of the peak amplitudes of either component did not index the information used in the decision process. The results we present show that a single-trial EEG neurophysiological measure for nominally identical stimuli can be used to sort behavioral response times and choices into those that index the quality of decision-relevant evidence.
Most visual stimuli we experience on a day-to-day basis are continuous sequences, with spatial structure highly correlated in time. During rapid serial visual presentation (RSVP), this correlation is absent. Here we study how subjects’ target detection responses, both behavioral and electrophysiological, differ between continuous serial visual sequences (CSVP), flashed serial visual presentation (FSVP) and RSVP. Behavioral results show longer reaction times for CSVP compared to the FSVP and RSVP conditions, as well as a difference in miss rate between RSVP and the other two conditions. Using mutual information, we measure electrophysiological differences in the electroencephalography (EEG) for these three conditions. We find two peaks in the mutual information between EEG and stimulus class (target vs. distractor), with the second peak occurring 30–40 ms earlier for the FSVP and RSVP conditions. In addition, we find differences in the persistence of the peak mutual information between FSVP and RSVP conditions. We further investigate these differences using a mutual information based functional connectivity analysis and find significant fronto-parietal functional coupling for RSVP and FSVP but no significant coupling for the CSVP condition. We discuss these findings within the context of attentional engagement, evidence accumulation and short-term visual memory.
We present a nonlinear unmixing approach for extracting the ballistocardiogram (BCG) from EEG recorded in an MR scanner during simultaneous acquisition of functional MRI (fMRI). First, an overcomplete basis is identified in the EEG based on a custom multipath EEG electrode cap. Next, the overcomplete basis is used to infer non-Kirchhoffian latent variables that are not consistent with a conservative electric field. Neural activity is strictly Kirchhoffian while the BCG artifact is not, and the representation can hence be used to remove the artifacts from the data in a way that does not attenuate the neural signals needed for optimal single-trial classification performance. We compare our method to more standard methods for BCG removal, namely independent component analysis and optimal basis sets, by looking at single-trial classification performance for an auditory oddball experiment. We show that our overcomplete representation method for removing BCG artifacts results in better single-trial classification performance compared to the conventional approaches, indicating that the derived neural activity in this representation retains the complex information in the trial-to-trial variability.
Magnetic resonance spectroscopic imaging (MRSI) is currently used clinically in conjunction with anatomical MRI to assess the presence and extent of brain tumors and to evaluate treatment response. Unfortunately, the clinical utility of MRSI is limited by significant variability of in vivo spectra. Spectral profiles show increased variability because of partial coverage of large voxel volumes, infiltration of normal brain tissue by tumors, innate tumor heterogeneity, and measurement noise. We address these problems directly by quantifying the abundance (i.e. volume fraction) within a voxel for each tissue type instead of the conventional estimation of metabolite concentrations from spectral resonance peaks. This ‘spectrum separation’ method uses the non-negative matrix factorization algorithm, which simultaneously decomposes the observed spectra of multiple voxels into abundance distributions and constituent spectra. The accuracy of the estimated abundances is validated on phantom data. The presented results on 20 clinical cases of brain tumor show reduced cross-subject variability. This is reflected in improved discrimination between high-grade and low-grade gliomas, which demonstrates the physiological relevance of the extracted spectra. These results show that the proposed spectral analysis method can improve the effectiveness of MRSI as a diagnostic tool.
This review summarizes linear spatiotemporal signal analysis methods that derive their power from careful consideration of spatial and temporal features of skull surface potentials. BCIs offer tremendous potential for improving the quality of life for those with severe neurological disabilities. At the same time, it is now possible to use noninvasive systems to improve performance for time-demanding tasks. Signal processing and machine learning are playing a fundamental role in enabling applications of BCI and in many respects, advances in signal processing and computation have helped to lead the way to real utility of noninvasive BCI.
Single-unit and multiunit recordings in primates have already established that decision making involves at least two general stages of neural processing: representation of evidence from early sensory areas and accumulation of evidence to a decision threshold from decision-related regions. However, the relay of information from early sensory to decision areas, such that the accumulation process is instigated, is not well understood. Using a cued paradigm and single-trial analysis of electroencephalography (EEG), we previously reported on temporally specific components related to perceptual decision making. Here, we use information derived from our previous EEG recordings to inform the analysis of fMRI data collected for the same behavioral task to ascertain the cortical origins of each of these EEG components. We demonstrate that a cascade of events associated with perceptual decision making takes place in a highly distributed neural network. Of particular importance is an activation in the lateral occipital complex implicating perceptual persistence as a mechanism by which object decision making in the human brain is instigated.
We present a large-scale anatomically constrained spiking neuron model of the lateral geniculate nucleus (LGN), which operates solely with retinal input, relay cells, and interneurons. We show that interneuron inhibition and sparse connectivity between LGN cells could be key factors for explaining a number of observed classical and extraclassical response properties in LGN of monkey and cat. Among them are 1 ) weak orientation tuning, 2 ) contrast invariance of spatial frequency tuning in the absence of cortical feedback, 3 ) extraclassical surround suppression, and 4 ) orientation tuning of extraclassical surround suppression. The model also makes two surprising predictions: 1 ) a possible pinwheel-like spatial organization of orientation preference in the parvo layers of monkey LGN, much like what is seen in V1, and 2 ) a stimulus-induced trend (bias) in the orientation and phase preference of surround suppression, originating from the stimulus discontinuity between center and surround gratings rather than from specific circuitry.
In this letter, we considered the application of parametric spectral analysis, namely a short-window directed transfer function (DTF) approach, to multichannel electroencephalography (EEG) data during a face discrimination task. We identified causal influences between occipitoparietal and centrofrontal electrode sites, the timing of which corresponded to previously reported EEG face-selective components. More importantly we present evidence that there are both feedforward and feedback influences, a finding that is in direct contrast to current computational models of perceptual discrimination and decision making which tend to favor a purely feedforward processing scheme.
The increasing role of metabolomics in system biology is driving the development of tools for comprehensive analysis of high-resolution NMR spectral datasets. This task is quite challenging since unlike the datasets resulting from other ‘omics’, a substantial preprocessing of the data is needed to allow successful identification of spectral patterns associated with relevant biological variability. HiRes is a unique stand-alone software tool that combines standard NMR spectral processing functionalities with techniques for multi-spectral dataset analysis, such as principal component analysis and non-negative matrix factorization. In addition, HiRes contains extensive abilities for data cleansing, such as baseline correction, solvent peak suppression, removal of frequency shifts owing to experimental conditions as well as auxiliary information management. Integration of these components together with multivariate analytical procedures makes HiRes very capable of addressing the challenges for assessment and interpretation of large metabolomic datasets, greatly simplifying this otherwise lengthy and difficult process and assuring optimal information retrieval.
Extraclassical receptive field phenomena in V1 are commonly attributed to long-range lateral connections and/or extrastriate feedback. We address 2 such phenomena: surround suppression and receptive field expansion at low contrast. We present rigorous computational support for the hypothesis that the phenomena largely result from local short-range (
Based on a large-scale neural network model of striate cortex (V1), we present a simulation study of extra- and intracellular response modulations for drifting and contrast reversal grating stimuli. Specifically, we study the dependency of these modulations on the neural circuitry. We find that the frequently used ratio of the first harmonic to the mean response to classify simple and complex cells is highly insensitive to circuitry. Limited experimental sample size for the distribution of this measure makes it unsuitable for distinguishing whether the dichotomy of simple and complex cells originates from distinct LGN axon connectivity and/or local circuitry in V1. We show that a possible useful measure in this respect is the ratio of the intracellular second- to first-harmonic response for contrast reversal gratings. This measure is highly sensitive to neural circuitry and its distribution can be sampled with sufficient accuracy from a limited amount of experimental data. Further, the distribution of this measure is qualitatively similar to that of the subfield correlation coefficient, although it is more robust and easier to obtain experimentally.
We describe a real-time electroencephalography (EEG)-based brain-computer interface system for triaging imagery presented using rapid serial visual presentation. A target image in a sequence of nontarget distractor images elicits in the EEG a stereotypical spatiotemporal response, which can be detected. A pattern classifier uses this response to reprioritize the image sequence, placing detected targets in the front of an image stack. We use single-trial analysis based on linear discrimination to recover spatial components that reflect differences in EEG activity evoked by target versus nontarget images. We find an optimal set of spatial weights for 59 EEG sensors within a sliding 50-ms time window. Using this simple classifier allows us to process EEG in real time. The detection accuracy across five subjects is on average 92%, i.e., in a sequence of 2500 images, resorting images based on detector output results in 92% of target images being moved from a random position in the sequence to one of the first 250 images (first 10% of the sequence). The approach leverages the highly robust and invariant object recognition capabilities of the human visual system, using single-trial EEG analysis to efficiently detect neural signatures correlated with the recognition event.
Machine learning offers a principled approach for developing sophisticated, automatic, and objective algorithms for analysis of high-dimensional and multimodal biomedical data. This review focuses on several advances in the state of the art that have shown promise in improving detection, diagnosis, and therapeutic monitoring of disease. Key in the advancement has been the development of a more in-depth understanding and theoretical analysis of critical issues related to algorithmic construction and learning theory. These include trade-offs for maximizing generalization performance, use of physically realistic constraints, and incorporation of prior knowledge and uncertainty. The review describes recent developments in machine learning, focusing on supervised and unsupervised linear methods and Bayesian inference, which have made significant impacts in the detection and diagnosis of disease in biomedicine. We describe the different methodologies and, for each, provide examples of their application to specific domains in biomedical diagnostics.
When does the brain know that a decision is difficult to make? How does decision difficulty affect the allocation of neural resources and timing of constituent cortical processing? Here, we use single-trial analysis of electroencephalography (EEG) to identify neural correlates of decision difficulty and relate these to neural correlates of decision accuracy. Using a cued paradigm, we show that we can identify a component in the EEG that reflects the inherent task difficulty and not simply a correlation with the stimulus. We find that this decision difficulty component arises ≈220 ms after stimulus presentation, between two EEG components that are predictive of decision accuracy [an “early” (170 ms) and a “late” (≈300 ms) component]. We use these results to develop a timing diagram for perceptual decision making and relate the component activities to parameters of a diffusion model for decision making.
We describe a spatio-temporal linear discriminator for single-trial classification of multi-channel electroencephalography (EEG). No prior information about the characteristics of the neural activity is required, i.e., the algorithm requires no knowledge about the timing and spatial distribution of the evoked responses. The algorithm finds a temporal delay/window onset time for each EEG channel and then spatially integrates the channels for each channel-specific onset time. The algorithm can be seen as learning discrimination trajectories defined within the space of EEG channels. We demonstrate the method for detecting auditory-evoked neural activity and discrimination of task difficulty in a complex visual-auditory environment.
Probabilistic models of image statistics underlie many approaches in image analysis and processing. An important class of such models have variables whose dependency graph is a tree. If the hidden variables take values on a finite set, most computations with the model can be performed exactly, including the likelihood calculation, training with the EM algorithm, etc. Crouse et al. developed one such model, the hidden Markov tree ( HMT). They took particular care to limit the complexity of their model. We argue that it is beneficial to allow more complex tree-structured models, describe the use of information theoretic penalties to choose the model complexity, and present experimental results to support these proposals. For these experiments, we use what we call the hierarchical image probability (HIP) model. The differences between the HIP and the HMT models include the use of multivariate Gaussians to model the distributions of local vectors of wavelet coefficients and the use of different numbers of hidden states at each resolution. We demonstrate the broad utility of image distributions by applying the HIP model to classification, synthesis, and compression, across a variety of image types, namely, electrooptical, synthetic aperture radar, and mammograms (digitized X-rays). In all cases, we compare with the HMT.
Single and multi-unit recordings in primates have identified spatially localized neuronal activity correlating with an animal’s behavioral performance. Due to the invasive nature of these experiments, it has been difficult to identify such correlates in humans. We report the first non-invasive neural measurements of perceptual decision making, via single-trial EEG analysis, that lead to neurometric functions predictive of psychophysical performance for a face versus car categorization task. We identified two major discriminating components. The earliest correlating with psychophysical performance was consistent with the well-known face-selective N170. The second component, which was a better match to the psychometric function, did not occur until at least 130 ms later. As evidence for faces versus cars decreased, onset of the later, but not the earlier, component systematically shifted forward in time. In addition, a choice probability analysis indicated strong correlation between the neural responses of the later component and our subjects’ behavioral judgements. These findings demonstrate a temporal evolution of component activity indicative of an evidence accumulation process which begins after early visual perception and has a processing time that depends on the strength of the evidence.
In this paper, we describe a simple set of “recipes” for the analysis of high spatial density EEG. We focus on a linear integration of multiple channels for extracting individual components without making any spatial or anatomical modeling assumptions, instead requiring particular statistical properties such as maximum difference, maximum power, or statistical independence. We demonstrate how corresponding algorithms, for example, linear discriminant analysis, principal component analysis and independent component analysis, can be used to remove eye-motion artifacts, extract strong evoked responses, and decompose temporally overlapping components. The general approach is shown to be consistent with the underlying physics of EEG, which specifies a linear mixing model of the underlying neural and non-neural current sources.
Several theories of early visual perception hypothesize neural circuits that are responsible for assigning ownership of an object’s occluding contour to a region which represents the “figure.” Previously, we have presented a Bayesian network model which integrates multiple cues and uses belief propagation to infer local figure-ground relationships along an object’s occluding contour. In this paper, we use a linear integrate-and-fire model to demonstrate how such inference mechanisms could be carried out in a biologically realistic neural circuit. The circuit maps the membrane potentials of individual neurons to log probabilities and uses recurrent connections to represent transition probabilities. The network’s “perception” of figure-ground is demonstrated for several examples, including perceptually ambiguous figures, and compared qualitatively and quantitatively with human psychophysics.
We present an algorithm for blindly recovering constituent source spectra from magnetic resonance (MR) chemical shift imaging (CSI) of the human brain. The algorithm, which we call constrained nonnegative matrix factorization (cNMF), does not enforce independence or sparsity, instead only requiring the source and mixing matrices to be nonnegative. It is based on the nonnegative matrix factorization (NMF) algorithm, extending it to include a constraint on the positivity of the amplitudes of the recovered spectra. This constraint enables recovery of physically meaningful spectra even in the presence of noise that causes a significant number of the observation amplitudes to be negative. We demonstrate and characterize the algorithm’s performance using 31P volumetric brain data, comparing the results with two different blind source separation methods: Bayesian spectral decomposition (BSD) and nonnegative sparse coding (NNSC). We then incorporate the cNMF algorithm into a hierarchical decomposition framework, showing that it can be used to recover tissue-specific spectra given a processing hierarchy that proceeds coarse-to-fine. We demonstrate the hierarchical procedure on 1H brain data and conclude that the computational efficiency of the algorithm makes it well-suited for use in diagnostic work-up.
One of the challenges faced by the visual system is integrating cues within and across processing streams for inferring scene properties and structure. This is particularly apparent in the inference of object motion, where psychophysical experiments have shown that integration of motion signals, distributed across space, must also be integrated with form cues. This has led several to conclude that there exist mechanisms which enable form cues to ‘veto’ or completely suppress ambiguous motion signals. We describe a probabilistic approach which uses a generative network model for integrating form and motion cues using the machinery of belief propagation and Bayesian inference. We show, using computer simulations, that motion integration can be mediated via a local, probabilistic representation of contour ownership, which we have previously termed ‘direction of figure’. The uncertainty of this inferred form cue is used to modulate the covariance matrix of network nodes representing local motion estimates in the motion stream. We show with results for two sets of stimuli that the model does not completely suppress ambiguous cues, but instead integrates them in a way that is a function of their underlying uncertainty. The result is that the model can account for the continuum of bias seen for motion coherence and perceived object motion in psychophysical experiments.
We develop a probabilistic network model over image spaces and demonstrate its broad utility in mammographic image analysis, particularly with respect to computer-aided diagnosis. The model employs a multi-scale pyramid decomposition to factor images across scale and a network of tree-structured hidden variables to capture long-range spatial dependencies. This factoring makes the computation of the density functions local and tractable. The result is a hierarchical mixture of conditional probabilities, similar to a hidden Markov model on a tree. The model parameters are found with maximum likelihood estimation using the expectation-maximization algorithm. The utility of the model is demonstrated for three applications: (1) detection of mammographic masses for computer-aided diagnosis; (2) qualitative assessment of model structure through mammographic synthesis; and (3) compression of mammographic regions of interest.
Conventional electroencephalography (EEG) and magnetoencephalography (MEG) analysis often rely on averaging over multiple trials to extract statistically relevant differences between two or more experimental conditions. We demonstrate that by linearly integrating information over multiple spatially distributed sensors within a predefined time window, one can discriminate conditions on a trial-by-trial basis with high accuracy. We restrict ourselves to a linear integration as it allows the computation of a spatial distribution of the discriminating source activity. In the present set of experiments the resulting source activity distributions correspond to functional neuroanatomy consistent with the task (e.g. contralateral sensory-motor cortex and anterior cingulate).
We present three datasets that were used to conduct an open competition for evaluating the performance of various machine-learning algorithms used in brain-computer interfaces. The datasets were collected for tasks that included: 1) detecting explicit left/right (L/R) button press; 2) predicting imagined L/R button press; and 3) vertical cursor control. A total of ten entries were submitted to the competition, with winning results reported for two of the three datasets.
We describe a brain-computer interface (BCI) system, which uses a set of adaptive linear preprocessing and classification algorithms for single-trial detection of error related negativity (ERN). We use the detected ERN as an estimate of a subject’s perceived error during an alternative forced choice visual discrimination task. The detected ERN is used to correct subject errors. Our initial results show average improvement in subject performance of 21% when errors are automatically corrected via the BCI. We are currently investigating the generalization of the overall approach to other tasks and stimulus paradigms.
In this short note we highlight the fact that linear blind source separation can be formulated as a generalized eigenvalue decomposition under the assumptions of non-Gaussian, non-stationary, or non-white independent sources. The solution for the unmixing matrix is given by the generalized eigenvectors that simultaneously diagonalize the covariance matrix of the observations and an additional symmetric matrix whose form depends upon the particular assumptions. The method critically determines the mixture coefficients and is therefore not robust to estimation errors. However it provides a rather general and unified solution that summarizes the conditions for successful blind source separation. To demonstrate the method, which can be implemented in two lines of matlab code, we present results for artificial mixtures of speech and real mixtures of electroencephalography (EEG) data, showing that the same sources are recovered under the various assumptions.
Identifying physiological and anatomical signatures of disease in signals and images is one of the fundamental challenges in biomedical engineering. The challenge is most apparent given that such signatures must be identified in spite of tremendous inter and intra-subject variability and noise. Crucial for uncovering these signatures has been the development of methods that exploit general statistical properties of natural signals. The signal processing and applied mathematics communities have developed, in recent years, signal representations which take advantage of Gabor-type and wavelet-type functions that localize signal energy in a joint time-frequency and/or space-frequency domain. These techniques can be expressed as multi-resolution transformations, of which perhaps the best known is the wavelet transform. In this paper we review wavelets, and other related multi-resolution transforms, within the context of identifying signatures for disease. These transforms construct a general representation of signals which can be used in detection, diagnosis and treatment monitoring. We present several examples where these transforms are applied to biomedical signal and imaging processing. These include computer-aided diagnosis in mammography, real-time mosaicking of ophthalmic slit-lamp imagery, characterization of heart disease via ultrasound, predicting epileptic seizures and signature analysis of the electroencephalogram, and reconstruction of positron emission tomography data.
This paper describes a pattern recognition architecture, which we term hierarchical pyramid/neural network (HPNN), that learns to exploit image structure at multiple resolutions for detecting clinically significant features in digital/digitized mammograms. The HPNN architecture consists of a hierarchy of neural networks, each network receiving feature inputs at a given scale as well as features constructed by networks lower in the hierarchy. Networks are trained using a novel error function for the supervised learning of image search/detection tasks when the position of the objects to be found is uncertain or ill defined. We have evaluated the HPNN’s ability to eliminate false positive (FP) regions of interest generated by the University of Chicago’s (UofC) Computer-aided diagnosis (CAD) systems for microcalcification and mass detection. Results show that the HPNN architecture, trained using the uncertain object position (UOP) error function, reduces the FP rate of a mammographic CAD system by approximately 50% without significant loss in sensitivity. Investigation into the types of FPs that the HPNN eliminates suggests that the pattern recognizer is automatically learning and exploiting contextual information. Clinical utility is demonstrated through the evaluation of an integrated system in a clinical reader study. We conclude that the HPNN architecture learns contextual relationships between features at multiple scales and integrates these features for detecting microcalcifications and breast masses.
Conventional analysis of electroencephalography (EEG) and magnetoencephalography (MEG) often relies on averaging over multiple trials to extract statistically relevant differences between two or more experimental conditions. In this article we demonstrate single-trial detection by linearly integrating information over multiple spatially distributed sensors within a predefined time window. We report an average, single-trial discrimination performance of Az ≈ 0.80 and fraction correct between 0.70 and 0.80, across three distinct encephalographic data sets. We restrict our approach to linear integration, as it allows the computation of a spatial distribution of the discriminating component activity. In the present set of experiments the resulting component activity distributions are shown to correspond to the functional neuroanatomy consistent with the task (e.g., contralateral sensory–motor cortex and anterior cingulate). Our work demonstrates how a purely data-driven method for learning an optimal spatial weighting of encephalographic activity can be validated against the functional neuroanatomy.
AIMS—To process video slit lamp biomicroscopic fundus image sequences in order to generate wide field, high quality fundus image montages which might be suitable for photodocumentation. METHODS—Slit lamp biomicroscopic fundus examination was performed on human volunteers with a contact or non-contact lens. A stock, charge coupled device camera permitted image capture and storage of the image sequence at 30 frames per second. Acquisition time was approximately 30 seconds. Individual slit lamp biomicroscope fundus image frames were aligned and blended with custom developed software. RESULTS—The developed algorithms allowed for highly accurate alignment and blending of partially overlapping slit lamp biomicroscopic fundus images to generate a seamless, high quality, wide field montage. CONCLUSIONS—Video image acquisition and processing algorithms allow for mosaicking and enhancement of slit lamp biomicroscopic fundus images. The improved quality and wide field of view may confer suitability for inexpensive, real time photodocumentation of disc and macular abnormalities.
We describe a neural simulator designed for simulating very large scale models of cortical architectures. This simulator, NEXUS, uses coarse-grain parallel computing by distributing computation and data onto multiple conventional workstations connected via a local area network. Coarse-grain parallel computing offers natural advantages in simulating functionally segregated neural processes. We partition a complete model into modules with locally dense connections–a module may represent a cortical area, column, layer, or functional entity. Asynchronous data communications among workstations are established through the Network File System, which, together with the implicit modularity, decreases communications overhead, and increases overall performance. Coarse-grain parallelism also benefits from the standardization of conventional workstations and LAN, including portability between generations and vendors.
The utility of combining neural networks with pyramid representations for target detection in aerial imagery is explored. First, it is shown that a neural network constructed using relatively simple pyramid features is a more effective detector, in terms of its sensitivity, than a network which utilizes more complex object-tuned features. Next, an architecture that supports coarse-to-fine search, context learning and data fusion is tested. The accuracy of this architecture is comparable to a more computationally expensive non-hierarchical neural network architecture, and is more accurate than a comparable conventional approach using a Fisher discriminant. Contextual relationships derived both from low-resolution imagery and supplemental data can be learned and used to improve the accuracy of detection. Such neural network/pyramid target detectors should be useful components in both user assisted search and fully automatic target recognition and monitoring systems
Visual processing has often been divided into three stages—early, intermediate, and high level vision, which roughly correspond to the sensation, perception, and cognition of the visual world. In this paper, we present a network-based model of intermediate-level vision that focuses on how surfaces might be represented in visual cortex. We propose a mechanism for representing surfaces through the establishment of “ownership”—a selective binding of contours and regions. The representation of ownership provides a central locus for visual integration. Our simulations show the ability to segment real and illusory images in a manner consistent with human perception. In addition, through ownership, other processes such as depth, transparency, and surface completion can interact with one another to organize an image into a perceptual scene.
We present a model of how objects can be visually discriminated based on the extraction of depth-from-occlusion. Object discrimination requires consideration of both the binding problem and the problem of segmentation. We propose that the visual system binds contours and surfaces by identifying “proto-objects”-compact regions bounded by contours. Proto-objects can then be linked into larger structures. The model is simulated by a system of interconnected neural networks. The networks have biologically motivated architectures and utilize a distributed representation of depth. We present simulations that demonstrate three robust psychophysical properties of the system. The networks are able to stratify multiple occluding objects in a complex scene into separate depth planes. They bind the contours and surfaces of occluded objects (for example, if a tree branch partially occludes the moon, the two “half-moons” are bound into a single object). Finally, the model accounts for human perceptions of illusory contour stimuli.
Rapid perceptual decision-making is believed to depend upon efficient allocation of neural resources to the processing of transient stimuli within task-relevant contexts…
Single-unit animal studies have consistently reported decision-related activity mirroring a process of temporal accumulation of sensory evidence to a fixed internal decision boundary…
EEG alpha-band activity is generally thought to represent an inhibitory state related to decreased attention and play a role in suppression of task-irrelevant stimulus processing, but a competing hypothesis suggests an active role in processing task-relevant information…
For day-to-day decisions, multiple factors influence our choice between alternatives. Two dimensions of decision making that substantially affect choice are the objective perceptual properties of the stimulus (e.g., salience) and its subjective value….
Given a decision that requires less than half a second for evaluating the characteristics of the incoming pitch and generating a motor response,…
Constrained non-negative matrix factorization (cNMF) with iterative data selection is described and demonstrated as a data analysis method for fast and automatic recovery of biochemically meaningful and diagnostically specific spectral patterns of the human brain from 1H MRS imaging (1H MRSI) data. To achieve this goal, cNMF decomposes in vivo multidimensional 1H MRSI data into two non-negative matrices representing (a) the underlying tissue-specific spectral patterns and (b) the spatial distribution of the corresponding metabolite concentrations. Central to the proposed approach is automatic iterative data selection which uses prior knowledge about the spatial distribution of the spectra to remove voxels that are due to artifacts and undesired metabolites/tissues such as the strong lipid and water components. The automatic recovery of diagnostic spectral patterns is demonstrated for long-TE1H MRSI data on normal human brain, multiple sclerosis, and serial brain tumor. The results show the ability of cNMF with iterative data selection to automatically and simultaneously recover tissue-specific spectral patterns and achieve segmentation of normal and diseased human brain tissue, concomitant with simplification of information content. These features of cNMF, which permit rapid recovery, reduction and interpretation of the complex diagnostic information content of large multi-dimensional spectroscopic imaging data sets, have the potential to enhance the clinical utility of in vivo1H MRSI.
In this paper, we use single-trial analysis of electroencephalography (EEG) to ascertain the cortical origins of response time variability in a rapid serial visual presentation (RSVP) task. We extract spatial components that maximally discriminate between target and distractor stimulus conditions over specific time windows between stimulus onset and the time of a motor response. We then compute the peak latency of this differential activity on a trial-by-trial basis, and correlate this with response time. We find, for our nine participants, that the majority of the latency is introduced by component activity which begins far-frontally 200 ms prior to the response and proceeds to become parietally distributed near the time of response. This activity is consistent with the hypothesis that cortical networks involved in generating the late positive complexes may be the origins of the observed response time variability in rapid discrimination of visual objects.
NEXUS is a novel simulation environment designed for modeling large neural systems. The simulator allows the user to incorporate complex functional properties and symbolic processing at the level of the individual unit. In addition, NEXUS takes advantage of the principle of topographic map organization, found throughout the mammalian nervous system, to facilitate the modeling and design process. An easy-to-use graphical interface allows the user to interactively build and test models. This paper describes the principles underlying the NEXUS design and its advantages over other current simulation approaches.
We present an example of how vision systems can be modeled and designed by integrating a top-down computationally- based approach with a bottom-up biologi cally-motivated architecture. The specific visual processing task we address is occlusion-based object segmentation — the discrimination of objects using cues derived from object interposition. We construct a model of object segmentation using hybrid neural networks—distributed parallel systems consisting of neural units modeled at different levels af abstraction. We show that such networks are particularly useful for systems which can be modeled using the combined top-down/bottom-up approach. Our hybrid model is capable of discriminat ing objects and stratifying them in relative depth. In addition, our system can account for several classes of human perceptual phenomena, such as illusory contours. We conclude that hybrid systems serve as a powerful paradigm for understanding the information processing strategies of biological vision and for constructing artificial vision-based applications.