Search PubMed⌕ Search

PubMed · 11674845

Predictability, complexity, and learning.

Abstract

We define predictive information I(pred)(T) as the mutual information between the past and the future of a time series. Three qualitatively different behaviors are found in the limit of large observation times T:I(pred)(T) can remain finite, grow logarithmically, or grow as a fractional power law. If the time series allows us to learn a model with a finite number of parameters, then I(pred)(T) grows logarithmically with a coefficient that counts the dimensionality of the model space. In contrast, power-law growth is associated, for example, with the learning of infinite parameter (or nonparametric) models such as continuous functions with smoothness constraints. There are connections between the predictive information and measures of complexity that have been defined both in learning theory and the analysis of physical systems through statistical mechanics and dynamical systems theory. Furthermore, in the same way that entropy provides the unique measure of available information consistent with some simple and plausible conditions, we argue that the divergent part of I(pred)(T) provides the unique measure for the complexity of dynamics underlying a time series. Finally, we discuss how these ideas may be useful in problems in physics, statistics, and biology.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

W Bialek, I Nemenman, N Tishby. 2001. Predictability, complexity, and learning.. https://doi.org/10.1162/089976601753195969

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related citations

[Whole body MRI--diagnostic strategy of the future?].

Whole body magnetic resonance imaging (MRI) opens new opportunities in diagnostic radiology as systemic disease entities can be examined with high sensitivity. This can lead to a change of paradigm, so that not only organ-related but rather disease-specific MRI examination protocols can be applied which focus on the underlying pathophysiology of the disease. Whole body MRI has already been successfully used for several oncological and non-oncological indications. In addition, whole body MRI has broadened the discussion regarding its use for secondary prevention. Compared to computed tomography, MRI does not use radiation. Although whole body MRI is still in an early stage, the enormous medical and economical potential can be envisioned.

Forecasting↗