Biomedical subjects
Publications and source records attributed to D Saad.
Using the magnetization enumerator method, we evaluate the practical and theoretical limitations of symmetric channels with real outputs. Results are presented for several regular Gallager code constructions.
The problem of vertex coloring in random graphs is studied using methods of statistical physics and probability. Our analytical results are compared to those obtained by exact enumeration and Monte Carlo simulations. We critically discuss the merits and shortcomings of the various methods, and interpret the results obtained. We present an exact analytical expression for the two-coloring problem as well as general replica symmetric approximated solutions for the thermodynamics of the graph coloring problem with p colors and K-body edges.
We determine the critical noise level for decoding low-density parity check error-correcting codes based on the magnetization enumerator (M), rather than on the weight enumerator (W) employed in the information theory literature. The interpretation of our method is appealingly simple, and the relation between the different decoding schemes such as typical pairs decoding, MAP, and finite temperature decoding (MPM) becomes clear. In addition, our analysis provides an explanation for the difference in performance between MN and Gallager codes. Our results are more optimistic than those derived using the methods of information theory and are in excellent agreement with recent results from another statistical physics approach.
Statistical physics is employed to evaluate the performance of error-correcting codes in the case of finite message length for an ensemble of Gallager's error correcting codes. We follow Gallager's approach of upper bounding the average decoding error rate, but invoke the replica method to reproduce the tightest general bound to date, and to improve on the most accurate zero-error noise level threshold reported in the literature. The relation between the methods used and those presented in the information theory literature are explored.
We study the dynamics of online learning in multilayer neural networks where training examples are sampled with repetition and where the number of examples scales with the number of network weights. The analysis is carried out using the dynamical replica method aimed at obtaining a closed set of coupled equations for a set of macroscopic variables from which both training and generalization errors can be calculated. We focus on scenarios whereby training examples are corrupted by additive Gaussian output noise and regularizers are introduced to improve the network performance. The dependence of the dynamics on the noise level, with and without regularizers, is examined, as well as that of the asymptotic values obtained for both training and generalization errors. We also demonstrate the ability of the method to approximate the learning dynamics in structurally unrealizable scenarios. The theoretical results show good agreement with those obtained from computer simulations.
The relation between Ising spin systems and public-key cryptography is investigated using methods of statistical physics. The insight gained from the analysis is used for devising a matrix-based cryptosystem whereby the ciphertext comprises products of the original message bits; these are selected by employing two predetermined randomly constructed sparse matrices. The ciphertext is decrypted using methods of belief propagation. The analyzed properties of the suggested cryptosystem show robustness against various attacks and competitive performance to modern cryptographical methods.
The performance of Gallager's error-correcting code is investigated via methods of statistical physics. In this approach, the transmitted codeword comprises products of the original message bits selected by two randomly constructed sparse matrices; the number of nonzero row/column elements in these matrices constitutes a family of codes. We show that Shannon's channel capacity is saturated for many of the codes while slightly lower performance is obtained for others which may be of higher practical relevance. Decoding aspects are considered by employing the Thouless-Anderson-Palmer approach which is identical to the commonly used belief-propagation-based decoding.
A method for improving the performance of sparse-matrix based parity check codes is proposed, based on insight gained from methods of statistical physics. The advantages of this approach are demonstrated on an existing encoding/decoding paradigm suggested by Sourlas. We also discuss the application of the same method to more advanced codes of a similar type.
A variation of Gallager error-correcting codes is investigated using statistical mechanics. In codes of this type, a given message is encoded into a codeword that comprises Boolean sums of message bits selected by two randomly constructed sparse matrices. The similarity of these codes to Ising spin systems with random interaction makes it possible to assess their typical performance by analytical methods developed in the study of disordered systems. The typical case solutions obtained via the replica method are consistent with those obtained in simulations using belief propagation decoding. We discuss the practical implications of the results obtained and suggest a computationally efficient construction for one of the more practical configurations.
We study the dynamics of supervised learning in layered neural networks, in the regime where the size p of the training set is proportional to the number N of inputs. Here the local fields are no longer described by Gaussian probability distributions and the learning dynamics is of a spin-glass nature, with the composition of the training set playing the role of quenched disorder. We show how dynamical replica theory can be used to predict the evolution of macroscopic observables, including the two relevant performance measures (training error and generalization error), incorporating the old formalism developed for complete training sets in the limit alpha=p/N-->infinity as a special case. For simplicity, we restrict ourselves in this paper to single-layer networks and realizable tasks. In the case of (on-line and batch) Hebbian learning, where a direct exact solution is possible, we show that our theory provides exact results at any time in many different verifiable cases. For non-Hebbian learning rules, such as PERCEPTRON and ADATRON, we find very good agreement between the predictions of our theory and numerical simulations. Finally, we derive three approximation schemes aimed at eliminating the need to solve a functional saddle-point equation at each time step, and we assess their performance. The simplest of these schemes leads to a fully explicit and relatively simple nonlinear diffusion equation for the joint field distribution, which already describes the learning dynamics surprisingly well over a wide range of parameters.
We investigate the performance of parity check codes using the mapping onto Ising spin systems proposed by Sourlas [Nature (London) 339, 693 (1989); Europhys. Lett. 25, 159 (1994)]. We study codes where each parity check comprises products of K bits selected from the original digital message with exactly C checks per message bit. We show, using the replica method, that these codes saturate Shannon's coding bound for K-->infinity when the code rate K/C is finite. We then examine the finite temperature case to assess the use of simulated annealing methods for decoding, study the performance of the finite K case, and extend the analysis to accommodate different types of noisy channels. The connection between statistical physics and belief propagation decoders is discussed and the dynamics of the decoding itself is analyzed. Further insight into new approaches for improving the code performance is given.
The dynamics of on-line learning is investigated for structurally unrealizable tasks in the context of two-layer neural networks with an arbitrary number of hidden neurons. Within a statistical mechanics framework, a closed set of differential equations describing the learning dynamics can be derived, for the general case of unrealizable isotropic tasks. In the asymptotic regime one can solve the dynamics analytically in the limit of a large number of hidden neurons, providing an analytical expression for the residual generalization error, the optimal and critical asymptotic training parameters, and the corresponding prefactor of the generalization error decay.
OBJECTIVES: We sought to identify the effects of endothelin (ET) subtype-A (ET(A))) receptor blockade during the development of congestive heart failure (CHF) on left ventricle (LV) function and contractility. BACKGROUND: Congested heart failure causes increased plasma levels of ET and ET(A) receptor activation. METHODS: Yorkshire pigs were assigned to four groups: 1) CHF: 240 beats/min for 3 weeks; n=7; 2) CHF/ET(A)-High Dose: paced for 2 weeks then ET(A) receptor blockade (BMS 193884, 50 mg/kg, b.i.d.) for the last week of pacing; n=6; 3) CHF/ET(A)-Low Dose: pacing for 2 weeks then ET(A) receptor blockade (BMS 193884, 12.5 mg/kg, b.i.d.) for the last week, n=6; and 4) CONTROL: n=8. RESULTS: Left ventricle fractional shortening decreased with CHF compared with control (12+/-1 vs. 39+/-1%, p < 0.05) and increased in the CHF/ET(A) High and Low Dose groups (23+/-3 and 25+/-1%, p < 0.05). The LV peak wall stress and wall force increased approximately twofold with CHF and remained increased with ET(A) receptor blockade. With CHF, systemic vascular resistance increased by 120%, was normalized in the CHF/ET(A) High Dose group, and fell by 43% from CHF values in the Low Dose group (p < 0.05). Plasma catecholamines increased fourfold in the CHF group and were reduced by 48% in both CHF/ET(A) blockade groups. The LV myocyte velocity of shortening was reduced with CHF (32+/-3 vs. 54+/-3 microm/s, p < 0.05), was higher in the CHF/ET(A) High Dose group (39+/-1 microm/s, p < 0.05), and was similar to CHF values in the Low Dose group. CONCLUSIONS: ET(A) receptor activation may contribute to the progression of LV dysfunction with CHF.
We explore the dependence of performance measures, such as the generalization error and generalization consistency, on the structure and the parametrization of the prior on "rules", instanced here by the noisy linear perceptron. Using a statistical mechanics framework, we show how one may assign values to the parameters of a model for a "rule" on the basis of data instancing the rule. Information about the data, such as input distribution, noise distribution and other "rule" characteristics may be embedded in the form of general Gaussian priors for improving net performance. We examine explicitly two types of general Gaussian priors which are useful in some simple cases. We calculate the optimal values for the parameters of these priors and show their effect in modifying the most probable, MAP, values for the rules. Copyright 1996 Elsevier Science Ltd.
Explore the source record for details and available documents.
Explore the source record for details and available documents.
The two-layer radial basis function network, with fixed centers of the basis functions, is analyzed within a stochastic training paradigm. Various definitions of generalization error are considered, and two such definitions are employed in deriving generic learning curves and generalization properties, both with and without a weight decay term. The generalization error is shown analytically to be related to the evidence and, via the evidence, to the prediction error and free energy. The generalization behavior is explored; the generic learning curve is found to be inversely proportional to the number of training pairs presented. Optimization of training is considered by minimizing the generalization error with respect to the free parameters of the training algorithms. Finally, the effect of the joint activations between hidden-layer units is examined and shown to speed training.
This paper proposes a new pruning method based on merging neurons with similar functional behavior which is defined by the internal representations of each neuron for the entire training set. Classification of neurons by their functional behavior with respect to the input vectors provides a powerful tool for pruning neurons and connections, thus reducing the network complexity and increasing its generalization capability. The most remarkable property of this pruning scheme is its ability to preserve net functionality by transferring the role of every removed neuron to the most fitted neuron of the surviving ones, using a unique merging and compensation procedure. The implementation of the proposed method is demonstrated using a detailed numerical example and its performance is examined by a statistical measure calculated by repeating the training procedure several times. The influence of parameter selection on pruning performance and generalization ability is discussed and demonstrated by examining statistical results.