

Appendix 1. Terminology


activation function - a usually nonlinear function that processes 
  a unit's net function into the unit's output. (See "multilayer 
  perceptron" and "net function") The sigmoid activation is

           1
  O = ------------ 
      1 + exp(net)


classification - the process of making one of a fixed number of
  possible decisions, given a fixed number of numerical inputs. 
  The output of classification is an integer which indicates 
  the class decision. A network for classifying images of 
  handprinted numerals (0 through 9) would have 10 outputs 
  (in uncoded format). A classifier for processing stock market 
  data could make buy/sell decisions but would not predict
  future prices.

clustering - see unsupervised learning

coded format outputs - in a classification network, coded output 
  format means that the number of outputs is Nout = Log2 (Nc) where 
  Nc is the number of classes. The Nc desired output vectors are 
  then just Nout-bit binary numbers between 0 and Nc-1. 

error function - the function which is minimized during neural net
  training or unsupervised learning. The specific error functions
  minimized in this software package are given in the help files
  for the algorithm in question.

Euclidean distance - given two vectors x and y with N elements each,
  the Euclidean distance is the square root of 

        N               2
       Sum (y(i) - x(i))
       i=1

functional link net - A functional link net is a network in which
  (1) nonlinear functions of the inputs are formed to augment or
  add to the input vector, and (2) the outputs are linear functions
  of the augmented input vector. In the most common form of the
  functional link net, the augmented inputs are multinomials 
  formed from the original inputs. Since linear equations can 
  be solved for the output weights, functional link net training 
  is multidimensional polynomial regression. One problem with 
  this type of network is that it suffers from combinatorial 
  explosion. In other words, the number of possible multinomials 
  grows explosively with the network degree.

KLT-Karhunen Loeve transform. A linear, orthogonal transform in which
  the rows of the N by N (for compression, M by N where M is less than
  N) transformation matrix are eigenvectors of the N by N autocovariance
  matrix of the N-dimensional input vectors to be transformed. The KLT
  transformation matrix is the transpose of the U matrix from the singular
  value decomposition (SVD) of the autocovariance matrix. The KLT is 
  the optimal transform for compressing data, if your goal is to later
  reconstruct the data with the least mean-square error, using a 
  given number of KLT coefficients. The KLT does not, in general, 
  optimally compress neural net inputs, since the KLT transformation
  matrix is not designed using information from the desired output
  vectors. However, it can optimally compress the desired output
  vectors if it was designed using them instead of the input vectors.
  Another name for the KLT is principal components.
    
k-means clustering - given Nc initial clusters, which could come from
  sequential leader clustering, k-means iteratively (1) calculates a new
  mean vector for each cluster (necessary if any input vectors have changed
  clusters) and (2) reclassifies the input vectors to their nearest
  cluster. The sum of the distances between the input vectors and the 
  closest mean vectors is reduced. A distance measure, usually the 
  Euclidean distance, is used. In adaptive k-means, the reclassification 
  and mean calculation steps are performed during one pass through the 
  data. 

mapping - In mapping, you process numerical inputs into a real-valued 
  (floating point) outputs. A mapper for processing stock market data 
  could predict future prices, but would not make a buy/sell decision.

modular network - a neural network which consists of several networks
  connected together. The modules may all work in parallel with their
  answers then combined, in series, or in parallel with only one at a
  time being switched on. We use the latter scheme in this software 
  package. 

multilayer perceptron (MLP) - An MLP, sometimes called a backpropagation
  neural network, is a feedforward (usually) network in which outputs
  are algebraic, nonlinear functions of inputs. The MLP has at least
  two layers of units or artificial neurons, the input and output layers. 
  Additional layers, which make the network nonlinear, are called 
  hidden layers. In each hidden layer or output layer unit, an inner 
  product of weights and signals from previous layers, called a net 
  function, is formed. The unit's output is formed by putting the net 
  function through the activation function, which is usually nonlinear. 
  (See "activation function" and "net function" )

multinomial - a one-term polynomial in two or more variables which 
  has no coefficient, such as 
         2    3
     (x1) (x2)

net function - In the multilayer perceptron, a unit's first operation 
  is to form a number, called the net, using an inner product of 
  weights or coefficients multiplied by signals or unit outputs 
  from previous layers. The net function is then fed into the activation 
  function, yielding the unit's output. (See "activation function" and 
  "multilayer perceptron")
 
network degree - the degree of a polynomial that approximates a 
  given MLP network with a given amount of approximation error. As
  the network degree increases, this approximation error decreases.
  Network degree estimation is performed by program Tmapc2.

number of iterations - one of two stopping parameters used in 
  functional link nets and MLPs. This is the maximum number of 
  iterations that can be performed, and is user-chosen.

pruning - finding and eliminating the less useful hidden units in a
  trained MLP.

self-organizing map (SOM) - given Nc initial random clusters, the SOM
  performs an adaptive k-means clustering, except that when a cluster mean
  is updated, its nearest neighbors are also updated. There is a learning 
  factor and a distance threshold which decrease as clustering progresses.

sizing - determining the required size of a MLP from a training data
  file.

standard form - All training data files are in standard form, which 
  means that the file is formatted, and that each pattern or vector 
  has inputs on the left and desired outputs on the right. You can type 
  out the files to examine them, and you can use these files with 
  other neural net software. If a testing data file includes desired 
  outputs, then it too will be in standard form.

testing data file - the same as a training data file except that 
  (1) it is used to test the performance of a trained network and 
  (2) it may or may not have desired outputs. 

training data file - a formatted file with Nv vectors or patterns.
  Each vector includes N inputs and Nout desired outputs. In 
  classification training data files, the correct class id, which
  is an integer, is stored rather than the Nout desired outputs. 
  See standard form.

training parameters - the learning factor (Z in this software 
  package), and the momentum factor alpha.

uncoded format outputs - in a classification network, uncoded output 
  format means that the number of outputs is Nout = Nc where 
  Nc is the number of classes. The desired output can then be 1 for 
  the correct class and 0 for the others, or 0 for the correct class 
  and 1 for the others (inverted uncoded format). The Nc desired 
  output vectors are then just Nc-bit binary numbers. In the 
  classification network package Neucls.zip, inverted uncoded format 
  and coded format are available.

units - artificial neurons used in the MLP network.

unsupervised learning - Unsupervised learning or clustering is the 
  process of organizing a set of vectors into groups of similar 
  vectors. In many clustering algorithms, each cluster is 
  characterized using a mean or center vector. Unsupervised learning 
  algorithms usually use a distance measure, such as the Euclidean
  distance, to measure the closeness of a data vector to a cluster
  or mean or center vector.

weight file - an unformatted file which gives the gains or coefficients
  along paths connecting the various units.




