Catalog Description

Modeling, simulation, and analysis of artificial neural networks. Relationship to biological neural networks. Design and optimization of discrete and continuous neural networks. Backpropagation, and other gradient descent methods. Hopfield and Boltzmann networks. Unsupervised learning. Self-organizing feature maps. Applications chosen from function approximation, signal processing, control, computer graphics, pattern recognition, time-series analysis. Relationship to fuzzy logic, genetic algorithms, and artificial life.

Prerequisites: CS 60 (Principles of Computer Science) and Mathematics 12 (Calculus and Linear Algebra), or permission of the instructor. 3 credit hours.

Instructor

• Bob Keller 1242 Olin (MTuW; 4-5 p.m. or by drop-in or appt.), keller@cs.hmc.edu, x 18483

Texts

• Main Textbook (abbreviated NND):

  Neural Network Design by Martin T. Hagan, Howard B. Demuth, and Mark Beale, reprint available from Huntley Bookstore, or University of Colorado Bookstore at 303-492-3648. ISBN 0-9717321-0-8.

  

• Software, which is installed on odin.ac.hmc.edu and various other machines (unfortunately, not turing):
  • Matlab (help)

Course Requirements

There will be some homework and programming assignments, but probably no exams. The assignments will constitute about 40% of your grade. 40% of your grade is from a substantial final project involving either creation of a working neural network application or a research paper. The grade on the project will be determined by the comprehensiveness and degree to which you explored competing approaches. The projects will be presented orally. 10% of your grade will be based on a preliminary presentation you make, ideally on material related to the your project. Finally, 10% will be based on general participation, which includes attendance.

CS 152 Topic Outline

• Week 1 (read NND chapters 1 to 4)

  Contexts for Neural Networks

  • Artificial Intelligence
  • Biological
  • Physics

  Artificial Neural Network overview

  • Perceptrons
  • Perceptron learning rule
  • Perceptron convergence theorem

• Week 2 (read NND chapter 5 to 7)
  • Linear transformations for neural networks
  • Supervised Hebbian learning
  • Pseudoinverse rule
  • Filtered learning rule
  • Delta rule
  • Unsupervised Hebbian learning

• Week 3 (read NND chapters 8 and 9)
  • Performance surfaces
  • Performance optimization
    • Steepest descent algorithm
    • Newton's method
    • Conjugate gradient

• Week 4 (read NND chapter 10)
  Widrow-Hoff Learning
  • Adaline
  • LMS rule
  • Adaptive filtering

• Week 5 (read NND chapters 11 and 12)
  • Backpropagation in Multi-Level Perceptrons (MLP)
  • Variations on backpropagation
    • Batching
    • Momentum
    • Variable learning rate
    • Levenberg-Marquardt (LMBP)
    • Quickprop

• Week 6 (supplementary material)
  • Radial basis function networks (RBF)

• Week 7 (read NND chapter 13)
  • Associative learning
    • Unsupervised Hebb rule
    • Hebb rule with decay
    • Instar rule
    • Kohonen rule
    • Outstar rule

• Week 8 (read NND chapter 14)
  • Competitive networks
    • Hamming network
    • Self-Organizing feature maps (SOM)
    • Counterpropagation networks (CPN)
    • Learning vector quanitization (LVQ)

• Week 9 (read NND chapters 15 and 16)
  • Grossberg networks
  • Adaptive resonance theory
    • ART1 networks

• Week 10 (read NND chapters 17 and 18)
  • Hopfield networks
  • Spin-glass model and simulated annealing
  • Boltzman networks
  • Cascade correlation learning
  • Bi-directional associative memory (BAM)

• Week 11 (supplementary material)
  • Sequential networks
    • Time series
    • Backpropagation through time
    • Finite Impulse Response (FIR) MLP
    • Method of temporal differences

First Assignment, Due Tues. 9/9/03

1. NND: E4.1
   
   
2. NND: E4.5
   
   
3. Create a running prototype program demonstrating the perceptron learning algorithm, using any language you choose. The idea is to learn from using, as well as constructing, the program.

   Your program should:

   • Accept a set of data points, each point consisting of
     • an input vector
     • an associated scalar desired response value.
     
     
   • Accept a learning rate LR.
     
     
   • Accept a limit on the number of iterations, in case the data points happen not to be linearly separable, or that convergence happens to take a long time.
     
     
   • Output a set of weights for the perceptron, if such a set can be be found within the limit. Otherwise output that the limit has been exceeded.
     
     
   • For each data point, show the input vector, the desired response, the perceptron's actual response, and the error value (desired-actual).
     
     
   • Optionally output the items in the previous two bullets before each iteration of the algorithm. (This can also be helpful for debugging as well as understanding the algorithm.)
     
     

   To make your life simple, by "accept" above, you don't need to provide for external file input for the data. Instead you can code each example case separately directly in code. Also, start with all weights initialized at 0.

   Show your program running on each of the following test cases.

   1. limit = 10, LR = 1

      Input Vector	Output Value
      (0 0)	0
      (0 1)	1
      (1 0)	1
      (1 1)	1

   2. limit = 50, LR = 0.5

      Input Vector	Output Value
      (-1 1)	1
      (0 0)	1
      (1 -1)	1
      (1 0)	0
      (0 1)	0

   3. limit = 50, LR = 0.5

      Input Vector	Output Value
      (0 0)	0
      (0 1)	1
      (1 0)	1
      (1 1)	0

   4. limit = 200, LR = 0.1

      Input Vector	Output Value
      (0.2    0.1    0.1    0.1    0.2    0.1    0.2    0.1    0.1)	1
      (0.2    0.1    0.1    0.1    0.2    0.1    0.3    0.1    0.1)	1
      (0.5    0.1    0.1    0.1    0.2    0.1    0.2    0.1    0.1)	1
      (0.5    0.4    0.6    0.8    0.4    0.1    0.8    1.0    0.1)	0
      (0.5    0.3    0.3    0.1    0.2    0.1    0.2    0.1    0.1)	1
      (0.2    0.3    0.1    0.1    0.3    0.1    0.1    0.1    0.1)	1
      (0.3    0.5    0.7    0.8    0.8    0.9    0.7    1.0    0.7)	0
      (1.0    0.5    0.6    1.0    0.6    1.0    0.7    0.7    1.0)	0