Catalog Description

Modeling, simulation, and analysis of artificial neural networks. Relationship to biological neural networks. Design and optimization of discrete and continuous neural networks. Backpropagation, and other gradient descent methods. Hopfield and Boltzmann networks. Unsupervised learning. Self-organizing feature maps. Applications chosen from function approximation, signal processing, control, computer graphics, pattern recognition, time-series analysis. Relationship to fuzzy logic, genetic algorithms, and artificial life.

Prerequisites: CS 60 and Mathematics 73 and 82, or permission of the instructor. 3 credit hours.

Instructor

• Robert Keller 242 Olin (4-5 p.m. MTuW or by appt.), keller@turing, x 18483
  
  

Texts

• Main Textbook:

  MMR: K. Mehrotra, C.K. Mohan, and S. Ranka, Elements of Artificial Neural Networks, M.I.T. Press, 1997, ISBN 0-262-13328-8.

  

• Related WWW links (for self-study and research) appear below.

• Software, which is installed on turing.cs.hmc.edu:
  • MATLAB Neural Network Toolbox, The MathWorks, Inc.
  • Matlab cross referenced
  • SNNS: Stuttgart Neural Network Simulator.
  • Genocop III

Course Requirements

There will be some homework and programming assignments, but no exams. These assignments will constitute about 50% of the grade. The other 50% of the grade is from a substantial final project involving either a working neural network application or a research paper. The grade on the project will be determined by the comprehensiveness and degree to which you explored competing approaches. The projects will be presented orally.

Optional voluntary oral presentations on textbook material can also be made during the term. These can act to cushion your grade. They are very much encouraged, as it they really help you learn the material at a higher level than you would otherwise. Please see me if you are interested in making a presentation.

CS 152 Topic Outline

• Week 1 (read MMR chapter 1)

  Introduction

  • Contexts for and Motivation Neural Networks: Artificial Intelligence | Biological | Physics
  • Artificial Neural Network overview

• Week 2 (read MMR chapter 2)

  Supervised Learning: Single-Layer Networks

  • Perceptrons
  • Adalines

• Weeks 3-6 (read MMR chapters 3 and 4)

  Supervised Learning: Multi-Layer Networks

  • Multi-Layer Perceptrons (MLPs)
  • Backpropagation
  • Conjugate Gradient method
  • Levenberg-Marquardt (LM) method
  • Madalines
  • Radial-Basis Networks
  • Cascade-Correlation Networks
  • Polynomial Networks
  • Recurrent Networks
    • Time series
    • Backpropagation through time
    • Finite Impulse Response (FIR) MLP
    • Temporal Differences method (TD)

• Weeks 7-8 (read MMR chapter 5)

  Unsupervised Learning

  • Simple Competitive Networks: Winner-take-all | Hamming network
  • Learning Vector Quantization (LVQ)
  • Counterpropagation Networks (CPN)
  • Adaptive Resonance Theory (ART)
  • Kohonen Self-Organizing Maps (SOMs)
  • Principal Component Analysis networks (PCA)

• Weeks 9-10 (read MMR chapter 6)

  Associative Models

  • Linear Associative Memory (LAM)
  • Hopfield Networks
  • Brain-State-in-a-Box (BSB)
  • Boltzmann Machines and Simulated Annealing
  • Bi-Directional Associative Memory (BAM)

• Week 11 (read MMR chapter 7)

  Optimization Problems

  • Neural Network Approaches
  • Evolutionary Programming

• Week 12

  Fuzzy logic and its connection to NNs
  

Auxiliary References (not required):

• Martin T. Hagan, Howard B. Demuth, and Mark Beale, Neural Network Design, PWS Publishing Company, Boston, 1996, ISBN 0-534-94332-2. This book was used in the 1996 offering of the course. It has excellent tutorial content and is strongly keyed to the use of MATLAB. Matrix formulations are used throughout.

• Simon Haykin, Neural networks - A comprehensive foundation, Macmillan, 1994. This book was used in the 1995 offering of the course. It has a strong mathematical and signal-processing orientation.

• Mohamad H. Hassoun, Fundamentals of artificial neural networks, MIT Press, 1995. This is another fairly thorough introduction.

• James A. Anderson, An introduction to neural networks, MIT Press, 1995. This is a more gentle introduction to the topic, by one of the pioneers in the field.

• Irwin B. Levitan and Leonard K. Kaczmarek, The Neuron, Oxford University Press, 1991. This book focuses on the biology and physics of neurons, if you wish to know more about this aspect; it will not be emphasized in the course.

• Marvin L. Minsky and Seymour Papert, Perceptrons (expanded addition), MIT Press, 1988. The historical importance of this book will be discussed in the course.

• Duda and Hart, Pattern classification and scene analysis, Wiley, 1972. This book gives a broad look at pattern classification problems, but is not on neural nets as such.

• Teuvo Kohonen Self-organizing maps, Springer-Verlag, 1995. This is a comprehensive reference by the originator of this concept.

• Bart Kosko Neural networks and fuzzy systems : a dynamical systems approach to machine intelligence, Prentice Hall, 1992. This book compares fuzzy and neural approaches to control problems.

• Zbigniew Michalewicz, Genetic Algorithms + Data Structures = Evolution Programs, Third Edition, Springer Verlag, 1996. This book describes the evolutionary approach, which in some cases can achieve results similar to neural approaches.

• John R. Koza, Genetic Programming, MIT Press, 1994. This book focuses on the evolutionary approach to producing programs.

• Christopher G. Langton (ed.), Artificial Life, an Overview, Addison-Wesley, 1989. This is an early colllection of articles on the topic.
• Neuro-Fuzzy and Soft Computing

Applications

• Glove Talk
• Pattern analysis of uncertain changes in the fetal ECG features during labour (fuzzy systems)

Neurocomputing People

• Charles W. Anderson
• Andrew G. Barto
• Stephen Grossberg
• Geoffrey Hinton
• John Hopfield
• Michael I. Jordan
• Stuart Kauffman
• Teuvo Kohonen
• Bart Kosko
• Todd K. Leen
• Lori Pratt
• Terrence J. Sejnowski
• Richard S. Sutton
• Gerald Tesauro
• Eric A. Wan
• Andreas Weigend
• Bernard Widrow

Software

• RBF networks in MatLab
• jaNet

Data

• NetTalk
• Protein Structure
• Santa Fe Time Series Data
• Chaotic Time Series Data

Demos

• Fuzzy Java Toolkit
• Biomorphs
• Self-Reproducing Cellular Automata
• L system
• L Systems
• Boids
• Fish Schooling
• Flies | Floys
• Evolving Robots
• aworld
• L System Modeller
• Tutorial Applets
• Manna Mouse (GA demo)
• Jafa: Neural Net for Financial Applications
• Evolutionary Computing Demonstration System
• 3D Biomorphs (not applet)
• Hopfield Net demo
• Demos by Mohammed H. Hassoun
• 1D Kohonen map
• 2D Kohonen map
• Another 2D Kohonen map
• 3D Kohonen map
• Travelling Salesman Problem using Kohonen map
• Competitive Learning Models
• Neural Character Recognition
• VQ
• Robot Arm (GCMAC)
• Elastic Net TSP
• N Queens
• DISCERN Neural Natural Language Demo
• GP Truck Backer
• Fuzzy Truck Backer
• Fuzzy Logic Controlled Car
• Cooperative Simulated Annealing
• Genetic Algorithm
• List of Links
• Other NN Demos
• OGI Links
• Temporal Difference Simulator
• Dynamic Channel Assignment (Reinforcement Learning)

Expositions on the web

• L Systems
• Introduction to The Mystery of the Human Brain
• Glossary
• Radial Basis Function Networks
• Time Series Prediction with Neural Nets
• Whole Brain Atlas
• Reinforcement Learning
• Reinforcement Learning Tutorial
• Control Theory

Worldwide Web Indices

• Files corresponding to the textbook
• Neural Network Links
• Introductory Links
• Journal Links
• More Journal Links
• Neuroscience Archives
• WEBSOM
• Digital Physics
• Self Replication (Moshe Sipper)
• Complexity Papers
• Encore
• Zooland
• Artificial Life Games Homepage
• Cybertations
• Neural Network books
• Books
• Neural Networks at your Fingertips
• Hassoun's Neural Network Sites
• Electronic Sources of Information
• NN FAQ
• Bibliographies
• Index to NN Bibliographies
• Top 500 Neural Network Sites
• Yahoo Search
• Alta Vista Search
• Lycos Search
• Magellan Search
• Machine Learning Course
• Neural Networks Archive
• Backpropagation
• SNN Neural Network Pointers
• Noise Biblio
• Financial Markets Biblio
• Neuro-Fuzzy Systems
• Neural Networks in Diagnosis and Forecasting Applications
• Autocorrelation function examples
• Neural Networks Survey
• Using NN's for web browsing
• Temporal Difference Learning and TD-Gammon
• Neuroscience index
• Sutton's Temporal Difference paper, and others
• Neural Net Transfer and Learning to Learn
• Notes on Neural Nets
• Dave Touretzky's Notes
• Slides re. Hertz, Krogh, and Palmer
• CMU
• Speech Translation Research
• University of Sheffield
• University of University of Skovde
• University of Nijmegen
• Cortex Project
• Aston University
• Caltech CNS 185: Collective Computation
• Neutrality links
• Machine Learning in Games
• Lip Reading
• Strategic Game-Playing Bibliograhy
• Notes on Control
• PDP++
• WEBSOM
• Boston University Center for Adaptive Systems
• ART (Adaptive Resonance Theory) FAQ
• Neural Net Resources
• Neural Net Links
• CMU AI Repository
• Fuzzy Logic Resources
• Optimization links