Methodology

In order to examine the greatest possible range of inference techniques, I am using Matlab's neural networks toolbox in conjunction with the Bayes Net Toolbox, which supports the use of many difference inference techniques.

In order to ensure that my tests reflect useful real-world applications and are sufficiently general that no inference scheme is put at an unfair disadvantage, I am using data sets from the DELVE Project. DELVE stands for Data for Evaluating Learning in Valid Experiments, and the project includes a battery of test data to use as a gauntlet when evaluating learning techniques. As such, this data should be appropriate to my purposes.

Unfortunately, in practice using DELVE's data is problematic. For many Bayesian inference techniques, a reasonable prior probability distribution is required for convergence. Since DELVE's data sets were not assembled with Bayesian techniques in mind, no information about these probabilities is available, and reasonable guesses are difficult without some background knowledge of the data. Luckily, there is a project with goals similar to DELVE's which accomodates a Bayesian approach. The Bayesian Network Repository contains data with full prior probability distribution information. I used data from this repository to conduct my trials.

These trials use a subset of the Bayesian network tests available at the Bayes Network Repository. Some tests were not included due to an incompatibility with the file format converter that I used to prepare the networks for analysis using the Bayesian Network Toolbox, available here. The incompatibility is due to an apparent bug in the variable name translator. Since source code was not available, I was unable to verify that this is the problem, but I plan to contact the author of the tool to clear this matter up.

Despite this problem, three networks were successfully translated: one modeling weather prediction, one for intrusion detection, and one for insurance predictions. For each network I measured the time taken to perform inference over 100 iterations with each of three inference engines: junction tree, variable elimination, and Pearl's algorithm. For each iteration there are two steps: integration of the data into the network, and recalculation of marginal node data. I have tracked time separately for each step because they are called separately in the Bayesian net API and because there is often a significant difference in performance for each step.

For extensive background on these inference techniques, consult the Bayes Network Toolbox documentation. In brief, the junction tree method is among the most widely used techniques due to its speed and generalizability and is the default inference engine. The variable elimination method is used mostly because it is relatively easy to implement and very broadly generalizable. It is a fairly straightforward application of Bayesian rules that often results in exponential time solutions. Pearl's algorithm is one of the first useful inference techniques and is now slightly outclassed by newer methods.

For my trials, I construct the initial network using data from the repository. Details on this process are available on the Bayes Net Toolbox web page. I assess the initial log probability associated with the network, for comparison with the value after inference. I then train the network on a set of 100 observations. I then update the network node weights and re-assess the log probability of the network, giving a good estimate of the accuracy of the inference process. Each step of the process is timed in Matlab. I perform these observations for each available inference technique. The Matlab scripts used to perform this testing process are available on the Files page.