Neural Network Program

This article describes a little neural network program that can be downloaded from Neural Network Program.

The program is not finished, I hope that people with interest in neural networks will help extend it and maintain this article.

So far the program can:

• Build a network
• Save it to a file
• Load it from a file
• Calculate the network

But it cannot train the net, as I don't know how training works.

Network, Neurons and Calculation

A network (in this program) has a number of neurons, the neurons are connected with weighted signal paths. Each neuron has a value:

• input neurons receive has their value set from outside (representing external stimuli of the network)
• the the value of the remaining neurons are calculated as the weighted sum of signals from the connected neurons send through a transfer function. In this program the transfer function is the same in the entire network.

The neurons are represented as objects. This makes it possible to represent the connections between neurons as direct links between objects. Such direct links are very efficient and therefore the network can be calculated rather fast. In fact, there are links both ways: each neuron has an outputTo list and an inputFrom list, the inputFrom list contains the weights.

When calculating the network you will set the initial values on the input neurons; invoke the calculate predicate and then retrieve the resulting values from the output neurons.

Calculation goes as follows:

• We run through all the neurons and calculate the number of remainingInputs
• Neurons that have zero remainingInputs are put in the ready_fact
• we continue calculating neurons from the ready_fact until it becomes empty, like this:
  • We calculate the weighted sum of the inputs and apply the transfer function to obtain the value of the neuron
  • We then subtract one from the remainingInputs count in all the neurons in the outputTo list
  • Neurons that have zero remainingInputs are put in the ready_fact

Notice this calculation will only work if the network is acyclic. If the network has a cyclic signal path, the remainingInputs count of the involved neurons will never reach zero, and thus never become ready.

Notice if an input neuron also receive input from other neurons the input value will actually be discarded when the neuron is calculated. Perhaps it would make sense to weight in the neurons own initial value?

Serialization

It is important to be able to save networks in files and reload them later. The netSerialize package contains classes for saving (i.e. netSaver) and loading (i.e. netLoader) networks.

The actual serialization is based on serializing a fact database in serializableNet.

When saving we build a serializableNet and save it, when loading we load a serializableNet and build an object net from it.

serializableNet's represent neurons by numbers. Here is a little example:

clauses
    i(0).
    i(1).
    i(2).
    i(3).
    i(4).
    i(8).
    i(7).
    i(6).
    i(5).
 
clauses
    o(9).
 
clauses
    c(5,4,0.25).
    c(6,4,0.25).
    c(7,4,0.25).
    c(8,4,0.25).
    c(3,4,0.25).
    c(2,4,0.25).
    c(1,4,0.25).
    c(0,4,0.25).
    c(4,9,0.5).

First is the numbers of the input neurons, then the numbers of the output neurons (here just a single) and finally the connections (there is a signal path from neuron 5 to neuron 4 with weight 0.25).

netSaver and netLoader uses red-black-trees (in exactly the same way) to speed up creation of the other representation of the net. When saving we need to translate neurons to numbers. So whenever we meet a neuron we call a predicate getOrCreate which will first look in the red-black-tree to see of that neuron has already been translated, or otherwise create the translation and insert it in the red-black-tree. The important thing about red-black-trees is that look-up and insert are proportional to the depth of the tree. And when the tree contains N elements it has a maximal depth of 2*log2(N). So if there are for example 65536 elements in the tree the maximal depth is 32.

Building Nets

It seems obvious that only very small nets can be created by hand in the serialized form. So the normal strategy for creating a net will be to program a net-builder. The example program contains a pyramidBuilder which can build a pyramid network.

The pyramid has N layers. The top layer has a single neuron, which is the (single) output of the network.

The second layer has nine neurons in a 3x3 square. The middle of these receive input from each of the surrounding neurons, and it sends output to the neuron in the top layer.

The third layer can best be seen like this: Consider 3x3 squares like in the second layer built together such that they share sides, like this:

+ + + + + + +
+ # + # + # +
+ + + + + + +
+ # + # + # +
+ + + + + + +
+ # + # + # +
+ + + + + + +

Each # is the center of a 3x3 square. Each of them receives input from all neighbors. The nine #'s are just below the nine neurons on the second layer, and each of them send input to the neuron above.

Next levels are created in the same way. So layer N is a (2^N-1)x(2^N-1) square.

The bottom layer is the input layer, and it can for example be feed with the pixel values from an image.