View source for Neural Network Program

From wiki.visual-prolog.com
You do not have permission to edit this page, for the following reasons:
The action you have requested is limited to users in the group: emailconfirmed.
You must confirm your email address before editing pages. Please set and validate your email address through your user preferences.
You can view and copy the source of this page.
This article describes a little neural network program that can be downloaded from [http://discuss.visual-prolog.com/viewtopic.php?t=7819 Neural Network Program].

The program is not finished, I hope that people with interest in neural networks will help extend it and maintain this article.

So far the program can:
* Build a network
* Save it to a file
* Load it from a file
* Calculate the network
But it cannot train the net, as I don't know how training works.

== Network, Neurons and Calculation ==

A network (in this program) has a number of neurons, the neurons are connected with weighted signal paths.  Each neuron has a value:
* input neurons receive has their value set from outside (representing external stimuli of the network)
* the the value of the remaining neurons are calculated as the weighted sum of signals from the connected neurons send through a <vp>transfer</vp> function.  In this program the <vp>transfer</vp> function is the same in the entire network.

The neurons are represented as objects.  This makes it possible to represent the connections between neurons as direct links between objects.  Such direct links are very efficient and therefore the network can be calculated rather fast.  In fact, there are links both ways:  each neuron has an <vp>outputTo</vp> list and an <vp>inputFrom</vp> list, the <vp>inputFrom</vp> list contains the weights.

When calculating the network you will set the initial values on the input neurons; invoke the calculate predicate and then retrieve the resulting values from the output neurons.

Calculation goes as follows:
* We run through all the neurons and calculate the number of <vp>remainingInputs</vp>
* Neurons that have zero <vp>remainingInputs</vp> are put in the <vp>ready_fact</vp>
* we continue calculating neurons from the <vp>ready_fact</vp> until it becomes empty, like this:
** We calculate the weighted sum of the inputs and apply the transfer function to obtain the value of the neuron
** We then subtract one from the <vp>remainingInputs</vp> count in all the neurons in the <vp>outputTo</vp> list
** Neurons that have zero <vp>remainingInputs</vp> are put in the <vp>ready_fact</vp>

'''Notice''' this calculation will only work if the network is acyclic. If the network has a cyclic signal path, the  <vp>remainingInputs</vp> count of the involved neurons will never reach zero, and thus never become ready.

'''Notice''' if an input neuron also receive input from other neurons the input value will actually be discarded when the neuron is calculated.  Perhaps it would make sense to weight in the neurons own initial value?

== Serialization ==

It is important to be able to save networks in files and reload them later.  The <vp>netSerialize</vp> package contains classes for saving (i.e. <vp>netSaver</vp>) and loading (i.e. <vp>netLoader</vp>) networks.

The actual serialization is based on serializing a fact database in <vp>serializableNet</vp>.

When saving we build a <vp>serializableNet</vp> and save it, when loading we load a <vp>serializableNet</vp> and build an object net from it.

<vp>serializableNet</vp>'s represent neurons by numbers.  Here is a little example:

<vip>clauses
i(0).
i(1).
i(2).
i(3).
i(4).
i(8).
i(7).
i(6).
i(5).
o(9).
c(5,4,0.25).
c(6,4,0.25).
c(7,4,0.25).
c(8,4,0.25).
c(3,4,0.25).
c(2,4,0.25).
c(1,4,0.25).
c(0,4,0.25).
c(4,9,0.5).</vip>  First is the numbers of the input neurons, then the numbers of the output neurons (here just a single) and finally the connections (there is a signal path from neuron <vp>5</vp> to neuron <vp>4</vp> with weight <vp>0.25</vp>).

<vp>netSaver</vp> and <vp>netLoader</vp> uses red-black-trees (in exactly the same way) to speed up creation of the other representation of the net.  When saving we need to translate neurons to numbers. So whenever we meet a neuron we call a predicate <vp>getOrCreate</vp> which will first look in the red-black-tree to see of that neuron has already been translated, or otherwise create the translation and insert it in the red-black-tree.  The important thing about red-black-trees is that look-up and insert are proportional to the depth of the tree.  And when the tree contains <vp>N</vp> elements it has a maximal depth of <vp>2*log2(N)</vp>.  So if there are for example <vp>65536</vp> elements in the tree the maximal depth is <vp>32</vp>.

== Building Nets ==

It seems obvious that only very small nets can be created by hand in the serialized form.  So the normal strategy for creating a net will be to program a ''net-builder''.  The example program contains a <vp>pyramidBuilder</vp> which can build a pyramid network.

The pyramid has <vp>N</vp> layers. The top layer has a single neuron, which is the (single) output of the network.

The second layer has nine neurons in a 3x3 square.  The middle of these receive input from each of the surrounding neurons, and it sends output to the neuron in the top layer.

The third layer can best be seen like this:  Consider 3x3 squares like in the second layer built together such that they share sides, like this:

 + + + + + + +
 + # + # + # +
 + + + + + + +
 + # + # + # +
 + + + + + + +
 + # + # + # +
 + + + + + + +

Each <vp>#</vp> is the center of a 3x3 square. Each of them receives input from all neighbors.  The nine <vp>#</vp>'s are just below the nine neurons on the second layer, and each of them send input to the neuron above.

Next levels are created in the same way.  So layer <vp>N</vp> is a <vp>(2^N-1)x(2^N-1)</vp> square.

The bottom layer is the input layer, and it can for example be feed with the pixel values from an image.
Return to Neural Network Program.
Retrieved from "https://wiki.visual-prolog.com/index.php?title=Neural_Network_Program"
View source for Neural Network Program

Search

Navigation

external links

collaboration

Personal tools

Views
