Qbasicnews.com
October 18, 2019, 11:16:27 AM
 Welcome, Guest. Please login or register. 1 Hour 1 Day 1 Week 1 Month Forever Login with username, password and session length
 Home Help Search Login Register
 Pages: [1] 2
 Author Topic: Array Not Dimensioned Problem  (Read 10248 times)
Torahteen
Ancient Guru

Posts: 744

 « on: December 28, 2005, 02:14:02 PM »

[syntax="qbasic"]DEFINT A-Z

Declare Function sig(x As Double) As Double
Declare Sub DoNet()
Declare Sub TrainNet()
Declare Sub RunNet()
Declare Sub InitNet()

Const MAX_NODES = 20                    'Maximum number of nuerons in one layer
Const NUM_IL = MAX_NODES                'Number of Nodes in Input Layer
Const NUM_HL = MAX_NODES                'Number of Nodes in Hidden Layer
Const NUM_OL = 10
Const IMG_WIDTH = 12
Const IMG_HEIGHT = 12
Const IMG_TOTAL = IMG_WIDTH * IMG_HEIGHT
Const LR = 0.025

Type Node
wt(1 To MAX_NODES) As Double        'Weights
act As Double                       'Output of this Node
th As Double                        'Threshold
e As Double                         'Error
End Type

Dim Shared il(1 To NUM_IL) As Node             'Input Layer
Dim Shared hl(1 To NUM_HL) As Node             'Hidden Layer
Dim Shared ol(1 To NUM_OL) As Node             'Output Layer

'Digit Images
Dim Shared img(0 To 9,1 To 12, 1 To 12) As Byte
Dim Shared curImg(1 To IMG_TOTAL) As Byte
'Begin Main
For i = 0 To 9
For y = 1 To 12
For x = 1 To 12
Next x
Next y
Next i

End

Sub DoNet()
Dim sum As Double
'Run through nodes in IL
For i = 1 To NUM_IL
sum = 0
For j = 1 To IMG_TOTAL
sum = sum + il(i).wt(j) * curImg(j)
il(i).act = sigmoid(sum - il(i).th)
Next j
Next i

'Run through nodes in HL
For i = 1 To NUM_HL
sum = 0
For j = 1 To NUM_IL
sum = sum + hl(i).wt(j) * il(j).act
hl(i).act = sigmoid(sum - hl(i).th)
Next j
Next i

'Run through nodes in OL
For i = 1 To NUM_OL
sum = 0
For j = 1 To NUM_HL
sum = sum + ol(i).wt(j) * hl(j).act
ol(i).act = sigmoid(sum - ol(i).th)
Next j
Next i
End Function

Sub InitNet()
Dim rand As Double
Randomize Timer

'Input Layer
For i = 1 To NUM_IL
For j = 1 To IMG_TOTAL
il(i).wt(j) = (Rnd(1)/4)
Next j
Next i

'Hidden Layer
For i = 1 To NUM_HL
For j = 1 To NUM_IL
hl(i).wt(j) = (Rnd(1)/4)
Next j
Next i

'Output Layer
For i = 1 To NUM_OL
For j = 1 To NUM_HL
ol(i).wt(j) = (Rnd(1)/4)
Next j
Next i
End Sub

Sub TrainNet()
Dim numCycles As Integer
Dim current As Integer
Dim j As Integer
Dim sum As Double

Input "How many training cycles", numCycles
For i = 1 To numCycles
j = 1

current = Int(Rnd(1) * 10)

For y = 1 To IMG_WIDTH
For x = 1 To IMG_HEIGHT
curImg(j) = img(current, x, y)
j = j + 1
Next x
Next y

RunNet

'Calculate OL Error
For j = 1 To NUM_OL
If j = current Then
ol(j).e = (1 - ol(j).a) * ol(j).a * (1 - ol(j).a)
Else
ol(j).e = (0 - ol(j).a) * ol(j).a * (1 - ol(j).a)
End If
Next j

'Calculate HL Error

For j = 1 To NUM_HL
sum = 0
For k = 1 To NUM_OL
sum = sum + ol(k).e * ol(k).wt(j)
hl(j).e = hl(j).act * (1 - hl(j).act) * sum
Next k
Next j

For j = 1 To NUM_OL
For k = 1 To NUM_HL
ol(j).wt(k) = ol(j).wt(k) + LR * ol(j).e * hl(k).act
ol(j).th = ol(j).th - LR * ol(j).e
Next k
Next j

For j = 1 To NUM_HL
For k = 1 To NUM_IL
hl(j).wt(k) = hl(j).wt(k) + LR * hl(j).e * il(k).act
hl(j).th = hl(j).th - LR * hl(j).e
Next k
Next j

Print ".";
Next i
End Sub

Sub RunNet()
Dim quit As Byte
Dim num As Byte

Input "Which number do you want to test?", num

For y = 1 To IMG_WIDTH
For x = 1 To IMG_HEIGHT
curImg(j) = img(num, x, y)
j = j + 1
Next x
Next y

DoNet

For i = 1 To NUM_OL
If ol(i).act > 0.9 Then
Print "The number is: ";
Print i
Exit For
End If
Next i
End Sub

Function sig(x As Double)
sig = 1/(1 + Exp(-x))
End Function[/syntax]

I'm getting an error on line 51.

Code:
Array not dimensioned, before: '('
il(i).act = sigmoid(sum - il(i).th)

It should have been declared already. I have as a SHARED variable too. Why isn't it recognizing it?

P.S. I have removed the DATA statements. They would've add another 100 or so lines.
 Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
speedlemon
I hold this place together

Posts: 874

 « Reply #1 on: December 28, 2005, 02:21:26 PM »

It isn't recognizing sigmoid, which isn't defined (as far as I can tell).
 Logged
Torahteen
Ancient Guru

Posts: 744

 « Reply #2 on: December 28, 2005, 03:11:02 PM »

Sigmoid is a function. It's il() that it isn't recognizing.

Edit: Ah crap! I forgot that the function is sig not sigmoid.
 Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
Torahteen
Ancient Guru

Posts: 744

 « Reply #3 on: December 28, 2005, 03:48:43 PM »

Okay. So now any of you AI guys able to help me with my Neural Network? It's not giving correct results. I even ran 10,000 training cycles on it. Here is my code.

[syntax="qbasic"]DEFINT A-Z

Declare Function sig(x As Double) As Double
Declare Sub DoNet()
Declare Sub TrainNet()
Declare Sub RunNet()
Declare Sub InitNet()

Const MAX_NODES = 20                    'Maximum number of nuerons in one layer
Const NUM_IL = MAX_NODES                'Number of Nodes in Input Layer
Const NUM_HL = MAX_NODES                'Number of Nodes in Hidden Layer
Const NUM_OL = 10
Const IMG_WIDTH = 12
Const IMG_HEIGHT = 12
Const IMG_TOTAL = IMG_WIDTH * IMG_HEIGHT
Const LR = 0.025

Type Node
wt(1 To MAX_NODES) As Double        'Weights
act As Double                       'Output of this Node
th As Double                        'Threshold
e As Double                         'Error
End Type

Dim Shared il(1 To NUM_IL) As Node             'Input Layer
Dim Shared hl(1 To NUM_HL) As Node             'Hidden Layer
Dim Shared ol(1 To NUM_OL) As Node             'Output Layer

'Digit Images
Dim Shared img(0 To 9,1 To 12, 1 To 12) As Byte
Dim Shared curImg(1 To IMG_TOTAL) As Byte
'Begin Main
For i = 0 To 9
For y = 1 To 12
For x = 1 To 12
Next x
Next y
Next i

Print "First, see how the NN performs before training"
RunNet
Sleep
Cls
Print "Now training network"
TrainNet
Cls
Print "Now see how the NN performs after training"
RunNet
Sleep
End

Sub DoNet()
Dim sum As Double
'Run through nodes in IL
For i = 1 To NUM_IL
sum = 0
For j = 1 To IMG_TOTAL
sum = sum + il(i).wt(j) * curImg(j)
il(i).act = sig(sum - il(i).th)
Next j
Next i

'Run through nodes in HL
For i = 1 To NUM_HL
sum = 0
For j = 1 To NUM_IL
sum = sum + hl(i).wt(j) * il(j).act
hl(i).act = sig(sum - hl(i).th)
Next j
Next i

'Run through nodes in OL
For i = 1 To NUM_OL
sum = 0
For j = 1 To NUM_HL
sum = sum + ol(i).wt(j) * hl(j).act
ol(i).act = sig(sum - ol(i).th)
Next j
Next i
End Sub

Sub InitNet()
Randomize Timer

'Input Layer
For i = 1 To NUM_IL
For j = 1 To IMG_TOTAL
il(i).wt(j) = (Rnd(1))
Next j
Next i

'Hidden Layer
For i = 1 To NUM_HL
For j = 1 To NUM_IL
hl(i).wt(j) = (Rnd(1))
Next j
Next i

'Output Layer
For i = 1 To NUM_OL
For j = 1 To NUM_HL
ol(i).wt(j) = (Rnd(1))
Next j
Next i
End Sub

Sub TrainNet()
Dim numCycles As Integer
Dim current As Integer
Dim j As Integer
Dim sum As Double

Input "How many training cycles", numCycles
For i = 1 To numCycles
j = 1

current = Int(Rnd(1) * 10)

For y = 1 To IMG_WIDTH
For x = 1 To IMG_HEIGHT
curImg(j) = img(current, x, y)
j = j + 1
Next x
Next y

DoNet

'Calculate OL Error
For j = 1 To NUM_OL
If j = current Then
ol(j).e = (1 - ol(j).act) * ol(j).act * (1 - ol(j).act)
Else
ol(j).e = (0 - ol(j).act) * ol(j).act * (1 - ol(j).act)
End If
Next j

'Calculate HL Error

For j = 1 To NUM_HL
sum = 0
For k = 1 To NUM_OL
sum = sum + ol(k).e * ol(k).wt(j)
hl(j).e = hl(j).act * (1 - hl(j).act) * sum
Next k
Next j

For j = 1 To NUM_OL
For k = 1 To NUM_HL
ol(j).wt(k) = ol(j).wt(k) + LR * ol(j).e * hl(k).act
ol(j).th = ol(j).th - LR * ol(j).e
Next k
Next j

For j = 1 To NUM_HL
For k = 1 To NUM_IL
hl(j).wt(k) = hl(j).wt(k) + LR * hl(j).e * il(k).act
hl(j).th = hl(j).th - LR * hl(j).e
Next k
Next j

Print ".";
Next i
End Sub

Sub RunNet()
Dim quit As Byte
Dim num As Byte

Input "Which number do you want to test?", num

For y = 1 To IMG_WIDTH
For x = 1 To IMG_HEIGHT
curImg(j) = img(num, x, y)
j = j + 1
Next x
Next y

DoNet

For i = 1 To NUM_OL
Print "Value of output node ";
Print i - 1
Print ol(i).act
Next i
End Sub

Function sig(x As Double) As Double
sig = 1/(1 + Exp(-x))
End Function

0:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

1:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,1,1,0,0,0,0,0
Data 0,0,0,0,1,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,1,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

2:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,1,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

3:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

4:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,0,1,1,0,0,0
Data 0,0,0,0,0,0,1,0,1,0,0,0
Data 0,0,0,0,0,1,0,0,1,0,0,0
Data 0,0,0,0,1,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,1,1,1,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

5:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,1,1,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,0,0,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,1,1,1,1,0,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

6:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,1,0,0,0,0,0,0,0,0,0
Data 0,0,1,0,0,0,0,0,0,0,0,0
Data 0,0,1,0,0,0,0,0,0,0,0,0
Data 0,0,1,0,1,1,1,1,0,0,0,0
Data 0,0,1,1,0,0,0,0,1,0,0,0
Data 0,0,1,0,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

7:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,1,0,0,0,0,0,0
Data 0,0,0,0,0,1,0,0,0,0,0,0
Data 0,0,0,0,1,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

8:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

9:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,1,1,0,0
Data 0,0,0,0,1,1,1,1,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0[/syntax]
 Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
vspickelen
Member

Posts: 26

 « Reply #4 on: December 29, 2005, 09:22:35 AM »

You started out well, Torahteen, but:
what's your program supposed to do in the 1st place, and what output do we have to expect?
It looks like backpropagation to me - quite ambitious to take a bite on,
or to debug, for that matter. ( that old SOM really is much easier )

Some remarks though:

Sub InitNet is never called.

You're probably misusing the 'sum'-result in sub DoNet,
i'd put it like this:

Code:
FOR i = 1 TO NUM_IL
sum = 0
FOR j = 1 TO IMG_TOTAL
sum = sum + il(i).wt(j) * curImg(j)
NEXT j
il(i).act = sig(sum - il(i).th)
NEXT i
etc. (Tip: reduce code by passing each layer as a subroutine argument.)

The size of the "input layer" (20) doesn't match your input (12 * 12)
In fact, the input layer reduces to activations only, you can strip
DIM SHARED il(1 TO NUM_IL) AS Node             'Input Layer
because three net layers are connected by two sets of weights only.
(144 -> 20 and 20 -> 10 in this case.)

This var is never used in sub RunNet:
DIM quit AS Byte
Though it might be important somehow.

I've compiled w/FB 016b, and This doesn't show:
INPUT "How many training cycles", numCycles
Though it should. A compiler bug, i suppose.
Quick patch:
print "How many training cycles";
INPUT ; numCycles

I'm sure you can work this out,
vspickelen
 Logged
Torahteen
Ancient Guru

Posts: 744

 « Reply #5 on: December 29, 2005, 01:02:52 PM »

Thanks for the reply. You're correct, it is a backprop NN. I am trying to recognize the digits from 0-9. The goal is to have 10 outputs, each representing one of the digits. If the answer is... say 7, then the output for that node should be a decent amount larger than the others. Make sense?

I forgot to call InitNet. No wonder my weights weren't getting set up.

Quote
The size of the "input layer" (20) doesn't match your input (12 * 12)
In fact, the input layer reduces to activations only, you can strip
DIM SHARED il(1 TO NUM_IL) AS Node 'Input Layer
because three net layers are connected by two sets of weights only.
(144 -> 20 and 20 -> 10 in this case.)

Aren't I supposed to have 3 layers? Input, Hidden, and Ouput? I feed all of the information into input layer, which get's calculated using the Sigmoid activation function, and that becomes the weights for the Hidden layer, and so on. What exactly am I misunderstanding here?

It doesn't work in v0.16b eh? Hmm... I don't know why that is. Check the changes log to see if there is some new feature I don't know about.
 Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
KiZ
__/--\__

Posts: 2879

 « Reply #6 on: December 29, 2005, 01:56:23 PM »

Quote from: "Torahteen"
It doesn't work in v0.16b eh? Hmm... I don't know why that is. Check the changes log to see if there is some new feature I don't know about.

No, it is a bug in the 0.16 cvs. Vic has found and addressed the problem, with the help of Antoni.
 Logged
Torahteen
Ancient Guru

Posts: 744

 « Reply #7 on: December 29, 2005, 02:46:13 PM »

Here is an example of my output:

Code:
First, see how the NN performs before training
Which number do you want to test? [color=red]3[/color]

Value of output node  0
0.8583611546483856
Value of output node  1
0.8843891112292291
Value of output node  2
0.9123689633377022
Value of output node  3
0.9446206548675346
Value of output node  4
0.9155865835716671
Value of output node  5
0.8699179837462892
Value of output node  6
0.9314443557889023
Value of output node  7
0.9060621891248365
Value of output node  8
0.8857971181483882
Value of output node  9
0.889199395400074
-----------------------------------------------
Now training network
How many training cycles [color=red]10000[/color]
-----------------------------------------------
Now see how the NN performs after training
Which number do you want to test? [color=red]3[/color]

Value of output node  0
7.923202849710609e-002
Value of output node  1
7.799303294393579e-002
Value of output node  2
0.1093219822478929
Value of output node  3
9.059998525862116e-002
Value of output node  4
0.1094775032751099
Value of output node  5
9.746875324353162e-002
Value of output node  6
0.1129354063528408
Value of output node  7
8.255943958120357e-002
Value of output node  8
0.1100881485219571
Value of output node  9
7.630521829981274e-003

Red = Input

Anyway, there are a few problems. As far as I know, the output should always be a number between 0 and 1 because of the sigmoid function. The sigmoid function looks fine to me, so what's the problem?

In the above situation, the output of node 3 should have been quite a bit higher (relatively speaking) than the others, signifying that 3 is what the NN chose.
 Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
vspickelen
Member

Posts: 26

 « Reply #8 on: December 29, 2005, 02:50:16 PM »

Quote from: "Torahteen"
Aren't I supposed to have 3 layers? Input, Hidden, and Ouput?

Compare it with this:

1 -> 2 -> 3

That's three layers, and two sets of weights (->) to connect them.

But the first layer is implied, so to say: it's your curimg() byte array, that's all you need there.

Now attach the connections between input and hidden layer to the latter,
Code:
DIM SHARED hl(1 TO NUM_HL) AS Node             'Hidden Layer
idem for the connections between hidden and output layer,
Code:
DIM SHARED ol(1 TO NUM_OL) AS Node             'Output Layer
that's only two shared Node-structures, and not three.

Your feedforward routine now becomes:
Code:
SUB DoNet()
DIM sum AS single

'Run through nodes in HL
FOR i = 1 TO NUM_HL
sum = 0
FOR j = 1 TO IMG_TOTAL
sum = sum + hl(i).wt(j) * curImg(j)
NEXT j
hl(i).act = sig(sum - hl(i).th)
NEXT i

'Run through nodes in OL
FOR i = 1 TO NUM_OL
sum = 0
FOR j = 1 TO NUM_HL
sum = sum + ol(i).wt(j) * hl(j).act
NEXT j
ol(i).act = sig(sum - ol(i).th)
NEXT i
END SUB

I almost forgot:
Since your input is binary, you need a constant GAIN factor, like
Code:
sig = 1/(1 + EXP(-GAIN * x))
to make the sigmoid steeper, else the output won't separate.
This gain also enters the derivatives of the sig-function.
I just got a stripped-down version of your program working fine with
GAIN = 5 and LR = 0.1

Giving away all changes will of course spoil your fun, so I won't.
vspickelen
 Logged
Torahteen
Ancient Guru

Posts: 744

 « Reply #9 on: December 29, 2005, 03:02:28 PM »

But how do I do BackPropagation for the hidden layer?
 Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
vspickelen
Member

Posts: 26

 « Reply #10 on: December 29, 2005, 03:29:23 PM »

Quote from: "Torahteen"
But how do I do BackPropagation for the hidden layer?

You already got that down, man, it's under 'Calculate HL Error
vspickelen (really likes this)
 Logged
Torahteen
Ancient Guru

Posts: 744

 « Reply #11 on: December 29, 2005, 03:45:29 PM »

Hmm... It's still not giving me correct results. Here is the code, followed by output.

[syntax="qbasic"]DEFINT A-Z

Declare Function sig(x As Double) As Double
Declare Sub DoNet()
Declare Sub TrainNet()
Declare Sub RunNet()
Declare Sub InitNet()

Const MAX_NODES = 20                    'Maximum number of nuerons in one layer
Const NUM_IL = MAX_NODES                'Number of Nodes in Input Layer
Const NUM_HL = MAX_NODES                'Number of Nodes in Hidden Layer
Const NUM_OL = 10
Const IMG_WIDTH = 12
Const IMG_HEIGHT = 12
Const IMG_TOTAL = IMG_WIDTH * IMG_HEIGHT
Const LR = 0.05
Const GAIN = 5

Type Node
wt(1 To MAX_NODES) As Double        'Weights
act As Double                       'Output of this Node
th As Double                        'Threshold
e As Double                         'Error
End Type

Dim Shared hl(1 To NUM_HL) As Node             'Hidden Layer
Dim Shared ol(1 To NUM_OL) As Node             'Output Layer

'Digit Images
Dim Shared img(0 To 9,1 To 12, 1 To 12) As Byte
Dim Shared curImg(1 To IMG_TOTAL) As Byte
'Begin Main
For i = 0 To 9
For y = 1 To 12
For x = 1 To 12
Next x
Next y
Next i

InitNet
Print "First, see how the NN performs before training"
RunNet
Sleep
Cls
Print "Now training network"
TrainNet
Cls
Print "Now see how the NN performs after training"
RunNet
Sleep
End

SUB DoNet()
DIM sum AS single

'Run through nodes in HL
FOR i = 1 TO NUM_HL
sum = 0
FOR j = 1 TO IMG_TOTAL
sum = sum + hl(i).wt(j) * curImg(j)
NEXT j
hl(i).act = sig(sum - hl(i).th)
NEXT i

'Run through nodes in OL
FOR i = 1 TO NUM_OL
sum = 0
FOR j = 1 TO NUM_HL
sum = sum + ol(i).wt(j) * hl(j).act
NEXT j
ol(i).act = sig(sum - ol(i).th)
NEXT i
END SUB

Sub InitNet()
Randomize Timer

'Hidden Layer
For i = 1 To NUM_HL
For j = 1 To NUM_IL
hl(i).wt(j) = (Rnd(1)/4)
Next j
Next i

'Output Layer
For i = 1 To NUM_OL
For j = 1 To NUM_HL
ol(i).wt(j) = (Rnd(1)/4)
Next j
Next i
End Sub

Sub TrainNet()
Dim numCycles As Integer
Dim current As Integer
Dim j As Integer
Dim sum As Double

Input "How many training cycles", numCycles
For i = 1 To numCycles
j = 1

current = Int(Rnd(1) * 10)

For y = 1 To IMG_WIDTH
For x = 1 To IMG_HEIGHT
curImg(j) = img(current, x, y)
j = j + 1
Next x
Next y

DoNet

'Calculate OL Error
For j = 1 To NUM_OL
If j = current Then
ol(j).e = (1 - ol(j).act) * ol(j).act * (1 - ol(j).act)
Else
ol(j).e = (0 - ol(j).act) * ol(j).act * (1 - ol(j).act)
End If
Next j

'Calculate HL Error

For j = 1 To NUM_HL
sum = 0
For k = 1 To NUM_OL
sum = sum + ol(k).e * ol(k).wt(j)
hl(j).e = hl(j).act * (1 - hl(j).act) * sum
Next k
Next j

For j = 1 To NUM_OL
For k = 1 To NUM_HL
ol(j).wt(k) = ol(j).wt(k) + LR * ol(j).e * hl(k).act
ol(j).th = ol(j).th - LR * ol(j).e
Next k
Next j

For j = 1 To NUM_HL
For k = 1 To IMG_TOTAL
hl(j).wt(k) = hl(j).wt(k) + LR * hl(j).e * curImg(k)
hl(j).th = hl(j).th - LR * hl(j).e
Next k
Next j

Print ".";
Next i
End Sub

Sub RunNet()
Dim quit As Byte
Dim num As Byte

Input "Which number do you want to test?", num

For y = 1 To IMG_WIDTH
For x = 1 To IMG_HEIGHT
curImg(j) = img(num, x, y)
j = j + 1
Next x
Next y

Print "You chose this number"
For y = 1 To IMG_WIDTH
For x = 1 To IMG_HEIGHT
If img(num, x, y) = 0 Then
Pset(x + 100,y),15
End If
Next x
Next y

DoNet

For i = 1 To NUM_OL
Print "Value of output node ";
Print i - 1
Print ol(i).act
Next i
End Sub

Function sig(x As Double) As Double
sig = 1/(1 + Exp(-GAIN * x))
End Function

0:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

1:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,1,1,0,0,0,0,0
Data 0,0,0,0,1,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,1,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

2:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,1,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

3:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

4:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,0,1,1,0,0,0
Data 0,0,0,0,0,0,1,0,1,0,0,0
Data 0,0,0,0,0,1,0,0,1,0,0,0
Data 0,0,0,0,1,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,1,1,1,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

5:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,1,1,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,0,0,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,1,1,1,1,0,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

6:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,1,0,0,0,0,0,0,0,0,0
Data 0,0,1,0,0,0,0,0,0,0,0,0
Data 0,0,1,0,0,0,0,0,0,0,0,0
Data 0,0,1,0,1,1,1,1,0,0,0,0
Data 0,0,1,1,0,0,0,0,1,0,0,0
Data 0,0,1,0,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

7:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,1,0,0,0,0,0,0
Data 0,0,0,0,0,1,0,0,0,0,0,0
Data 0,0,0,0,1,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

8:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

9:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,1,1,0,0
Data 0,0,0,0,1,1,1,1,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0[/syntax]

Code:
First, see how the NN performs before training
Which number do you want to test? 2
You chose this number
Value of output node  0
0.9999946772066498
Value of output node  1
0.9998794164495293
Value of output node  2
0.9999991414321088
Value of output node  3
0.9999969437467565
Value of output node  4
0.999989835095912
Value of output node  5
0.999954813071024
Value of output node  6
0.9999966488577209
Value of output node  7
0.999993763156863
Value of output node  8
0.9999949938112089
Value of output node  9
0.9999959078217268
------------------------------

Now training network
How many training cycles 5000
--------------------------------

Now see how the NN performs after training
Which number do you want to test? 5
You chose this number
Value of output node  0
0.9999930039506503
Value of output node  1
4.505908216785114e-002
Value of output node  2
0.9999991069143672
Value of output node  3
0.9999964581486549
Value of output node  4
0.9999813061324169
Value of output node  5
7.279828978456202e-002
Value of output node  6
0.9999960467329503
Value of output node  7
0.9999913546369986
Value of output node  8
0.999993535158855
Value of output node  9
0.9999948572862908

Any idea why it is still giving results above 1?
 Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
Deleter
Na_th_an

Posts: 1292

 « Reply #12 on: December 29, 2005, 06:14:21 PM »

none of those results are above 1  ::
 Logged

vspickelen
Member

Posts: 26

 « Reply #13 on: December 29, 2005, 07:26:32 PM »

O.k, last post for today

Sad to say, your code needs more tidying up still.
Things that will be quite obvious, once you understand
what's goin' on in all those lil'nifty-nested for-loops.

The constants are messed up, this will do
Code:
CONST IMG_WIDTH = 12
CONST IMG_HEIGHT = 12
CONST NUM_IL = IMG_WIDTH * IMG_HEIGHT    'Number of Nodes in Input Layer
CONST NUM_HL = 20                        'Number of Nodes in Hidden Layer
CONST NUM_OL = 10                        'Number of Nodes in Output Layer
CONST LR = 0.1
CONST GAIN = 5

Replace MAX_NODES and IMG_TOTAL accordingly.

Use
Code:
hl(i).wt(j) = (RND - .5) * .1
for small zero mean random weights

While this is fine
Code:
current = INT(RND * NUM_OL)
it can still go wrong
Code:
FOR j = 1 TO NUM_OL
IF j = current THEN

I've changed the signs before the bias terms here,
guess you can justify the displaced outer-loop exprssions yourself

Code:
'Calculate HL Error

FOR j = 1 TO NUM_HL
sum = 0
FOR k = 1 TO NUM_OL
sum = sum + ol(k).e * ol(k).wt(j)
hl(j).e = hl(j).act * (1 - hl(j).act) * sum
NEXT k
NEXT j

FOR j = 1 TO NUM_OL
FOR k = 1 TO NUM_HL
ol(j).wt(k) = ol(j).wt(k) + LR * ol(j).e * hl(k).act
ol(j).th = ol(j).th + LR * ol(j).e
NEXT k
NEXT j

FOR j = 1 TO NUM_HL
FOR k = 1 TO IMG_TOTAL
hl(j).wt(k) = hl(j).wt(k) + LR * hl(j).e * curImg(k)
hl(j).th = hl(j).th + LR * hl(j).e
NEXT k
NEXT j

This innocent one is a classic:

Code:
INPUT "Which number do you want to test?", num

FOR y = 1 TO IMG_WIDTH
FOR x = 1 TO IMG_HEIGHT
curImg(j) = img(num, x, y)
j = j + 1
NEXT x
NEXT y

might drive you nuts.

The only thing left then, is to remember that
Quote from: "vspickelen"
This GAIN also enters the derivatives of the sig-function.

Good night  ( snow here )
 Logged
Torahteen
Ancient Guru

Posts: 744

 « Reply #14 on: December 30, 2005, 01:29:43 AM »

Hmm... Maybe I'm just a bad programmer. Do you have a working version of my code? Is it actually presenting the correct results? Could you post the code? I view this as a learning experience, so don't worry about spoiling my fun .
 Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
 Pages: [1] 2