Qbasicnews.com
October 18, 2019, 11:16:27 AM *
Welcome, Guest. Please login or register.

Login with username, password and session length
News: Back to Qbasicnews.com | QB Online Help | FAQ | Chat | All Basic Code | QB Knowledge Base
 
   Home   Help Search Login Register  
Pages: [1] 2
  Print  
Author Topic: Array Not Dimensioned Problem  (Read 10248 times)
Torahteen
Ancient Guru
****
Posts: 744



« on: December 28, 2005, 02:14:02 PM »

[syntax="qbasic"]DEFINT A-Z

Declare Function sig(x As Double) As Double
Declare Sub DoNet()
Declare Sub TrainNet()
Declare Sub RunNet()
Declare Sub InitNet()

Const MAX_NODES = 20                    'Maximum number of nuerons in one layer
Const NUM_IL = MAX_NODES                'Number of Nodes in Input Layer
Const NUM_HL = MAX_NODES                'Number of Nodes in Hidden Layer
Const NUM_OL = 10
Const IMG_WIDTH = 12
Const IMG_HEIGHT = 12
Const IMG_TOTAL = IMG_WIDTH * IMG_HEIGHT
Const LR = 0.025

Type Node
    wt(1 To MAX_NODES) As Double        'Weights
    act As Double                       'Output of this Node
    th As Double                        'Threshold
    e As Double                         'Error
End Type

Dim Shared il(1 To NUM_IL) As Node             'Input Layer
Dim Shared hl(1 To NUM_HL) As Node             'Hidden Layer
Dim Shared ol(1 To NUM_OL) As Node             'Output Layer
   
'Digit Images
Dim Shared img(0 To 9,1 To 12, 1 To 12) As Byte
Dim Shared curImg(1 To IMG_TOTAL) As Byte
'Begin Main
'Load the Images
For i = 0 To 9
    For y = 1 To 12
        For x = 1 To 12
            Read img(i,x,y)
        Next x
    Next y
Next i

End

Sub DoNet()
    Dim sum As Double
    'Run through nodes in IL
    For i = 1 To NUM_IL
        sum = 0
        For j = 1 To IMG_TOTAL
            sum = sum + il(i).wt(j) * curImg(j)
            il(i).act = sigmoid(sum - il(i).th)
        Next j
    Next i
   
    'Run through nodes in HL
    For i = 1 To NUM_HL
        sum = 0
        For j = 1 To NUM_IL
            sum = sum + hl(i).wt(j) * il(j).act
            hl(i).act = sigmoid(sum - hl(i).th)
        Next j
    Next i
   
    'Run through nodes in OL
    For i = 1 To NUM_OL
        sum = 0
        For j = 1 To NUM_HL
            sum = sum + ol(i).wt(j) * hl(j).act
            ol(i).act = sigmoid(sum - ol(i).th)
        Next j
    Next i
End Function

Sub InitNet()
    Dim rand As Double
    Randomize Timer
   
    'Input Layer
    For i = 1 To NUM_IL
        For j = 1 To IMG_TOTAL
            il(i).wt(j) = (Rnd(1)/4)
        Next j
    Next i
   
    'Hidden Layer
    For i = 1 To NUM_HL
        For j = 1 To NUM_IL
            hl(i).wt(j) = (Rnd(1)/4)
        Next j
    Next i
   
    'Output Layer
    For i = 1 To NUM_OL
        For j = 1 To NUM_HL
            ol(i).wt(j) = (Rnd(1)/4)
        Next j
    Next i
End Sub

Sub TrainNet()
    Dim numCycles As Integer
    Dim current As Integer
    Dim j As Integer
    Dim sum As Double
   
    Input "How many training cycles", numCycles
    For i = 1 To numCycles
        j = 1
       
        current = Int(Rnd(1) * 10)
           
            For y = 1 To IMG_WIDTH
                For x = 1 To IMG_HEIGHT
                    curImg(j) = img(current, x, y)
                    j = j + 1
                Next x
            Next y
           
        RunNet
       
        'Calculate OL Error
        For j = 1 To NUM_OL
            If j = current Then
                ol(j).e = (1 - ol(j).a) * ol(j).a * (1 - ol(j).a)
            Else
                ol(j).e = (0 - ol(j).a) * ol(j).a * (1 - ol(j).a)
            End If
        Next j
       
        'Calculate HL Error
       
        For j = 1 To NUM_HL
            sum = 0
            For k = 1 To NUM_OL
                sum = sum + ol(k).e * ol(k).wt(j)
                hl(j).e = hl(j).act * (1 - hl(j).act) * sum
            Next k
        Next j
       
        'Adjust Weights
        For j = 1 To NUM_OL
            For k = 1 To NUM_HL
                ol(j).wt(k) = ol(j).wt(k) + LR * ol(j).e * hl(k).act
                ol(j).th = ol(j).th - LR * ol(j).e
            Next k
        Next j
       
        For j = 1 To NUM_HL
            For k = 1 To NUM_IL
                hl(j).wt(k) = hl(j).wt(k) + LR * hl(j).e * il(k).act
                hl(j).th = hl(j).th - LR * hl(j).e
            Next k
        Next j
       
        Print ".";
    Next i
End Sub

Sub RunNet()
    Dim quit As Byte
    Dim num As Byte
   
    Input "Which number do you want to test?", num
   
    For y = 1 To IMG_WIDTH
        For x = 1 To IMG_HEIGHT
            curImg(j) = img(num, x, y)
            j = j + 1
        Next x
    Next y
   
    DoNet
   
    For i = 1 To NUM_OL
        If ol(i).act > 0.9 Then
            Print "The number is: ";
            Print i
            Exit For
        End If
    Next i
End Sub

Function sig(x As Double)
    sig = 1/(1 + Exp(-x))
End Function[/syntax]

I'm getting an error on line 51.

Code:
Array not dimensioned, before: '('
il(i).act = sigmoid(sum - il(i).th)


It should have been declared already. I have as a SHARED variable too. Why isn't it recognizing it?

P.S. I have removed the DATA statements. They would've add another 100 or so lines.
Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
speedlemon
I hold this place together
*****
Posts: 874



« Reply #1 on: December 28, 2005, 02:21:26 PM »

It isn't recognizing sigmoid, which isn't defined (as far as I can tell).
Logged
Torahteen
Ancient Guru
****
Posts: 744



« Reply #2 on: December 28, 2005, 03:11:02 PM »

Sigmoid is a function. It's il() that it isn't recognizing.

Edit: Ah crap! I forgot that the function is sig not sigmoid.
Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
Torahteen
Ancient Guru
****
Posts: 744



« Reply #3 on: December 28, 2005, 03:48:43 PM »

Okay. So now any of you AI guys able to help me with my Neural Network? It's not giving correct results. I even ran 10,000 training cycles on it. Here is my code.

[syntax="qbasic"]DEFINT A-Z

Declare Function sig(x As Double) As Double
Declare Sub DoNet()
Declare Sub TrainNet()
Declare Sub RunNet()
Declare Sub InitNet()

Const MAX_NODES = 20                    'Maximum number of nuerons in one layer
Const NUM_IL = MAX_NODES                'Number of Nodes in Input Layer
Const NUM_HL = MAX_NODES                'Number of Nodes in Hidden Layer
Const NUM_OL = 10
Const IMG_WIDTH = 12
Const IMG_HEIGHT = 12
Const IMG_TOTAL = IMG_WIDTH * IMG_HEIGHT
Const LR = 0.025

Type Node
    wt(1 To MAX_NODES) As Double        'Weights
    act As Double                       'Output of this Node
    th As Double                        'Threshold
    e As Double                         'Error
End Type

Dim Shared il(1 To NUM_IL) As Node             'Input Layer
Dim Shared hl(1 To NUM_HL) As Node             'Hidden Layer
Dim Shared ol(1 To NUM_OL) As Node             'Output Layer
   
'Digit Images
Dim Shared img(0 To 9,1 To 12, 1 To 12) As Byte
Dim Shared curImg(1 To IMG_TOTAL) As Byte
'Begin Main
'Load the Images
For i = 0 To 9
    For y = 1 To 12
        For x = 1 To 12
            Read img(i,x,y)
        Next x
    Next y
Next i

Print "First, see how the NN performs before training"
RunNet
Sleep
Cls
Print "Now training network"
TrainNet
Cls
Print "Now see how the NN performs after training"
RunNet
Sleep
End

Sub DoNet()
    Dim sum As Double
    'Run through nodes in IL
    For i = 1 To NUM_IL
        sum = 0
        For j = 1 To IMG_TOTAL
            sum = sum + il(i).wt(j) * curImg(j)
            il(i).act = sig(sum - il(i).th)
        Next j
    Next i
   
    'Run through nodes in HL
    For i = 1 To NUM_HL
        sum = 0
        For j = 1 To NUM_IL
            sum = sum + hl(i).wt(j) * il(j).act
            hl(i).act = sig(sum - hl(i).th)
        Next j
    Next i
   
    'Run through nodes in OL
    For i = 1 To NUM_OL
        sum = 0
        For j = 1 To NUM_HL
            sum = sum + ol(i).wt(j) * hl(j).act
            ol(i).act = sig(sum - ol(i).th)
        Next j
    Next i
End Sub

Sub InitNet()
    Randomize Timer
   
    'Input Layer
    For i = 1 To NUM_IL
        For j = 1 To IMG_TOTAL
            il(i).wt(j) = (Rnd(1))
        Next j
    Next i
   
    'Hidden Layer
    For i = 1 To NUM_HL
        For j = 1 To NUM_IL
            hl(i).wt(j) = (Rnd(1))
        Next j
    Next i
   
    'Output Layer
    For i = 1 To NUM_OL
        For j = 1 To NUM_HL
            ol(i).wt(j) = (Rnd(1))
        Next j
    Next i
End Sub

Sub TrainNet()
    Dim numCycles As Integer
    Dim current As Integer
    Dim j As Integer
    Dim sum As Double
   
    Input "How many training cycles", numCycles
    For i = 1 To numCycles
        j = 1
       
        current = Int(Rnd(1) * 10)
           
            For y = 1 To IMG_WIDTH
                For x = 1 To IMG_HEIGHT
                    curImg(j) = img(current, x, y)
                    j = j + 1
                Next x
            Next y
           
        DoNet
       
        'Calculate OL Error
        For j = 1 To NUM_OL
            If j = current Then
                ol(j).e = (1 - ol(j).act) * ol(j).act * (1 - ol(j).act)
            Else
                ol(j).e = (0 - ol(j).act) * ol(j).act * (1 - ol(j).act)
            End If
        Next j
       
        'Calculate HL Error
       
        For j = 1 To NUM_HL
            sum = 0
            For k = 1 To NUM_OL
                sum = sum + ol(k).e * ol(k).wt(j)
                hl(j).e = hl(j).act * (1 - hl(j).act) * sum
            Next k
        Next j
       
        'Adjust Weights
        For j = 1 To NUM_OL
            For k = 1 To NUM_HL
                ol(j).wt(k) = ol(j).wt(k) + LR * ol(j).e * hl(k).act
                ol(j).th = ol(j).th - LR * ol(j).e
            Next k
        Next j
       
        For j = 1 To NUM_HL
            For k = 1 To NUM_IL
                hl(j).wt(k) = hl(j).wt(k) + LR * hl(j).e * il(k).act
                hl(j).th = hl(j).th - LR * hl(j).e
            Next k
        Next j
       
        Print ".";
    Next i
End Sub

Sub RunNet()
    Dim quit As Byte
    Dim num As Byte
   
    Input "Which number do you want to test?", num
   
    For y = 1 To IMG_WIDTH
        For x = 1 To IMG_HEIGHT
            curImg(j) = img(num, x, y)
            j = j + 1
        Next x
    Next y
   
    DoNet
   
    For i = 1 To NUM_OL
        Print "Value of output node ";
        Print i - 1
        Print ol(i).act
    Next i
End Sub

Function sig(x As Double) As Double
    sig = 1/(1 + Exp(-x))
End Function

0:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

1:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,1,1,0,0,0,0,0
Data 0,0,0,0,1,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,1,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

2:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,1,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

3:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

4:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,0,1,1,0,0,0
Data 0,0,0,0,0,0,1,0,1,0,0,0
Data 0,0,0,0,0,1,0,0,1,0,0,0
Data 0,0,0,0,1,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,1,1,1,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

5:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,1,1,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,0,0,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,1,1,1,1,0,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

6:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,1,0,0,0,0,0,0,0,0,0
Data 0,0,1,0,0,0,0,0,0,0,0,0
Data 0,0,1,0,0,0,0,0,0,0,0,0
Data 0,0,1,0,1,1,1,1,0,0,0,0
Data 0,0,1,1,0,0,0,0,1,0,0,0
Data 0,0,1,0,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

7:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,1,0,0,0,0,0,0
Data 0,0,0,0,0,1,0,0,0,0,0,0
Data 0,0,0,0,1,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

8:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

9:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,1,1,0,0
Data 0,0,0,0,1,1,1,1,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0[/syntax]
Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
vspickelen
Member
*
Posts: 26



WWW
« Reply #4 on: December 29, 2005, 09:22:35 AM »

You started out well, Torahteen, but:
what's your program supposed to do in the 1st place, and what output do we have to expect?
It looks like backpropagation to me - quite ambitious to take a bite on,
or to debug, for that matter. ( that old SOM really is much easier Wink )

Some remarks though:

Sub InitNet is never called.

You're probably misusing the 'sum'-result in sub DoNet,
i'd put it like this:

Code:
  FOR i = 1 TO NUM_IL
        sum = 0
        FOR j = 1 TO IMG_TOTAL
            sum = sum + il(i).wt(j) * curImg(j)
        NEXT j
        il(i).act = sig(sum - il(i).th)
    NEXT i
etc. (Tip: reduce code by passing each layer as a subroutine argument.)

The size of the "input layer" (20) doesn't match your input (12 * 12)
In fact, the input layer reduces to activations only, you can strip
DIM SHARED il(1 TO NUM_IL) AS Node             'Input Layer
because three net layers are connected by two sets of weights only.
(144 -> 20 and 20 -> 10 in this case.)

This var is never used in sub RunNet:
DIM quit AS Byte
Though it might be important somehow.

I've compiled w/FB 016b, and This doesn't show:
INPUT "How many training cycles", numCycles
Though it should. A compiler bug, i suppose.
Quick patch:
print "How many training cycles";
INPUT ; numCycles

I'm sure you can work this out,
vspickelen
Logged
Torahteen
Ancient Guru
****
Posts: 744



« Reply #5 on: December 29, 2005, 01:02:52 PM »

Thanks for the reply. You're correct, it is a backprop NN. I am trying to recognize the digits from 0-9. The goal is to have 10 outputs, each representing one of the digits. If the answer is... say 7, then the output for that node should be a decent amount larger than the others. Make sense?

I forgot to call InitNet. No wonder my weights weren't getting set up.

I'm confused about this:

Quote
The size of the "input layer" (20) doesn't match your input (12 * 12)
In fact, the input layer reduces to activations only, you can strip
DIM SHARED il(1 TO NUM_IL) AS Node 'Input Layer
because three net layers are connected by two sets of weights only.
(144 -> 20 and 20 -> 10 in this case.)


Aren't I supposed to have 3 layers? Input, Hidden, and Ouput? I feed all of the information into input layer, which get's calculated using the Sigmoid activation function, and that becomes the weights for the Hidden layer, and so on. What exactly am I misunderstanding here?

It doesn't work in v0.16b eh? Hmm... I don't know why that is. Check the changes log to see if there is some new feature I don't know about.
Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
KiZ
__/--\__
*****
Posts: 2879


WWW
« Reply #6 on: December 29, 2005, 01:56:23 PM »

Quote from: "Torahteen"
It doesn't work in v0.16b eh? Hmm... I don't know why that is. Check the changes log to see if there is some new feature I don't know about.


No, it is a bug in the 0.16 cvs. Vic has found and addressed the problem, with the help of Antoni.
Logged
Torahteen
Ancient Guru
****
Posts: 744



« Reply #7 on: December 29, 2005, 02:46:13 PM »

Here is an example of my output:

Code:
First, see how the NN performs before training
Which number do you want to test? [color=red]3[/color]

Value of output node  0
 0.8583611546483856
Value of output node  1
 0.8843891112292291
Value of output node  2
 0.9123689633377022
Value of output node  3
 0.9446206548675346
Value of output node  4
 0.9155865835716671
Value of output node  5
 0.8699179837462892
Value of output node  6
 0.9314443557889023
Value of output node  7
 0.9060621891248365
Value of output node  8
 0.8857971181483882
Value of output node  9
 0.889199395400074
-----------------------------------------------
Now training network
How many training cycles [color=red]10000[/color]
-----------------------------------------------
Now see how the NN performs after training
Which number do you want to test? [color=red]3[/color]

Value of output node  0
 7.923202849710609e-002
Value of output node  1
 7.799303294393579e-002
Value of output node  2
 0.1093219822478929
Value of output node  3
 9.059998525862116e-002
Value of output node  4
 0.1094775032751099
Value of output node  5
 9.746875324353162e-002
Value of output node  6
 0.1129354063528408
Value of output node  7
 8.255943958120357e-002
Value of output node  8
 0.1100881485219571
Value of output node  9
 7.630521829981274e-003


Red = Input

Anyway, there are a few problems. As far as I know, the output should always be a number between 0 and 1 because of the sigmoid function. The sigmoid function looks fine to me, so what's the problem?

In the above situation, the output of node 3 should have been quite a bit higher (relatively speaking) than the others, signifying that 3 is what the NN chose.
Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
vspickelen
Member
*
Posts: 26



WWW
« Reply #8 on: December 29, 2005, 02:50:16 PM »

Quote from: "Torahteen"
Aren't I supposed to have 3 layers? Input, Hidden, and Ouput?

Compare it with this:

1 -> 2 -> 3

That's three layers, and two sets of weights (->) to connect them.

But the first layer is implied, so to say: it's your curimg() byte array, that's all you need there.

Now attach the connections between input and hidden layer to the latter,
Code:
DIM SHARED hl(1 TO NUM_HL) AS Node             'Hidden Layer
idem for the connections between hidden and output layer,
Code:
DIM SHARED ol(1 TO NUM_OL) AS Node             'Output Layer
that's only two shared Node-structures, and not three.

Your feedforward routine now becomes:
Code:
SUB DoNet()
    DIM sum AS single
   
    'Run through nodes in HL
    FOR i = 1 TO NUM_HL
        sum = 0
        FOR j = 1 TO IMG_TOTAL
            sum = sum + hl(i).wt(j) * curImg(j)
        NEXT j
        hl(i).act = sig(sum - hl(i).th)
    NEXT i
   
    'Run through nodes in OL
    FOR i = 1 TO NUM_OL
        sum = 0
        FOR j = 1 TO NUM_HL
            sum = sum + ol(i).wt(j) * hl(j).act
        NEXT j
        ol(i).act = sig(sum - ol(i).th)
    NEXT i
END SUB


I almost forgot:
Since your input is binary, you need a constant GAIN factor, like
Code:
  sig = 1/(1 + EXP(-GAIN * x))
to make the sigmoid steeper, else the output won't separate.
This gain also enters the derivatives of the sig-function.
I just got a stripped-down version of your program working fine with
GAIN = 5 and LR = 0.1

Giving away all changes will of course spoil your fun, so I won't.
vspickelen
Logged
Torahteen
Ancient Guru
****
Posts: 744



« Reply #9 on: December 29, 2005, 03:02:28 PM »

But how do I do BackPropagation for the hidden layer?
Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
vspickelen
Member
*
Posts: 26



WWW
« Reply #10 on: December 29, 2005, 03:29:23 PM »

Quote from: "Torahteen"
But how do I do BackPropagation for the hidden layer?


You already got that down, man, it's under 'Calculate HL Error
vspickelen (really likes this)
Logged
Torahteen
Ancient Guru
****
Posts: 744



« Reply #11 on: December 29, 2005, 03:45:29 PM »

Hmm... It's still not giving me correct results. Here is the code, followed by output.

[syntax="qbasic"]DEFINT A-Z

Declare Function sig(x As Double) As Double
Declare Sub DoNet()
Declare Sub TrainNet()
Declare Sub RunNet()
Declare Sub InitNet()

Const MAX_NODES = 20                    'Maximum number of nuerons in one layer
Const NUM_IL = MAX_NODES                'Number of Nodes in Input Layer
Const NUM_HL = MAX_NODES                'Number of Nodes in Hidden Layer
Const NUM_OL = 10
Const IMG_WIDTH = 12
Const IMG_HEIGHT = 12
Const IMG_TOTAL = IMG_WIDTH * IMG_HEIGHT
Const LR = 0.05
Const GAIN = 5

Type Node
    wt(1 To MAX_NODES) As Double        'Weights
    act As Double                       'Output of this Node
    th As Double                        'Threshold
    e As Double                         'Error
End Type

Dim Shared hl(1 To NUM_HL) As Node             'Hidden Layer
Dim Shared ol(1 To NUM_OL) As Node             'Output Layer
   
'Digit Images
Dim Shared img(0 To 9,1 To 12, 1 To 12) As Byte
Dim Shared curImg(1 To IMG_TOTAL) As Byte
'Begin Main
'Load the Images
For i = 0 To 9
    For y = 1 To 12
        For x = 1 To 12
            Read img(i,x,y)
        Next x
    Next y
Next i


InitNet
Print "First, see how the NN performs before training"
RunNet
Sleep
Cls
Print "Now training network"
TrainNet
Cls
Print "Now see how the NN performs after training"
RunNet
Sleep
End

SUB DoNet()
    DIM sum AS single
   
    'Run through nodes in HL
    FOR i = 1 TO NUM_HL
        sum = 0
        FOR j = 1 TO IMG_TOTAL
            sum = sum + hl(i).wt(j) * curImg(j)
        NEXT j
        hl(i).act = sig(sum - hl(i).th)
    NEXT i
   
    'Run through nodes in OL
    FOR i = 1 TO NUM_OL
        sum = 0
        FOR j = 1 TO NUM_HL
            sum = sum + ol(i).wt(j) * hl(j).act
        NEXT j
        ol(i).act = sig(sum - ol(i).th)
    NEXT i
END SUB

Sub InitNet()
    Randomize Timer
   
    'Hidden Layer
    For i = 1 To NUM_HL
        For j = 1 To NUM_IL
            hl(i).wt(j) = (Rnd(1)/4)
        Next j
    Next i
   
    'Output Layer
    For i = 1 To NUM_OL
        For j = 1 To NUM_HL
            ol(i).wt(j) = (Rnd(1)/4)
        Next j
    Next i
End Sub

Sub TrainNet()
    Dim numCycles As Integer
    Dim current As Integer
    Dim j As Integer
    Dim sum As Double
   
    Input "How many training cycles", numCycles
    For i = 1 To numCycles
        j = 1
       
        current = Int(Rnd(1) * 10)
           
            For y = 1 To IMG_WIDTH
                For x = 1 To IMG_HEIGHT
                    curImg(j) = img(current, x, y)
                    j = j + 1
                Next x
            Next y
           
        DoNet
       
        'Calculate OL Error
        For j = 1 To NUM_OL
            If j = current Then
                ol(j).e = (1 - ol(j).act) * ol(j).act * (1 - ol(j).act)
            Else
                ol(j).e = (0 - ol(j).act) * ol(j).act * (1 - ol(j).act)
            End If
        Next j
       
        'Calculate HL Error
       
        For j = 1 To NUM_HL
            sum = 0
            For k = 1 To NUM_OL
                sum = sum + ol(k).e * ol(k).wt(j)
                hl(j).e = hl(j).act * (1 - hl(j).act) * sum
            Next k
        Next j
       
        'Adjust Weights
        For j = 1 To NUM_OL
            For k = 1 To NUM_HL
                ol(j).wt(k) = ol(j).wt(k) + LR * ol(j).e * hl(k).act
                ol(j).th = ol(j).th - LR * ol(j).e
            Next k
        Next j
       
        For j = 1 To NUM_HL
            For k = 1 To IMG_TOTAL
                hl(j).wt(k) = hl(j).wt(k) + LR * hl(j).e * curImg(k)
                hl(j).th = hl(j).th - LR * hl(j).e
            Next k
        Next j
       
        Print ".";
    Next i
End Sub

Sub RunNet()
    Dim quit As Byte
    Dim num As Byte
   
    Input "Which number do you want to test?", num
   
    For y = 1 To IMG_WIDTH
        For x = 1 To IMG_HEIGHT
            curImg(j) = img(num, x, y)
            j = j + 1
        Next x
    Next y
   
    Print "You chose this number"
    For y = 1 To IMG_WIDTH
        For x = 1 To IMG_HEIGHT
            If img(num, x, y) = 0 Then
                Pset(x + 100,y),15
            End If
        Next x
    Next y
               
    DoNet
   
    For i = 1 To NUM_OL
        Print "Value of output node ";
        Print i - 1
        Print ol(i).act
    Next i
End Sub

Function sig(x As Double) As Double
    sig = 1/(1 + Exp(-GAIN * x))
End Function

0:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

1:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,1,1,0,0,0,0,0
Data 0,0,0,0,1,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,1,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

2:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,1,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,1,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

3:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

4:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,0,1,1,0,0,0
Data 0,0,0,0,0,0,1,0,1,0,0,0
Data 0,0,0,0,0,1,0,0,1,0,0,0
Data 0,0,0,0,1,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,1,1,1,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

5:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,1,1,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,0,0,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,1,1,1,1,0,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

6:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,0,0,0,0
Data 0,0,1,0,0,0,0,0,0,0,0,0
Data 0,0,1,0,0,0,0,0,0,0,0,0
Data 0,0,1,0,0,0,0,0,0,0,0,0
Data 0,0,1,0,1,1,1,1,0,0,0,0
Data 0,0,1,1,0,0,0,0,1,0,0,0
Data 0,0,1,0,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

7:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,1,1,1,1,1,1,1,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,0,1,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,0,1,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,0,1,0,0,0,0,0
Data 0,0,0,0,0,1,0,0,0,0,0,0
Data 0,0,0,0,0,1,0,0,0,0,0,0
Data 0,0,0,0,1,0,0,0,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

8:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0

9:
Data 0,0,0,0,0,0,0,0,0,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,1,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,1,1,0,0
Data 0,0,0,0,1,1,1,1,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,0,0,0,0,0,0,1,0,0
Data 0,0,0,1,0,0,0,0,1,0,0,0
Data 0,0,0,0,1,1,1,1,0,0,0,0
Data 0,0,0,0,0,0,0,0,0,0,0,0[/syntax]

Code:
First, see how the NN performs before training
Which number do you want to test? 2
You chose this number
Value of output node  0
 0.9999946772066498
Value of output node  1
 0.9998794164495293
Value of output node  2
 0.9999991414321088
Value of output node  3
 0.9999969437467565
Value of output node  4
 0.999989835095912
Value of output node  5
 0.999954813071024
Value of output node  6
 0.9999966488577209
Value of output node  7
 0.999993763156863
Value of output node  8
 0.9999949938112089
Value of output node  9
 0.9999959078217268
------------------------------

Now training network
How many training cycles 5000
--------------------------------

Now see how the NN performs after training
Which number do you want to test? 5
You chose this number
Value of output node  0
 0.9999930039506503
Value of output node  1
 4.505908216785114e-002
Value of output node  2
 0.9999991069143672
Value of output node  3
 0.9999964581486549
Value of output node  4
 0.9999813061324169
Value of output node  5
 7.279828978456202e-002
Value of output node  6
 0.9999960467329503
Value of output node  7
 0.9999913546369986
Value of output node  8
 0.999993535158855
Value of output node  9
 0.9999948572862908


Any idea why it is still giving results above 1?
Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
Deleter
Na_th_an
*****
Posts: 1292



WWW
« Reply #12 on: December 29, 2005, 06:14:21 PM »

none of those results are above 1  :Huh:
Logged

vspickelen
Member
*
Posts: 26



WWW
« Reply #13 on: December 29, 2005, 07:26:32 PM »

O.k, last post for today

Sad to say, your code needs more tidying up still.
Things that will be quite obvious, once you understand
what's goin' on in all those lil'nifty-nested for-loops.

The constants are messed up, this will do
Code:
CONST IMG_WIDTH = 12
CONST IMG_HEIGHT = 12
CONST NUM_IL = IMG_WIDTH * IMG_HEIGHT    'Number of Nodes in Input Layer
CONST NUM_HL = 20                        'Number of Nodes in Hidden Layer
CONST NUM_OL = 10                        'Number of Nodes in Output Layer
CONST LR = 0.1
CONST GAIN = 5

Replace MAX_NODES and IMG_TOTAL accordingly.


Use
Code:
hl(i).wt(j) = (RND - .5) * .1
for small zero mean random weights

While this is fine
Code:
current = INT(RND * NUM_OL)
it can still go wrong
Code:
FOR j = 1 TO NUM_OL
   IF j = current THEN


I've changed the signs before the bias terms here,
guess you can justify the displaced outer-loop exprssions yourself
 
Code:
     'Calculate HL Error
       
        FOR j = 1 TO NUM_HL
            sum = 0
            FOR k = 1 TO NUM_OL
                sum = sum + ol(k).e * ol(k).wt(j)
                hl(j).e = hl(j).act * (1 - hl(j).act) * sum
            NEXT k
        NEXT j
       
        'Adjust Weights
        FOR j = 1 TO NUM_OL
            FOR k = 1 TO NUM_HL
                ol(j).wt(k) = ol(j).wt(k) + LR * ol(j).e * hl(k).act
                ol(j).th = ol(j).th + LR * ol(j).e
            NEXT k
        NEXT j
       
        FOR j = 1 TO NUM_HL
            FOR k = 1 TO IMG_TOTAL
                hl(j).wt(k) = hl(j).wt(k) + LR * hl(j).e * curImg(k)
                hl(j).th = hl(j).th + LR * hl(j).e
            NEXT k
        NEXT j
 


This innocent one is a classic:
 
Code:
  INPUT "Which number do you want to test?", num

    FOR y = 1 TO IMG_WIDTH
        FOR x = 1 TO IMG_HEIGHT
            curImg(j) = img(num, x, y)
            j = j + 1
        NEXT x
    NEXT y

might drive you nuts.


The only thing left then, is to remember that
Quote from: "vspickelen"
This GAIN also enters the derivatives of the sig-function.


Good night  ( snow here )
Logged
Torahteen
Ancient Guru
****
Posts: 744



« Reply #14 on: December 30, 2005, 01:29:43 AM »

Hmm... Maybe I'm just a bad programmer. Do you have a working version of my code? Is it actually presenting the correct results? Could you post the code? I view this as a learning experience, so don't worry about spoiling my fun Wink.
Logged

quote="Deleter"]judging gameplay, you can adaquately compare quake 4 with pong[/quote]
Pages: [1] 2
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.21 | SMF © 2015, Simple Machines Valid XHTML 1.0! Valid CSS!