NN cannot determine the correct output bias

Training a feedforward neural net with one hidden layer leads always to an output value that is shifted for a certain amount. Increasing the number of iterations or the size of the testdataset does not improve the the result.

These are some of the parameters I used to build/train the NN.

title	Bias
class	class vsoc.training.BatchSizeTraining$
learningRate	1.0E-04
trainingData	playerpos_x A 500000
batchSizeTrainingDataRelative	0.10
testData	playerpos_x A 1000, playerpos_x B 1000, playerpos_x C 1000, playerpos_x D 1000, playerpos_x E 1000
iterations	200
optAlgo	STOCHASTIC_GRADIENT_DESCENT
numHiddenNodes	100, 300, 500
regularisation	None
seed	-751785241836862251

 
Testing the trained NNs against different testdatasets leads to the following result.



The output value is always shifted for a certain amount no matter what testdataset is used.

Open Question

Is there a way to determine the correct output bias during training the NN ?

Manual Bias

Parameters

title	Manual Bias
class	class vsoc.training.BatchSizeTraining$
learningRate	1.0E-04
trainingData	playerpos_x A 500000
batchSizeTrainingDataRelative	0.10
testData	playerpos_x B 1000
iterations	200
optAlgo	STOCHASTIC_GRADIENT_DESCENT
numHiddenNodes	100, 200, 300, 500
regularisation	None
seed	-8876029723576768028, -3375934931721753594, 1696412033065026326, 6283369899114212938, -3587145959788761006, -49633614034392359, 839473069744556995, 2684180910583920968, -7386426029039824287, 3837550597042215702, -4997332816031131463, 3815956045927450641, -3403545420327942156, 3661589999326108919, 2455182172019959569, 3162067559174869087, -1669667726484573068, -397364268861304359, -7913912163456363611, 7909811908685690708, 2311254609887171408, 8435778613472697536, 504319445395541907, -433047265908778508, -5423116490601089927, 8381378201148666576, -6904400535536458560, -4437293041802599002, 5946303504748415202, 7150743982293367765, 2242790324642662994, -3814375723625398089

Manually adopting the output bias leads to the following result.



Manually added bias values

A_100	-1.1
A_200	-1.35
A_300	-1.5
A_500	1.5
B_100	-0.8
B_200	1.7
B_300	1.6
B_500	1.6
C_100	-1.0
C_200	-1.5
C_300	1.4
C_500	1.6

Conclusion

Manually adopting the output bias leads to better results

Open question

Why does the learning algorithm not minimize the output bias ?

Bias Regularisation

title	Bias Regularisation
class	class vsoc.training.RegularisationTraining$
learningRate	1.0E-04
trainingData	playerpos_x A 500000
batchSizeTrainingDataRelative	0.10
testData	playerpos_x B 1000
iterations	200
optAlgo	STOCHASTIC_GRADIENT_DESCENT
numHiddenNodes	50, 100, 200
regularisation	Some(Regularisation(0.0,0.0,2.0,2.0)), Some(Regularisation(0.0,0.0,1.0,1.0)), Some(Regularisation(0.0,0.0,0.001,0.001)), Some(Regularisation(0.0,0.0,1.0E-5,1.0E-5)), Some(Regularisation(0.0,0.0,0.0,0.0))
seed	3915220284829678

Conclusion

Expected that bias regularisation could fix the wrong bias for nets with more than 100 intermediate nodes, but there was no effect.
Taking 100 intermediate nodes leads so far to the best results.

Further Work

Find a way to learn the correct bias.

Number of hidden nodes

title	Number of hidden nodes
class	class vsoc.training.NumHiddenNodesTraining$
learningRate	1.0E-04
trainingData	playerpos_x A 500000
batchSizeTrainingDataRelative	0.10
testData	playerpos_x B 1000
iterations	200
optAlgo	STOCHASTIC_GRADIENT_DESCENT
numHiddenNodes	50, 100, 200, 500, 700
seed	4278888856359104810, 4039877965451516678, -4611576926702079432, -1188411120859739005

Conclusion

Variance gets smaller until 500 hidden nodes.
There is quite a big bios on the error that gets worse the more hidden layers are used.

Work to do

Try out bios Regularisation.

Regularisation l1 l2

title	Regularisation l1 l2
class	class vsoc.training.SizeIterationsTraining$
learningRate	1.0E-04
trainingData	playerpos_x A 500000
batchSizeTrainingDataRelative	0.10
testData	playerpos_x B 1000
iterations	200
optAlgo	STOCHASTIC_GRADIENT_DESCENT
seed	4228810264686955519

Conclusion

l1, l2 Regularisation seems to have no effect

multiple iterations for different sized datasets

Same data with different scaling of the y-axis.

title	multiple iterations for different sized datasets
class	class vsoc.training.SizeIterationsTraining$
learningRate	1.0E-04
trainingData	playerpos_x A 100000, playerpos_x A 500000, playerpos_x A 1000000, playerpos_x A 5000000
batchSizeTrainingDataRelative	0.10
testData	playerpos_x B 5000
iterations	50, 100, 200, 500
optAlgo	STOCHASTIC_GRADIENT_DESCENT
seed	6779276617664099510

Conclusion
The difference of the error variation between a dataset of 10k and 5M is not very big.
A Dataset of 10k with 500 iterations seems to be sufficient for reasonable results.

Test Optimzation Algorithm | iterations: 300

CG	CONJUGATE_GRADIENT
LB	LBFGS
LG	LINE_GRADIENT_DESCENT
SG	STOCHASTIC_GRADIENT_DESCENT

title	Test Optimzation Algorithm | iterations: 300
class	class vsoc.training.LearningRateIterationsTraining$
learningRate	1.0E-04
trainingData	playerpos_x A 50000, playerpos_x A 100000, playerpos_x A 500000, playerpos_x A 1000000
batchSizeTrainingDataRelative	0.10
testData	playerpos_x B 1000
iterations	300
optAlgo	CONJUGATE_GRADIENT, LBFGS, LINE_GRADIENT_DESCENT, STOCHASTIC_GRADIENT_DESCENT
seed	8726567394090864187

Conclusion

Stochastic gradient descent seems to be the best optimisation. Eventually the other algorithms perform better with other meta parameters.