Artificial Neural Networks using the Backpropagation algorithm is a traditional method for classification and forecasting. Though several versions of Deep Neural Networks (DNN) are now popular powerful tools for analysis, still the backbone behind all architectures of Neural Networks continue to be the gradient descent method used in Feedforward and Backpropagation algorithms. Both ANN and DNN have a wide range of important commercial applications. There have been numerous efforts to design artificial neural networks based on Von Neumann’s architecture, trying to produce intelligent programs that mimic biological neural network. Neurons are very special cells in the human brain, interconnected with each other and responding to stimuli using chemical and electric reactions with connections called synapses. The idea of ANN is to simulate neurons stimuli process and let this neurons to learn by themselves.

ANN can perform complex classification problems. For a simple binary classification, the idea is to construct a decision function h x to approximate the outcome or label y (which is binary, either zero or one). The decision can be interpreted as a simple weighted function, a linear combination of, let’s say, two features x 1 and x 2 :

h ( x ; θ , b ) = θ 1 x 1 + θ 2 x 2 + b (1)

Which can also be written in the form:

y = h ( x ; θ , b ) = θ T x + b (2)

The main objective is to find the vector of weights θ and the parameter b (bias) that may be used to describe and predict the outcome y . A Neural process is a biological process that describes how a neural cell learns. A neural cell processes information from stimuli it receives and then uses a series of synapses to pass on new information already processed. An ANN tries to reproduce the same process, using inputs nodes to receive information, one hidden layer to process the information using an activation function, and then output nodes where the processed information is received. Given an output y we must train our network to learn and obtain these output values. At the end, the weights θ will be obtained and will be used for prediction.

The process of training a ANN will depend on the activation function we want to use as well as the method to find the appropriate weights recursively. Usually, we may initiate to train the ANN with random input values and then apply weights to every data point that will pass on information to a hidden layer where the information will be processed by an activation function. Weights θ T are recalculated iteratively until convergence is achieved. It is desirable that the values we get from the decision function h x ; θ , b being in a small range, for example, between 0 and 1. One way to achieve this is to map the decision function h x ; θ , b into a new function what will give an output between 0 and 1:

h ( x ; θ , b ) = g ( θ T x + b ) (3)

To represent g as a new function of z x = θ T x + b , we may start to use a sigmoid function of the type (although there are other types of functions that may be useful to represent g ):

g ( z ) = 1 1 + e - z (4)

This is akin to a logistic regression function with g z ∈ 0,1 . We call this activation function because this function will process the information and put forward an output. This activation function will be in every neuron, in every hidden layer of the neural network.

The decision function will approximate the label h x i ; θ , b ≈ y i for every data point i = 1,..., n . The idea is to use all the past data to learn the values θ , b and to approximate the values of y (supervised learning). We may try to minimize the sum of square errors (learning rule) as our objective function or loss function as follows:

H ( θ , b ) = ∑ i = 1 n ‍ ( h ( x ( i ) ; θ , b ) - y ( i ) ) 2 (5)

The crucial step is to minimize the error function ( h x i ; θ , b − y i ) 2 . The main task will be to obtain the parameters θ and b in such way that the error is minimized. One optimization technique is to use iteration in order to approach to the optimum values of θ and b . One optimization method is called the stochastic gradient descent algorithm:

θ 1 = θ 1 + α Δ θ 1 (6)

θ 2 = θ 2 + α Δ θ 2 (7)

b = b + α Δ b (8)

Where the Δ θ i and Δ b are the gradients of the parameters. The important in this method is to find the gradients in Δ θ i and Δ b in order to iterate until convergence. The term stochastic comes from the idea of initializing the algorithm using random weights, usually between zero and one. The idea to start with random weights during iteration in order to avoid local minima. Another problem in this method is the learning rate determination of the network α . If α is too small the algorithm is very slow and if too large can bring about large fluctuation. Additional analysis is needed to find α , for example using Cross-Validation analysis or optimization.

By computing partial derivatives of the error function with respect to the parameters, the gradients become:

Δ θ 1 = 2 g ( θ T x + b ) - y i 1 - g ( θ T x + b ) g ( θ T x + b ) x 1 i (9)

Δ θ 2 = 2 g ( θ T x + b ) - y i 1 - g ( θ T x + b ) g ( θ T x + b ) x 2 i (10)

Δ b = 2 g ( θ T x + b ) - y i 1 - g ( θ T x + b ) g ( θ T x + b ) (11)

The stochastic gradient descent algorithm allows to learn the decision function h x ; θ , b computing the above gradients by iteration. First we must set a random value for θ and b and use the initial values x i and y i to compute the derivatives 9, 10 and 11. Once the parameters θ 1 , θ 2 and b have been computed, we use these values in 6, 7 and 8 and then go back to the input values x i and y i again. This iteration continues until convergence, when the decision function h x * ; θ * , b * is obtained and used for prediction. To predict y we use the parameters θ * multiplied by x plus the term b * after been put through the sigmoid function. In this simple network the inputs represent the first layer l and then a single neuron (perceptron) in the next layer l + 1 where the activation function processes the information and pass it over to the final output which is in the final layer L .

The gradient descent algorithm is a key feature of an ANN. Although more sophisticated algorithms are being developed, still gradient descent algorithm is still the core method in ANN. There is also the disadvantage of vanishing gradient when weights are too small and make the gradient to go to zero. Perhaps the vanishing gradient problems was the main disadvantage of the ANN and also the main motivation to develop more sophisticated networks.

Another idea is to separate the data into smaller problems and to solve for each problem separately. For example, in our binary classification problem, some data with a label equal to zero will be a single cluster between two separated clusters of data labelled one. Now we will need two decision functions with more parameters and we need to construct a neural network with two neurons. The idea is to make a decision function of decision functions so that to predict the label h h 1 x , h 2 x ; ω , c ≈ y . We can use the previous gradient descent method as before, but now we have a more complex structure.

What we are building now is a neural network where the hidden layer that store the activation function g z has two neurons. Both inputs now go through the whole network and the parameters ω and θ are now the weights for the decision functions while the biases are now b and c . The new decision function is:

h ( x ) = g ( ω 1 h 1 ( x ) + ω 2 h 2 ( x ) + c ) = g ( ω 1 g ( θ 1 x 1 + θ 2 x 2 + b 1 ) + ω 2 g ( θ 3 x 1 + θ 4 x 2 + b 2 ) + c ) (12)

In the above decision function all parameters and biases must be found at the same time using gradient descent. We are iterating forward, which means that iteration to update parameters must go back to each input data point in the training sample x i , y i in layer l until convergence is accomplished. This iteration process is called feedforward Neural Networks.

Another way to learn is to use the backpropagation algorithm (BP). But before we may consider the possibility of more complex ANN architectures. We may consider adding more neurons but also additional hidden layers to our Neural Networks, on what is commonly known as Multilayer Perceptron Network. BP requires that once the feedforward process has been completed and we have arrived to the output layer L , then we go back on the entire network to perform a backward pass for all layers l = L − 1, L − 2,...,2 . In order to perform a backward pass we must redefine our variable z and the decision function. The function z is now:

z ( l ) = θ ( l ) T h ( l - 1 ) + b ( l ) (13)

And the decision function becomes:

h ( l ) = g ( z ( l ) ) = g ( θ ( l ) T h ( l - 1 ) + b ( l ) ) (14)

The first decision function is just the layer of inputs h 1 = x , which are used to update the next decision function h 2 = g θ 1 T h 1 + b 1 and so forth until we get the output h x = h L = g θ L − 1 T h L − 1 + b L − 1 which is a scalar vector. Because we are now using multiple neurons, we can understand h l as a vector as well as the input vector x and the parameters θ and b . We must proceed in a similar way we obtained the gradients in 9, 10 and 11. First we must find the derivative of the function g z respect to the parameters θ and b . For the vector of weights:

∂ g ( z ( l ) ) ∂ θ ( l ) = ∂ g ( z ( l ) ) ∂ z ( l ) ∂ z ( l ) ∂ θ ( l ) (15)

And for the bias:

∂ g ( z ( l ) ) ∂ b ( l ) = ∂ g ( z ( l ) ) ∂ z ( l ) ∂ z ( l ) ∂ b ( l ) (16)

The second part of the above derivatives, ∂ z l / ∂ θ l and ∂ z l / ∂ b l comes from 13:

∂ z ( l ) ∂ θ ( l ) = h ( l - 1 ) (17)

∂ z ( l ) ∂ b ( l ) = I (18)

To find the first part of the above derivatives 15 and 16, we define δ l = ∂ g z l / ∂ z l . Because z l + 1 = θ l + 1 T h l + b l + 1 , then:

∂ z ( l + 1 ) ∂ h ( l ) = θ ( l + 1 ) T (19)

and since h l = g z l , then:

∂ h ( l ) ∂ z ( l ) = ∂ g ( z ( l ) ) ∂ z ( l ) = g ' ( z ( l ) ) (20)

Now we can obtain the derivative δ l :

δ ( l ) = ∂ h ( l ) ∂ z ( l ) ∂ z ( l + 1 ) ∂ h ( l ) ∂ g ( z ( l ) ) ∂ z ( l + 1 ) = θ ( l + 1 ) T δ ( l + 1 ) ⊙ g ' ( z ( l ) ) (21)

Where the ⊙ is for the Hadamard product. Now we can find the updates for the gradient descent and perform the backpropagation:

Δ θ ( l ) = ∂ g ( z ( l ) ) ∂ θ ( l ) = δ ( l ) h ( l - 1 ) T (22)

Δ b ( l ) = ∂ g ( z ( l ) ) ∂ b ( l ) = δ ( l ) (23)

The backpropagation algorithm allows the network to learn and get the parameters θ and b in a more refined way. The main drawback of both feedforward and backpropagation is that, some gradients may become zeros and then some neurons may deactivate themselves. In rare cases the data and initial values produce the deactivation of neurons weakening the network and decreasing the predictive power, what is know as vanishing gradient problem. This is the main motivation to develop more complex architectures called Deep Neural Networks or DNN.

Another algorithm commonly used in ANN is the heuristic Resilient Backpropagation algorithm (RBP). This algorithm is a slightly different version of the Backpropagation but instead of using the magnitude of the gradients Δ θ 1 , Δ θ 2 and Δ b we use their sign. The purpose of this modification, first proposed by Riedmiller and Braun [14] in 1993 and applied by Anastasiadis et al. [2] in 2005, was to adapt the learning rate over the entire neural network. The RBP converges faster than the common BP algorithm but keeps its more general attributes. Gunther and Fritsch [5] produced an efficient algorithm to estimate the RBP with weight backtraking.

With the Backpropagation algorithm we have seen that the weights are updated following the general form:

θ i j ( l + 1 ) = θ i j l + α Δ θ i j ( l )

Where θ i j l means the weight from neuron j to neuron i in layer l . In the previous section we found that the updated amount Δ θ i j l was Δ θ l = δ l h l − 1 (we removed the transpose upper letter T to facilitate explanation). We know that this is also:

Δ θ i j ( l ) = ∂ z ( l ) ∂ θ i j ( l ) δ ( l )

The RBP algorithm proposes that the update is performed with the sign of the derivative rather than the size of it as follows:

Δ θ i j ( l ) = - s i g n ∂ z ( l ) ∂ θ i j ( l ) δ i j ( l ) (24)

Another importance change is that the update parameter δ l is changed to δ i j l in the following form:

δ i j ( l ) = m i n ( η + ⋅ δ i j ( l - 1 ) , δ m a x )   , i f   ∂ z ( l - 1 ) ∂ θ i j ( l - 1 ) ⋅ ∂ z ( l ) ∂ θ i j ( l )   > 0 m a x ( η - ⋅ δ i j ( l - 1 ) , δ m i n )   , i f   ∂ z ( l - 1 ) ∂ θ i j ( l - 1 ) ⋅ ∂ z ( l ) ∂ θ i j ( l )   < 0 δ i j ( l ) o t h e r w i s e

Where 0 < η − < 1 < η + are pre-set values. The value of δ i j l is updated in every step according with the change of sign of the derivative in 24 and the above definition. When the sign changes this means that a minimum has been missed so the network applies the either η − ⋅ δ i j l − 1 or δ m i n which ever larger. The reader may notice that this process also eliminates the need of determining exogenously the learning rate α as in the traditional backpropagation algorithm, because the parameter η is a close adaptative substitute.

The methods of weight backtraking is also based on heuristics, and the idea is to keep using previous weights for updating (some weights only). For example, if:

∂ z ( l - 1 ) ∂ θ i j ( l - 1 ) ⋅ ∂ z ( l ) ∂ θ i j ( l )   ≥ 0 ,   t h e n   Δ θ i j ( l ) = - s i g n ∂ z ( l ) ∂ θ i j ( l ) δ i j ( l )

But if less than zero, we use the previous update:

∂ z ( l - 1 ) ∂ θ i j ( l - 1 ) ⋅ ∂ z ( l ) ∂ θ i j ( l )   < 0 ,   t h e n   Δ θ i j ( l ) = Δ θ i j ( l - 1 )

This implementation trick avoid the updating of the learning rate then avoiding using the otherwise option above. The advantages of using RBP algorithm is that reduces computation with the advantage of similar, if not better, precision. It is very useful when the data contains noise, which means that it preforms well when applied to financial time series data sets. In the next section we will present some results using RBP neural networks.

The first task in this work is to forecast a time series using binary classification with ANN methods. A basic classification would be to describe the behaviour of a stock or stock index in order to predict its movement. Predicting stock prices is important as we would want to decide if we need to buy or sell a stock or predict the ups and downs of a price index. In this case we would want to define a label y with binary values 0,1 with zero for a drop in the stock prices compared to a day before, and one for an increase in the stock prices. We can use for example, the closing prices of stocks or stock indexes to construct this label. This is a practical way to observe movements in the stock prices, as we need to know if prices will go up or down for very practical decision making. For example, we may decide to sell if a stock price is likely to go down or buy when price is likely to go up. Broadly speaking, trading in the stock market is based in these simple decisions, and usually a good trader uses, among other tools, technical analysis to decide whether is time to sell or buy stock. Although we are not applying this binary classification problem to any specific stock, the underlying principle of classification remains the same.

The next question is defining the features that will be used to predict the movements in stock prices. In other words, we need the matrix of features x that will help to define the label y. We decided to use some technical analysis concepts as in Kim [11], most of them taken from Achelis [1]. Technical analysis indicators will be our features matrix x. An experienced trader may read the concepts in table 1 and along with additional information then try to predict changes in stock prices. These features are mostly ratios of prices, moving averages or both.

Table 1

Table 1 shows twelve well known technical indicators for trading. These are constructed with simple market data such as closing price (CP), lowest low price (LL), highest high price (HH), high (H) and low (L) prices during the trading day or period and it is very common to use Moving Averages (MA) for their construction. The stochastic oscillators such as stochastic % K and % D are used by traders to know if an stock is overbought or oversold. The Momentum is used to see any change on the price trend while the Rate of Change (ROC) measures the speed of the ascent/descent of every new trend. The Williams’ Accumulation/Distribution indicator measures the market pressure, which advices to sell when Williams’ A/D is low and stock prices are high and buy when Williams’ A/D is high and market prices are low. The Disparity5 is just an ratio of the closing price with respect to the 5 days Moving Average and the OSCP or Price Oscillator is the growth of the moving average. The Commodity Channel Index (CCI) is used as a leading indicator to observe the strength of a stock, whether a stock is overbought or oversold. The Relative Strength Index (RSI) is used as a leading indicator to observe the historical strength of a stock using the ups and downs in historical closing prices. The WilliamsR is a momentum indicator that reflects how the close price compare with the highest price.

All features in table 1 are associated with the prices of the stocks and are used to interpret the trends of stock market prices. The entire data set for a given stock market index will be the label y and the feature matrix x that describes the label. All technical analysis indicators will be used to classify our label in both directions, ups and downs for the entire stock market index. There are dozens more technical analysis indicators that can be used, but we are trying to use some the most popular and also applied in other similar research.

This section contains an empirical analysis using RBP algorithm in order to predict time series, particularly changes in stock price indexes. We chose to predict changes in six major European, Asian and North American stock market indexes. We used six stock indexes: The European STOXX50 that contains blue chip stock from the 50 best performing companies in leading sectors in Europe; the DAX which is also an index that contains 30 blue chip German companies; the Nikkei stock exchange index, the Hang Seng index which is the stock exchange index from Hong Kong, the Canadian Toronto Stock Exchange index and the Mexican Stock Exchange index IPC.

We decided to use daily data for each stock exchange from January 2000 to June 2019, less than five thousands daily observation in each market. Compared with the same data from 1980’s and 1990’s, the period of analysis is high frequency data and contains sharp financial crashes, perhaps due to the new trading methods using electronic platforms and the availability of information online. Financial markers are now more competitive as communication technology has improved along with capital mobility. Table 5 in the appendix contains the summary statistics for the six markets on closing, high, low and open market prices.

With the information on the average prices in each market, we first constructed our matrix of features x that were used to describe the label y, where y are the changes in close price. The idea is to approach to the daily changes in close prices in each stocks market. In this case, the classification problem will be to assign a label of 1 if the close price index in time C P t is higher than the previous day C P t − 1 , and 0 otherwise so that y → 0,1 . Our data points x will be the features that help us to find the weights w and the bias b .

Because ANN is a supervised machine learning method, we are going to demand to the network to find the best way to predict y using the twelve technical analysis features constructed using indicators in table 1 (matrix x ). This matrix will serve as the input data and the main objective of the ANN algorithm is to find patterns in x so that to approach the vector y. Because the weights produced by the ANN are sensitive to the scale of each feature we must normalize the features matrix x and transform them into a matrix with values in the range of 0,1 to ensure large values do not overcome features with small values.

When the whole data set with y and x have been constructed, the next step is to divide the data set into a training and test sets. The training set will be used by each algorithm to learn and the second set for testing if predicted values match with the data points. In our analysis, we used the first 70% data points as training set and the remaining as test data set. In the appendix the reader will find the summary statistics for each feature x and the label y by stock market.

The only thing left for clarification will be the estimation of the Hit ratio for each prediction. After running each of the ANN models, we will get the predicted values using the test data into a new data set with predicted values for the label y^. One way to evaluate the performance of the ANN predictions, is to construct a hit ratio. Because the predicted values are real numbers and our test data set has a binary label, we must transform the predicted values into binary outcomes. We set up a threshold of 0.5 which mean that any predicted values higher than 0.5 will be set up to 1 (close price was higher than in previous day) if not equal to zero (close price was lower than in previous day). The prediction results are matched in the form:

R i = 1         i f   P r e d i c t e d   v a l u e = t e s t   d a t a 0         o t h e r w i s e (25)

The prediction performance is measured using a hit ratio, defined by:

H R = 1 m ∑ i = 1 m ‍ R i (26)

This hit ratio is the percentage of correct matches where y = y^. This is a simple coefficient that can be used to compare performance (prediction) for each neural network model.

The main part of the empirical analysis requires to use ANN to predict time series. We trained different single layer networks using traditional Backpropagation and Resilient Backpropagation Neural Network algorithm. At first, single layer neural networks were constructed with 6, 12, 18 and 24 neurons each using standard logistic and error functions. Later we trained multi-layer networks with 6 and 12 neurons in three hidden layers. The results are shown in table 2.

Table 2

One disadvantage of the ANN is the cost in training increases when the architecture becomes more complex. As the number of neurons and hidden layers increase, the longer the training time is required. On the other hand, ANN with backpropagation may obtain better performance due to a more flexible updating. Table 2 contains the hit ratios for different ANN models with different architectures. The upper part contains the hit ratios using traditional backpropagation with a learning rate of 0.1 while the lower part contains the hit ratios using resilient backpropagation with weight backtracking.

With the only exception being the Europe50 index, we find larger hit ratios using resilient backpropagation. This does not mean that we cannot achieve better results in traditional backpropagation, but for that we need to find the best learning rate and architecture. And this will require additional statistical analysis in order to decide the correct learning rate, as each model is different.

On the other hand, resilient backpropagation has a flexible and heuristic way to choose the learning rate and update the gradient for a better descent. The reader may notice that there is little room for improvement in each model using backpropagation as we use a single learning rate for every model. However, resilient backpropagation has room for improvement as the learning is controlled during convergence. Estimation in table 2 will vary as long as we choose different activation functions, learning rates and gradient methods for updating, but we decided to leave model selection for future research.