This tutorial provides a conceptual framework and practical insights to work in the Machine Learning field using python programming language. The content of the tutorial combines theoretical concepts with programming examples about how to use these algorithms through the Scikit learn library from Python. All the examples are related to the application of machine learning in the finance domain.



We give readers conceptual tools to understand the algorithms and the workflow of a machine learning project. In order to work on a machine learning project, there are a lot of steps and actions that we should take care to guarantee good results of the models.



Some of these steps are the quality and quantity of the dataset, the right balance between the bias and variance of the model to avoid overfitting or under fitting, the correct selection of the features that will be used to train the model as well as the creation of new features with the original data.



The tutorial starts describing the first steps on a machine learning projects such as data preprocessing and the exploration of the features of the dataset to get a sense about the&nbsp; relationship between them and the variable that they aim to predict.&nbsp;



Then we explain the process of training and testing a dataset for Supervised learning problems as well as the&nbsp; importance of the definition and interpretation of the loss function when we evaluate the ability of a certain model. Afterwards we explain some issues in the training step such as the bias-variance-trade-off and the need of good balance between these elements.



Once we have pointed out the main concepts of the workflow of a machine learning project, the subsequent sections describes popular algorithms from Supervised Learning as well as useful techniques to prevent overfitting issues such as the K-Fold Cross Validation technique.



In addition to the theoretical explanation of the algorithms, the tutorial provides interesting examples of Supervised learning models such as&nbsp; Multiple Linear Regression and Classification problems. The subsequent section describes the Unsupervised Learning field with a complete example of cluster analysis.&nbsp;&nbsp;



Finally we explain basic concepts of Neural Networks and their structure.&nbsp; This is an advanced technique that is suitable for financial applications as it&nbsp; is capable to understand complex relationships among the data and work with different distributions of&nbsp; financial time series.&nbsp;

Machine Learning with Python

Machine Learning is the field which applies statistical analysis and computer science for employing algorithms that learn how to perform tasks such as prediction or classification of a target variable as well as grouping data. These algorithms learn from data and are widely diverse as they range from traditional statistical models based on inference to complex deep neural networks architectures.



Machine learning has many applications in <a href="https://financetrain.com/introduction-to-quantitative-trading/">quantitative finance</a> and their techniques have evolved and improved very fast in recent years. Some examples of problems that could be addressed with machine learning are the following:



<ul><li>Prediction of future asset price movements</li><li>Prediction of liquidity movements due to redemption of capital in large funds </li><li>Finding mispricing opportunities in niche markets</li><li>Categorize stocks for <a href="https://financetrain.com/portfolio-optimisation-in-r/">optimizing portfolio</a> allocations</li><li>Image recognition for use in commodity supply/demand signals</li><li>Natural Language Processing to gauge market sentiment and create <a href="https://financetrain.com/designing-a-trading-strategy-for-profit/">trading signals</a> for asset price forecasting</li></ul>



The big three categories in the machine learning field are Supervised Learning, Unsupervised Learning and Reinforced Learning.  All three methods have their own procedures and are used for different tasks such as prediction, find data patterns, categorizing data among others.



In this tutorial, we will focus on Supervised and Unsupervised Learning and we will provide an explanation of both fields showing examples in python programming for a practical overview and giving theoretical explanation to understand how these models works.



<h1>Machine Learning Steps</h1>



Working in machine learning involves many skills not only from statistics and mathematics, but also in the field of <a href="https://financetrain.com/cleaning-data-in-r-discarding-attributes/">data cleaning</a>, and data preparation/preprocessing.  Along this tutorial, we will focus in data preprocessing, training and testing models, model selection, performance evaluation and hyper-parameter tuning (Change default values of parameters of the models). The end-to-end workflow in machine learning can be described by the following picture: 



<figure class="wp-block-image size-large"><img src="https://financetrain.com/wp-content/uploads/machine-learning-steps.png" alt="" class="wp-image-26584"/></figure>



These steps at first follow this order but it is very common that once we have the predictions of the model, we start to work again on it and re-process the data in order to make new features, remove non-significant features, modify model parameters (hyperparameter tuning) and change the algorithm of the model (model selection).



In the next section we will explain important steps on the first stages of a machine learning project.  Along the course we provide code examples using Scikit Learn library from Python that has already implemented a great number of machine learning algorithms. 



The Scikit Learn is the most used library for Machine Learning in Python which has an extensive API for running machine learning algorithms.

What is Machine Learning?

Data preprocessing is where data scientist spent most of their time. These tasks involve selecting the appropriate features as well as clean and prepare them to become the inputs or independent variables in a machine learning model.&nbsp;



Model performance is strictly related with the selection and cleaning of the features.&nbsp; Below we describe common tasks which are necessary to conduct before fitting and evaluate a model. These tasks will improve the accuracy of the model due to the increase of the inputs quality.&nbsp;



<h3>Handling Data Types of the Features</h3>



In python data analysis, each column of a dataset is loaded into the environment with a specific data type. Most common data types are float, string, integer, datetime. In many occasion numerical data, is loaded as character or string type because could contains specific character that lead to interpret all column as a character column.



Likewise, the datetime fields if not have the correct format, is interpreted as strings or character object. Prior to start any analysis that uses dates is necessary to convert this variable into a datetime object.&nbsp;



To ensure the integrity of the data, one of the first steps is to explore the data types of a dataset in order to observe if they have the correct type.  



<h3>Handling Missing Data</h3>



Missing values are one of the common issues in machine learning models, and their causes are related to human errors, interruptions in the data flow, and incorrect measurements among others. Most of the algorithms in machine learning do not accept missing values and through errors if a dataset has missing values.



Therefore it is necessary to solve this issue utilizing some mechanism to deal or remove them. Removing rows or columns with missing data, can affect the performance of the model, as the size of the data decrease.



 A more elegant method is the Numerical imputation of the median of the variable in places with missing values. Median is preferable to the mean because it is not affected by outliers. This solution would preserve the size of the data. In case of categorical values, missing values can be replaced with the most common categorical value of the column.



<h3>Handling Outliers</h3>



For <a href="https://financetrain.com/new-course-financial-time-series-analysis-r/">financial time series</a>, outliers can be detected by plotting the distribution of the returns and observe if the distribution has extreme fat tails, which is a hint of some anomaly in the data.



Also outliers can be detected by using the percentiles of the data. Researchers can define certain percentage threshold in the upper-bottom of the distribution where values beyond these limits are considered outliers.&nbsp;



<h3>Binning&nbsp;</h3>



Binning means group a numerical or categorical variable into bins.&nbsp; In cases where categorical values have low frequency in a huge dataset, they can be binned into a category called “others”, which convert the model more robust.&nbsp; 



<figure class="wp-block-image size-large"><img src="https://financetrain.com/wp-content/uploads/binning-examples.png" alt="" class="wp-image-26588"/></figure>



<h3>Logarithmic Transformation</h3>



This technique is used in many statistical analysis and machine learning models, as the log remove the <a href="https://financetrain.com/interpretation-of-skewness-kurtosis-coskewness-cokurtosis/">skewness</a> of the data an approximate it distribution to a <a href="https://financetrain.com/normal-distribution/">normal distribution</a>. On the other hand, the log transformation of the variables decreases the effect of outliers.



<h3>Encoding data</h3>



One of the most common encoding methods in machine learning is called One Hot Encoding. The method spread the values in a column to multiple columns, where the values in the original columns are used to rename the new columns . 



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/one-hot-encoding.png" alt="" class="wp-image-26589"/></figure></div>



After the transformation the new columns takes two possible values that are 1 or 0. This method is mainly used with categorical variable and is similar to create dummy variables for each of the categorical values on a specific column.



<h3>Scaling</h3>



After scaling a dataset, all the continuous variables become identical in terms of the range. This process is not critic for some algorithms, but there are algorithms such as K-means (Unsupervised technique) &nbsp;that work with the distance measure, so it is required to scale all the inputs in order to have values that can be compared.



Some techniques to scale values on a dataset are Normalization and Standardization.



Normalization



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/normalization.png" alt="" class="wp-image-26590"/></figure></div>



Normalization scales all values in the range 0 and 1. Each value of the variable is subtracted by the min value and divided by the difference between the max value and the min value. The procedure does not change the distribution of the feature. Before normalization, outliers should be handled.



Standardization



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/standardization.png" alt="" class="wp-image-26591"/></figure></div>



This method is also called the z-score and scale each of the values of a column by removing from them the mean of the column and divided by the standard deviation. This technique decreases the effect of outliers in each feature.

Data Preprocessing in Data Science and Machine Learning

Feature Selection is one of the core concepts in machine learning and has a high impact on the performance of the model. Irrelevant or partially irrelevant features can negatively impact the model performance.



In this process those features which contribute most to the prediction variable are selected. In order to get an idea about which features could have more predictive power in a machine learning model, we will load Open, High, Low, Close, Volume (OHLCV) data for AMZ stock, and create some new features using Python.



Afterwards, we will create data visualizations on the new features and apply other common approaches for a smart selection of the features.



<div class="wp-block-file"><a href="https://financetrain.com/wp-content/uploads/AMZ.csv">AMZ Stock Data</a><a href="https://financetrain.com/wp-content/uploads/AMZ.csv" class="wp-block-file__button" download>Download</a></div>



We will be performing using Python. The example below has 4 main steps:



<ul><li>Import the Python libraries that will be used.</li><li>Calculate Technical Indicators with the <code>get_technical_indicators()</code> function.</li><li>Plot scatterplots among features and target variable <code>AdjClose.</code></li><li>Make a Heat Map to show the correlation between each of the features and the target variable.</li><li>Fit a Random Forest Model to extract feature importance of the independent variables (we will explain this algorithm in future sections but now will use some of the tools that Random Forest provides).</li></ul>



<pre class="wp-block-code"><code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from sklearn.ensemble import RandomForestRegressor</code></pre>



The data for AMZ stock is loaded into the environment using the <code>read_csv()</code> method.



<pre class="wp-block-code"><code># To read the file, make sure to provide path to the correct directory in your computer.

amz = pd.read_csv("C:/Users/Nicolas/Documents/Machine_Learning_Course/AMZ.csv")</code></pre>



We use the amz object dataframe which has OHLCV data from AMZ ticker and pass into the <code>get_technical_indicators() </code>function. This function generates technical indicators to be used as features for a machine learning model.



<pre class="wp-block-code"><code>def get_technical_indicators(dataset):
 '''
 params:
 dataset: OHLCV data for AMZ ticker from 1999-09-01 to 2019-09-20
 returns 
 features dataframe with the calculations of all technical Indicators such as 
 MACD, 20 period’s standard deviation, ROC, CCI, EMA
 '''
 # Sort values by dates. Old dates at top 
 dataset.sort_index(inplace=True)
 # Create the features dataframe to store only the features
 features = pd.DataFrame(index=dataset.index)
 
 # Create 7 and 21 days Moving Average
 features&#91;'ma7'] = dataset&#91;'AdjClose'].rolling(window=7).mean()
 features&#91;'ma21'] = dataset&#91;'AdjClose'].rolling(window=21).mean()
 
 # Create MACD
 features&#91;'26ema'] = dataset&#91;'AdjClose'].ewm(span=26).mean()
 features&#91;'12ema'] = dataset&#91;'AdjClose'].ewm(span=12).mean()
 features&#91;'MACD'] = (features&#91;'12ema']-features&#91;'26ema'])
 
 # Create Bollinger Bands
 features&#91;'20sd'] = dataset&#91;'AdjClose'].rolling(20).std()
 features&#91;'upper_band'] = features&#91;'ma21'] + (features&#91;'20sd']*2)
 features&#91;'lower_band'] = features&#91;'ma21'] - (features&#91;'20sd']*2)
 
 # Create Exponential moving average
 features&#91;'ema'] = dataset&#91;'AdjClose'].ewm(span=20).mean()
 
 # ROC Rate of Change
 N = dataset&#91;'AdjClose'].diff(10)
 D = dataset&#91;'AdjClose'].shift(10)
 features&#91;'ROC'] = N/D
 
 # CCI Commodity Channel Index
 TP = (dataset&#91;'High'] + dataset&#91;'Low'] + dataset&#91;'AdjClose']) / 3 
 features&#91;'CCI'] = (TP - TP.rolling(20).mean()) / (0.015 * TP.rolling(20).std() )
 # Create Average True Range 
 features&#91;'TR'] = dataset&#91;'High'] - dataset&#91;'Low']
 features&#91;'ATR'] = features&#91;'TR'].ewm(span = 10).mean()
 
 return features


# Store the output of get_technical_indicators() function in the features object

features = get_technical_indicators(amz)
</code></pre>



The features object stores all the calculations of the <code>get_technical_indicators()</code> function.



<pre class="wp-block-code"><code># Retrieve the features that will be used in the vars_ dataframe
vars_ = features&#91;&#91;'MACD','20sd','TR','ma21','ROC','CCI']]
 
# The correlation() function would make scatterplots between each of the features and the target variable AdjClose
 
jet= plt.get_cmap('jet')
colors = iter(jet(np.linspace(0,1,10)))
 
def correlation(df,features,variables, n_rows, n_cols):
 fig = plt.figure(figsize=(8,6))
 #fig = plt.figure(figsize=(14,9))
 for i, var in enumerate(variables):
 ax = fig.add_subplot(n_rows,n_cols,i+1)
 asset = features.loc&#91;:,var]
 ax.scatter(df&#91;"AdjClose"], asset, c = next(colors))
 ax.set_xlabel("AdjClose")
 ax.set_ylabel(" {}".format(var))
 ax.set_title(var +" vs AdjClose")
 fig.tight_layout() 
 plt.show()
 
columns = vars_.columns 
correlation(amz,vars_,columns,2,3)
</code></pre>



Figure 1: Feature Selection Scatterplots



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image6.png" alt="" class="wp-image-26604"/></figure></div>



The scatterplot shows that there is an extremely positive correlation between the ma21 variable and the AdjClose. We can also visualize that there is a positive correlation between the 20sd and the TR variables with the target variable AdjClose. 



On the other hand, there is a weak correlation of the features CCI and MACD with the target variable AdjClose.&nbsp;



Heat Maps are another tool that we can use to explore the relevance of each feature with respect to the target variable. The following lines make a Heat Map between the features and the target variable:



<pre class="wp-block-code"><code># Copy the vars_ dataframe into a new dataframe called df. 
# Add the target variable to the vars_ dataframe to make a correlation matrix among features and target variable. Finally show a Heat Map with the values of the correlation matrix.
 
df = vars_.copy()
df&#91;'AdjClose'] = amz&#91;'AdjClose']
 
colormap = plt.cm.inferno
plt.figure(figsize=(10,5))
corr = df.corr()
sns.heatmap(corr&#91;corr.index == 'AdjClose'], linewidths=0.1, vmax=1.0, square=True, cmap=colormap, linecolor='white', annot=True);
plt.show()
</code></pre>



Figure 2: Heat Map between AdjClose variable and features



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image4.png" alt="" class="wp-image-26605"/></figure></div>



From the Heat Map we observe that the ma21 has perfect correlation with the AdjClose variable. Also 20sd and TR have positive correlation with the AdjClose, while the MACD and CCI don’t show significant correlation with AdjClose.



Lastly, we will use a method that is provided on the Random Forest algorithm  that gives information about the feature importance.  Firstly, we need to fit the model and then utilize the method called feature_importance_ (these steps are explained in next sections, but here we want to inspect feature_importance_ method ) on the model object.In the next line, we will fit a Random Forest Regressor model between the features and the target variable of the model, and extract the features_importance_ method. Finally we make a bar plot of the features importance measure.



<pre class="wp-block-code"><code># Fit Random Forest Regressor and extract feature importance
prices = pd.DataFrame(amz&#91;'AdjClose'])
vars_model = prices.join(vars_)
vars_model= vars_model.dropna()
 
X = vars_model&#91;&#91;'MACD','20sd','TR','ROC','CCI']].values
y = vars_model&#91;'AdjClose'].values
 
forest = RandomForestRegressor(n_estimators=1000)
forest = forest.fit(X, y)
importances = forest.feature_importances_
 
values = list(zip(vars_model.columns&#91;1:],importances))
headers = &#91;'feature','score']
values_df = pd.DataFrame(values,columns = headers)
 
# Plot the feature importance
columns = &#91;'MACD','20sd','TR','ROC','CCI']
nd = np.arange(len(columns))
width=0.5
fig = plt.bar(nd, values_df&#91;'score'].values, color=sns.color_palette("deep", 5))
plt.legend(fig, columns, loc = 'upper right',bbox_to_anchor=(1.1, 1), title = "Feature Importance")
plt.show()
</code></pre>



Figure 3: Feature Importance



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image5.png" alt="" class="wp-image-26606"/></figure></div>



The bar plot of the feature importance show that 20sd has the greater importance in the prediction of the target variable AdjClose. On the second place comes the TR feature with significantly lower importance.



In this section we provided some tools to analyze the power prediction of the features before start to training a Machine Learning model. Is important to inspect the independent variables first for selecting the best features.&nbsp;



After the inspection of the features, the next step consists in the training and testing steps where is necessary to <a href="https://financetrain.com/predictive-modelling-splitting-data-training-test-set/">split the data among  the train and test data</a>. This is explained in the following section.

Feature Selection in Machine Learning

Once we have cleaned the data and have selected the features from the data for building the model, the next step  is to generate the <a href="https://financetrain.com/predictive-modelling-splitting-data-training-test-set/">train and test dataset</a>. We will divide our data into two different data sets, namely training and testing datasets. The model will be built using the training set and then we will test it on the testing set to evaluate how our model is performing. There are many ways in which we can split the data, for example, we can split the data randomly.



Splitting the data into training and testing set is required for Supervised Learning problems. Unsupervised Learning models don’t require a train and a test dataset. 



Classification and Regression problems would supervise or “train” a model with specific data in order to provide predictions of the target variable y. The process of training a dataset is conducted by choosing the set of relevant features or independent variables and combining these with a response y (labelled data) that is the observed value of the target variable.



In this phase, the algorithm is trained on the data and will determine the influence of each feature on the response y. Finally, we can make predictions for out-of-sample or unseen data based on the prior training experience.&nbsp;



This process has two main stages that are called training and testing the model. In the training phase as we described above, we fit the model with the data and afterwards we use the test data to assess the model performance.&nbsp;



The training dataset includes both features and the target variable while the testing dataset includes only features that are used to run the model and get the predictions of the target variable. The training dataset usually represent 70%-80% of the total data and the test dataset is the remaining portion of the data which is preserved to test the model accuracy.

Train-Test Datasets in Machine Learning

The predictions that the model returns will be compared with the real observation of the variable to obtain a measure of model performance. In order to evaluate the model performance, both classification and regression problems need to define a function called loss function that will compute the error of the model.&nbsp;



This loss function is a measure of the accuracy of the model as it calculates the differences between the true value of the response y and the estimation from the model y.



The loss function is different depending on whether we are working on a classification problem or a regression problem. In the classification setting the common loss functions are the 0-1 loss and the cross entropy. 0-1 loss Is a very basic loss function that assigns 1 to correct predictions and 0 to incorrect predictions. Measure the performance of a classification model whose output is a probability between 0 and 1. It does not care about how the errors are made. The Cross Entropy increases as the predicted value diverge from the actual label. On the other hand, in the regression problem, a common loss function is the Mean Squared Error (MSE). We will explain these metrics in the Model Selection section.



<h3>Loss Function Interpretation</h3>



The accuracy of the model is higher when the loss function is at a minimum, that is, when the difference between the true values and the estimated values is small. There are many factors that account for the minimization of the loss function such as the quality of the data, the amount of features used to train the model as well as the size of the data used.&nbsp;



Researchers and machine learning engineers will work with the model in order to minimize the loss function. However, if the loss function is minimized too severely, the model can get good results with the training data, but could fail in their performance to predict new data.



The above issue is generated when the model is “overfitted” to the data used in the training phase but has not learned how to generalize to new, unseen data.  In the <a href="https://financetrain.com/what-is-machine-learning/">machine learning</a> field, this situation is called the overfitting. 



Overfitting happens when a model captures the noise of the underlying pattern in data. These models have low bias and high variance. Bias is the difference between the average prediction of the model and the correct value which we are trying to predict. A model with high bias pays very little attention to the training data and oversimplifies the model.



Variance is the variability of model prediction for a data point. A model with high variance pays a lot of attention to training data and is not good to generalize on data which it hasn’t seen before. The results is that the model performs very well on training data but has high error rates for new data.Both bias and variance lead to a common situation in machine learning that is called the bias variance trade off.

Evaluate Model Performance - Loss Function

Model selection refers to choose the best statistical machine learning model for a particular problem. For this task we need to compare the relative performance between models. Therefore the loss function and the metric that represent it, becomes fundamental for selecting the right and non-overfitted model.&nbsp;



We can state a machine learning supervised problem with the following equation:



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/supervised-equation.png" alt="" class="wp-image-26614"/></figure></div>



This equation is composed with the x matrix that contains the predictor’s factors x1,x2,x3,…xn. These factors can be the lagged prices/returns of a time series or some others factors such as volume, foreign exchange rates, etc.&nbsp; y is the response vector that depend of the function f and the predictors x.



f contain the underlying relationship between the x features and the y response and can be modeled with a linear regression if the underlying relationship is linear or with a Random Forest or Support Vector Machine algorithm if the underlying relationship is non-linear.



Ε represent the error term, which is often assumed to have mean zero and a standard deviation of one.



Once we fit a particular model for a certain dataset, we need to define the loss function that we will use to assess model performance. Many measures can be used for the loss function. Some common measures for the loss function are the Absolute Error and the Squared Error between predicted values and real values. 



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/loss-function.png" alt="" class="wp-image-26615"/></figure></div>



Both choices are non-negative, so the best value for the loss function is zero. The Absolute Error and Squared Error above, compute the difference between the true value (y)&nbsp; and the prediction (y)&nbsp; for each observation of the dataset.



Both the Absolute Error and Squared Error are vectors or arrays of n x 1 dimension, reflecting the error term per each of the observations. In order to aggregate the error term of a certain model between all the predicted and real values of a variable, a popular measure is the Mean Squared Error which is simply the average of the squared loss: 



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/mean-squared-error.png" alt="" class="wp-image-26616"/><figcaption> Where n is the number of observations</figcaption></figure></div>

Model Selection in Machine Learning

The interesting property of a machine learning model is its capacity to predict or categorize new unseen data (data that was not used in training the model). For this reason the important measure is the MSE error with test data, which is denominated as test MSE. The goal is to choose a model where the test MSE is the lowest across other models. 



The bias variance tradeoff  is generated when at some point if we increase the bias of the model by creating additional features, the variance of the model increases too (overfitting), and on the other hand if the model is too simple (has very few parameters),  it will have high bias and low variance (under fitting). 



It is necessary to find the right balance between bias and variance without overfitting and under fitting the data. The prediction error in a Supervised machine learning algorithm can be divided into three different parts:



<ul><li>Bias Error</li><li>Variance Error</li><li>Irreducible Error</li></ul>



First we will write the equation which breaks these three factors:



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/bias-1.png" alt="" class="wp-image-26628"/></figure></div>



Variance Error



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/bias-2.png" alt="" class="wp-image-26629"/></figure></div>



The first term on the right hand side is the variance of the estimation across many testing sets. This measures the average model deviation among different testing data. In particular, a model with high variance is suggestive that it is overfit to the training data. In this scenario, the model is capturing the noise of the training dataset but it is poor for new data. 



Bias Error



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/bias-3.png" alt="" class="wp-image-26630"/></figure></div>



The squared bias characterizes the difference between the averages of the estimate and the true values. A model that has high bias, would not capture the underlying behavior of the true functional form well. One example of this case is when a linear regression is used to fit a model with a non-linearity relationship between their features.



Irreducible error



Var(∈)



This term represents the minimum lower bound for the test MSE. This error cannot be reduced regardless what algorithm is used. This error reflects the influence that unknown variables have between the interaction of the features and the target variable.



Bias and variance moves in different direction and the relative rate of change between these two factors determines whether the expected test MSE increases or decreases.



In the plot below we show the trade-off between bias and variance. At first time, as flexibility increase the bias tend to drop quickly, faster than the increase in variance generating a reduction on the test MSE of the model (Total Error). However, as flexibility increases further, there is less reduction in bias and the variance rapidly increase, causing model overfitting.



The goal of any supervised machine learning algorithm is to achieve low bias and low variance, and all models need a well balance between these factors. For this reason is necessary to understand both factors to reach the optimal point that consist in the lower bias-variance.



Figure: Bias-Variance Trade Off



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image7.png" alt="" class="wp-image-26631"/></figure></div>

Bias Variance Trade Off

As we pointed out earlier, both classification and regression models are in the field of Supervised Learning. These models are characterized by having a group of features or independent variables and a target variable that is the variable that the model aims to predict.&nbsp;



This target variable is called the labelled data and is the main property of the Supervised Learning models because it acts as the orientation for constructing the model in the training phase and to evaluate model performance.&nbsp;&nbsp;



In a classification problem the target variable to predict (y) is a categorical variable and can take a finite set of possible choices, K. On the other hand, in the regression problem, the target variable y is a real value rather than categorical.



The classification problem has the goal of estimating membership for a set of features into a particular group. A common classification problem in the financial sector is to determine the price direction for the next day based on N days of asset price history.



The regression problem involves estimate a real value response with a set of features or predictors as the independent variables. In the financial field, an example is to estimate tomorrows asset price based on the historical prices (or other features) of the price. The regression problem would estimate the real value of the price and not just its direction.



Common classification algorithms include Logistic Regression, Naïve Bayes Classifiers, Support Vector Machines, Decision Tree, and Deep Convolution Neural Networks. Common regression techniques include Linear Regression, Support Vector Regression and Random Forest.



In the next lessons, we will explain relevant concepts of the most popular algorithms used for Supervised Learning. These algorithms are the following:



<ul><li>Multiple Linear Regression</li><li>Logistic Regression</li><li>Decision Tree-Random Forest</li><li>Support Vector Machine</li><li>Linear Discriminant Analysis</li></ul>

Supervised Learning Models

The multiple linear regression algorithm states that a response y can be estimated with a set of input features x and an error term ɛ. The model can be expressed with the following mathematical equation:



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/reg-eq.png" alt="" class="wp-image-26636"/></figure></div>



&beta;TX is the matrix notation of the equation, where &beta;T, X ϵ ʀp+1 and ɛ ~ N(μ,&sigma;2)



βT(transpose of β) and X are both real-valued vectors with dimension p+1 and ɛ is the residual term which represents the difference between the predictions of the model and the true observation of the variable y.



The vector βT = (β0,β1,…βP) stores all the beta coefficients of the model. These coefficients measure how a change on some of the independent variable impact on the dependent or target variable.



The vector X = (1,x1,x2, …xp) hold all the values of the independent variables. Both vectors (T and X) are p+1 dimensional because of the need to include an intercept term. 



The goal of the <a href="https://financetrain.com/introduction-to-linear-regression/">linear regression</a> model is to minimize the difference between the predictions and the real observations of the target variable. For this purpose, a method called <a href="https://financetrain.com/ordinary-least-squares-ols/">Ordinal Least Squares (OLS)</a> is used which will derive the optimal set of coefficients for fitting the model. 



<h3>Ordinal Least Squares</h3>



Formally the OLS model will minimize the Residual Sum of Squares (RSS) between the observations of the target variable and the predictions of the model. The RSS is the loss function metric to assess model performance in the linear regression model and has the following formulation:



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/rss.png" alt="" class="wp-image-26637"/></figure></div>



Residual Sum of Squares also known as the Sum of Squared Errors (SSE) between the predictions βTxi and the observations yi. With the minimization of this function, it is possible to get the optimal parameter estimation of the vector β.



In matrix notation, the RSS equation is the following:



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/rss1.png" alt="" class="wp-image-26638"/></figure></div>



To get the optimal values of β, it is necessary to derivate RSS respect to β:



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/rss-derivative.png" alt="" class="wp-image-26639"/></figure></div>



Remember that X is a matrix with all the independent variables and has N observations and p features. Therefore, the dimension of X is N (rows) x p+1 (columns).



One assumption of this model is that the matrix XTX should be positive-define. This means that the model is valid only when there are more observations than dimensions. In cases of high-dimensional data (e.g. text document classification), this assumption is not true.



Under the assumption of a positive-definite XTX the differentiated equation is set to zero and the β parameters are calculated:



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/b-params.png" alt="" class="wp-image-26640"/></figure></div>



Later we will show an example using a dataset of Open, High, Low, Close and Volume of the S&amp;P 500 to fit and evaluate a multiple linear regression algorithm using Scikit learn library.

Multiple Linear Regression

In machine learning, the <a href="https://financetrain.com/logistic-regression-model-in-r-2/">Logistic Regression</a> algorithm is used for classification problems. It provides an output that we can interpret as a probability that a new observation belongs to a certain class. Generally, logistic regression is used to classify binary classes but works on multiple and ordinal classes too.



Logistic regression estimates a continuous quantity which is the probability that an event occurs. This probability is compared  with a certain threshold that allow taking the decision about the classification of the new data.



If the threshold of the probability is equal to 0.5, we can classify the new data to each of the classes by comparing the probability value with that threshold.



Instead of fitting a straight line or hyperplane as in a <a href="https://financetrain.com/introduction-to-linear-regression/">linear regression model</a>, the logistic regression model uses the logistic function to reflect the output of a linear equation between 0 and 1. With this function is possible to map real values of the predictions into probabilities. (Note: Logistic regression uses the sigmoid function for the two class logistic regression and the softmax function for the multiclass logistic regression).



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/logistic-eq.png" alt="" class="wp-image-26643"/></figure></div>



Logistic Regression Sigmoid Function



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image8.png" alt="" class="wp-image-26644"/></figure></div>



The logistic regression is a kind of reformulation of the linear regression for classification problems, as the linear regression is not good to separate classes. In a linear regression, we model the relationship between the features and the target variable with the following equation:



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/lin-eq.png" alt="" class="wp-image-26645"/></figure></div>



In order to address classification problems, we convert this equation to obtain probabilities between 0 and 1. The right side of the above equation is wrapped into a logistic function, and therefore forces the output to be between 0 and 1.



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/logi-eq.png" alt="" class="wp-image-26646"/></figure></div>



<h3>Types of Logistic Regression</h3>



<ul><li>Binary Logistic Regression: The target variable has only two possible outcomes.</li><li>Multiple Logistic Regression: The target variable has three or more nominal categories.</li><li>Ordinal Logistic Regression: The target variable has three or more ordinal categories such as assign movies score from 1 to 5. </li></ul>

Logistic Regression

Using the scikit-learn package from python, we can fit and evaluate a <a href="https://financetrain.com/logistic-regression/">logistic regression </a>algorithm with a few lines of code. Also, for binary classification problems the library provides interesting metrics to evaluate model performance such as the <a href="https://financetrain.com/create-a-confusion-matrix-in-r/">confusion matrix</a>, Receiving Operating Curve (ROC) and the Area Under the Curve (AUC).



Hyperparameter Tuning in Logistic Regression in Python



In the Logistic Regression model (as well as in the rest of the models), we can change the default parameters from scikit-learn implementation, with the aim of avoiding model overfitting or to change any other default behavior of the algorithm.&nbsp;



For the Logistic Regression some of the parameters that could be changed are the following:



<ul><li>Penalty: (string) specify the norm used for penalizing the model when its complexity increases, in order to avoid overfitting.&nbsp; The possible parameters are “l1”, “l2” and “none”. “l1” is the Lasso Regression and “l2” is the Ridge Regression that represents two different ways to increase the magnitude of the loss function. Default value is “l2”. “none” means no <a href="https://financetrain.com/regularization-data-science-lasso-ridge-elastic-net/">regularization</a> parameter. l1, lasso regression, adds “absolute value of magnitude” of coefficient as penalty term to the loss function. l2, ridge regression, adds “squared magnitude” of coefficient as penalty term to the loss function.</li><li>C: (float): the default value is 1. With this parameter we manage the λ value of regularization as C = 1/λ.&nbsp; Smaller values of C mean strong regularization as we penalize the model hard.</li><li>multi_class: (string) the default value is “ovr” that will fit a binary problem. To fit a multiple classification should pass “multinomial”.</li><li>solver: (string) algorithm to use in the optimization problem: “newton-cg”, “lbfgs”,”liblinear”,”sag”, and ”saga”. Default is “liblinear”. These algorithms are related to how the optimization problem achieves the global minimum in the loss function.<ul><li>“liblinear”: is a good choice for small datasets</li><li>“Sag” or “saga”: useful for large datasets</li><li>“lbfgs”,”sag”,”saga”, or “newton-cg”: handle multinomial loss, so they are suitable for multinomial problems.</li><li>“liblinear” and “saga”: handle “l1” and “l2” penalty</li></ul></li></ul>



The penalty has the objective to introduce regularization which is a method to penalize complexity in models with high amount of features by adding new terms to the loss function.&nbsp;



It is possible to tune the model with the regularization parameter lambda (λ) and handle collinearity (high correlation among features), filter out noise and prevent overfitting.&nbsp;



By increasing the value of lambda (λ) in the loss function we can control how well the model is fit to the training data. As stated above, the value of λ in the logistic regression algorithm of scikit learn is given by the value of the parameter C, which is 1/λ.&nbsp;



To show these concepts mathematically, we write the loss function without regularization and with the two ways of regularization: “l1” and “l2” where the term 



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/term.png" alt="loss function term" class="wp-image-26662"/></figure></div>



are the predictions of the model.



Loss Function without regularization



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/l0.png" alt="Loss Function Without Regularization" class="wp-image-26663"/></figure></div>



Loss Function with l1 regularization



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/l1.png" alt="Loss Function with l1 Regularization" class="wp-image-26664"/></figure></div>



Loss Function with l2 regularization



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/l2.png" alt="Loss function with l2 regularization" class="wp-image-26665"/></figure></div>

Logistic Regression in Python using scikit-learn Package

Decision tree is a popular Supervised learning algorithm which can handle classification and regression problems. For both problems, the algorithm breaks down a dataset into smaller subsets by using if-then-else decision rules within the features of the data.&nbsp;



The general idea of a decision tree is that each of the features are evaluated by the algorithm and used to split the tree based on the capacity that they have to explain the target variable. The features could be categorical or continuous variables.&nbsp;&nbsp;



In the process of building the tree, the most important features are selected by the algorithm in a top-down approach creating the decision nodes and branches of the tree and making predictions at points when the tree cannot be expanded any more.&nbsp;



The tree is composed by a root node, decision nodes and leaf nodes. The root node is the top most decision on the tree, and is the first time where the tree is split based on the best predictor of the dataset.&nbsp;



The decision nodes are the intermediate steps in the construction of the tree and are the nodes used to split the tree based on different values of the independent variables or features of the model. The leaf nodes represent the end points of the tree, and hold the prediction of the model.&nbsp;&nbsp;



Decision Tree Structure



<figure class="wp-block-image size-large"><img src="https://financetrain.com/wp-content/uploads/image9.png" alt="Decision Trees in Machine Learning" class="wp-image-26668"/></figure>



The methods to split the tree are different depending if we are in a classification or regression problem. We will provide an overview of the different measures that are used to split the tree in both types of problems.



<h3>Classification Problem</h3>



In this type of problem the algorithm decides which is the best feature to split the tree based on two measures that are Entropy and Information Gain.&nbsp;



Entropy



Entropy is a coefficient that takes values between 0 and 1. It is a measure of randomness in data. If entropy is equal to zero, it means that all the data belongs to the same class, and if entropy is equal to 1, it means that half of the data belong to one class and the half belong to the other class (in binary classification). In this last situation, the data is perfect randomness.



The mathematical formula of entropy is as follows:



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/entropy.png" alt="" class="wp-image-26669"/></figure></div>



The goal of a decision tree problem is to reduce the measure of entropy while building the tree. As we stated above, entropy is a measure of the randomness of the data, and this measure needs to be reduced.&nbsp; For this purpose, the entropy is combined with the Information Gain in order to select the feature that has higher value of Information Gain.



Information Gain



The Information gain is based on the decrease in Entropy after a dataset is split on an attribute or feature. The attribute that has the high value of Information Gain will be used to split the tree in nodes in a top-down approach.The mathematical formula of Information Gain is the following:



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/information-gain.png" alt="" class="wp-image-26670"/></figure></div>



In the equation above, the Information Gain subtracts the entropy of the target variable Y given X (E(Y/X)) from the entropy of the target variable Y (EY), to calculate the reduction of uncertainty of Y given additional piece of information X.



The aim of the decision tree in this type of problem is to reduce the entropy of the target variable. For this, the decision tree algorithm would use the Entropy and the Information Gain of each feature to decide what attribute will provide more information (or reduce the uncertainty of the target variable).&nbsp;



<h3>Decision Tree Regression Problem</h3>



In the regression problem, the tree is composed (like in the classification problem) with nodes, branches and leaf nodes, where each node represents a feature or attribute, each branch represents a decision rule and the leaf nodes represent an estimation of the target variable.



Contrary to the classification problem the estimation of the target variable in a regression problem is a continuous value and not a categorical value. The ID3 algorithm (Iterative Dichotomiser 3) can be used to construct a decision tree for regression which replace the Information Gain metric with the Standard Deviation Reduction (SDR).&nbsp;



The Standard Deviation Reduction is based on the decrease in standard deviation after a dataset is split on an attribute. The tree will be split by the attribute with the higher Standard Deviation Reduction of the target variable.



The standard deviation is used as a measure of homogeneity of the data. The tree is built top-down from the root node and data is partitioned into subsets that contain homogeneous instances of the target variable. If the samples are homogeneous, its standard deviation is zero.



The mathematical formula of the Standard Deviation Reduction is the following: 



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/sdr1.png" alt="" class="wp-image-26672"/></figure></div>



Y is target variable



X is a specific attribute of the dataset



SDRY,X is the reduction of the standard deviation of the target variable Y given the information of X



SY is the standard deviation of the target variable



S(Y,X)&nbsp; is the standard deviation of the target variable given the information of X



The calculation of S(Y,X) can be broken down with the following formula that is important to understand the process:



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/syx.png" alt="" class="wp-image-26673"/></figure></div>



Sc is the standard deviation of the target variable based on unique values of the independent variable.



Pc&nbsp; number of times (probability) of the value of the independent variables over the total values of the independent variable&nbsp;&nbsp;



So the S(Y,X) is the sum of the probability of each of the values of the independent variable multiplied by the standard deviation of the target variable based on those values of the independent variable.&nbsp;



The Standard deviation S(Y,X) is calculated for each feature or independent variable of the dataset. The feature with the great value of S(Y,X) is the feature that has more capability to explain the variations in the target variable, therefore is the feature that the decision tree algorithm will use to split the tree.



The same idea is that the feature that will be used to split the tree is the one that has the greatest Standard Deviation Reduction. The overall process of splitting a decision tree under the Standard Deviation Reduction can be synthetized in the following steps:



<ul><li>Calculate the standard deviation of the target variable</li><li>Calculate the Standard Deviation Reduction for all the independent variables</li><li>The feature with the higher value on the Standard Deviation Reduction is chosen for the decision node.</li><li>In each branch, the Coefficient Variation (CV) is calculated. This coefficient is a measure of homogeneity in the target variable. CV = Sx*100 (Standard deviation) /(mean of the target variable) * 100</li><li>&nbsp;If the CV is less than certain threshold such as 10% that subset does not need further splitting and the node become a leaf node.</li></ul>



<h3>Overfitting in Decision Tree</h3>



One of the most common problems with decision tree is that they tend to overfit a lot. There are situations where the tree is built with specific features of the training data and the predictive power for unseen data is reduced.&nbsp;



This is because the branches of the tree follow strict and specific rules of the train dataset but for samples that are not part of the training set the accuracy of the model is not good. There are two main ways to address over-fitting in Decision tree. One is by using the Random Forest algorithm and the second is by pruning the tree.



<h4>Random Forest</h4>



Random Forest avoid overfitting as it is an ensemble of n decision trees (not just one) with n results in the end. The final result of Random forest is the most frequent response variable among the n results of the n Decision Trees if the target variable is a categorical variable or the average of the target variable in the leaf node if the target variable is a continuous variable.



<h4>Pruning</h4>



Pruning is done after the initial training is complete, and involves the removal of nodes and branches starting from the leaf nodes such that the overall accuracy is not affected.&nbsp; The pruning technique will reduce the size of the tree by removing nodes that provide little power to classify or predict new instances.&nbsp;

Decision Trees in Machine Learning

The Random Forest algorithm can be described in the following conceptual steps:



<ul><li>Select k features randomly from the dataset and build a decision tree from those features where k &lt; m (total number of features)</li><li>Repeat this n times in order to have n decision trees from different random combinations of k features.&nbsp;</li><li>Take each of the n built Decision Trees and pass a random variable to predict the outcome and store this outcome to get a total of n outcomes from n decision trees.</li><li>If the target variable is a categorical variable, each tree in the forest would predict the category to which the new record belongs and the new record is assigned to the category that has the majority vote.&nbsp;</li><li>If the target variable is a continuous variable, each tree in the forest predicts a value for the target variable and the final value is calculated by taking the average of all the values predicted by the trees that are part of the forest.</li></ul>



Using the scikit learn package from Python, it is possible to use and tune a Random Forest model based on predefined conditions that will give instructions to the algorithm regarding the construction of&nbsp; the trees that are part of the forest.



Hyperparameter Tuning



The scikit learn library allows to tune some important parameters in the tree construction that could increase the predictive power of the model or make the model faster.&nbsp;



Parameters to increase the predictive power:



<ul><li>n_estimators: represents the number of trees in the forest. In general, a higher number of trees increases the performance and makes the predictions more stable. The default value of this parameter is 10.</li><li>max_features: this parameter reflects the maximum number of features from the dataset that the Random Forest is allowed to use in an individual tree when considered for the best split. The default value of the parameter is sqrt(n_features) - Root Square of the n features of the model.</li><li>min_samples_leaf: the minimum number of observations in each leaf node. This parameter would prevent further splitting when the number of observations in the node is less than the value of this parameter. The default value of this parameter is 1.</li><li>max_depth: the maximum depth of the tree. The depth of a decision tree is the length of the longest path from a root to a leaf. The default value of this parameter is None. In this case the tree is split until nodes contain less than min_samples_split samples.</li></ul>



Parameters to increase the model speed:



<ul><li>n_jobs: tells the engine the number of jobs to run in parallel for fit and predict.</li><li>random_state: this parameter makes the model reproducible, as it will output the same result every time the model is run.</li><li>oob_score: Boolean value that allows to test the model performance keeping a portion of the data to test the model performance.</li></ul>

Random Forest Algorithm in Python

The Support Vector Machine is a Supervised <a href="https://financetrain.com/what-is-machine-learning/" class="rank-math-link">Machine Learning</a> algorithm that can be used for both classification and regression problems. However, it is most used in classification problems. The goal of the algorithm is to classify new unseen objects into two separate groups based on their properties and a set of examples that are already classified. 



Some common applications for this algorithm are the following:



<ul><li>Classify new documents into positive and negative sentiment categories based on other documents that have already been classified.</li><li>Classify new emails into spam or not spam based on a large set of emails that have already been marked as spam or not spam.</li><li>Bioinformatics (Protein Classification, Cancer Classification)</li><li>Other tasks that are characterized to have a higher feature dimensional space.</li></ul>



The algorithm creates a partition of the features space into two subspaces. Once these subspaces are created, new unseen data can be classified in some of these locations.&nbsp;



To address non-linear relationship, the algorithm follows a technique called the kernel trick to transform the data and find an optimal boundary between the possible outputs. Basically these are techniques to project the data into a higher dimension such that a linear separator&nbsp;would&nbsp;be sufficient to split the features space.



The separation between the two zones is given by a boundary which is called the hyperplane. Let us look at how the concept of hyperplane works in the Support Vector Machine algorithm.



<h3>Hyperplane</h3>



An interesting feature about the Support Vector Machine algorithm is that this hyperplane does not need to be a linear hyperplane. The hyperplane can introduce various types of non-linear decision boundaries. This feature increases the usability of the Support Vector Machine algorithm as it is capable to address non-linear relationships between features and target variables.



The algorithm will find the hyperplane that best separates the classes of the target variable by training a set of n observation of the features. Each observation is a pair of the p-dimensional vector of features (x1, x2, x3, …, xp) and the class yi which is the label. 



Usually the classes of the target variable are defined as +1 and -1, where each of these classes has their own meaning (negative vs positive, spam vs non-spam, malign vs benign).&nbsp;



When yi = 1, it implies that the sample with the feature vector xi belongs to class 1 and if yi = −1, it implies that the sample belongs to class -1.



The +1 class could be “non-spam” or “positive” sentiment and the class -1 could be “spam” and “negative” sentiment.&nbsp; Depending on which side of the hyperplane a new observation is located on, we assign it a particular class (-1, +1).



<h3>Deriving the hyperplane</h3>



The dimension of the hyperplane depends on the number of features. If the number of input features is two (R2), the hyperplane can be drawn as a line. If the number of input features is three, then the hyperplane becomes a two dimensional plane.



In order to optimize the hyperplane or the separation boundary between the two classes there are two important concepts that need to be described. These are the concepts of Maximal Margin Hyperplane (MMH) and the Maximal Margin Classifier (MMC).



<h4>Maximal Margin Hyperplane</h4>



The classes of a target variable can be well segregated by more than one hyperplane as we can observe in the figure below. However, we should select the hyperplane with the maximum distance between the nearest observations data points and the hyperplane, and this condition is achieved by the Maximal Margin Hyperplane. 



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image10.png" alt="" class="wp-image-26786"/><figcaption>Multiples Separating Hyperplane</figcaption></figure></div>



Therefore, maximizing the distance between the nearest points of each class and the hyperplane would result in an optimal separating hyperplane. 



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image11.png" alt="" class="wp-image-26787"/><figcaption>Maximum Separating Hyperplane</figcaption></figure></div>



The continuous grey line in the above figure is the Maximal Margin hyperplane, as is the hyperplane that maximizes the distance between it and the nearest data points of the two classes. To find the Maximal Margin Hyperplane, it is necessary to compute the perpendicular distance from each of the training observations xi&nbsp; for a given separating hyperplane.



The smallest perpendicular distance to a training observation from the hyperplane is known as the margin. The Maximum Margin Hyperplane is the separating hyperplane where the margin is the largest.



<h3>Non-linear relationships</h3>



Support Vector machine handles situations of non-linear relations in the data by using a kernel function which map the data into a higher dimensional space where a linear hyperplane can be used to separate classes.



If the data can’t be separated by linear discrimination, it is possible to use some trick to project the data in a higher dimension and apply a linear hyperplane. The figure below shows a scenario where the data cannot be separated using a linear hyperplane:



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image12.png" alt="" class="wp-image-26788"/><figcaption>Non-linear separation to the data</figcaption></figure></div>



In situations such as in this figure, Support Vector Machines algorithm becomes extremely powerful because using the kernel trick technique they can project the data into a higher-dimensional space and thereby we’re able to fit nonlinear relationships with a linear classifier.



Making a transformation of the space, and adding one more dimension to the data, will allow the data to be linearly separable. The following plot shows how the previous classes can be linearly separated with the increment of the dimension of the space from 2 dimensions to 3 dimensions.&nbsp;



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image13.png" alt="" class="wp-image-26789"/><figcaption>Kernel Transformation to 3D</figcaption></figure></div>



The kernel trick represent a real strength in the Support Vector Machine algorithm as we can solve non-linear classification problems with efficiency. 



<h3>Support Vector Machine Python - Parameter Tuning</h3>



The Support Vector Machine  algorithm can be fit with the scikit learn library from python which provides a straightforward method to use this algorithm. It is possible to parametrize and tune important parameters of the algorithm that are explained below:



<ul><li>Kernel: (string) select the type of hyperplane used to separate the data. The options are “linear”, “rbf”, “poly”. “linear” will use a linear hyperplane whereas “rbf” and “poly” uses a non-linear hyperplane. The default argument is “rbf”.</li><li>Gamma:&nbsp; (float) is a parameter for non-linear hyperplanes. The higher the gamma value more precisely it tries to fit the training data set. Default argument is “auto” which uses 1 / n_features. Low gamma value means to consider far away point when deciding the decision boundary.</li><li>C: (float) is the penalty parameter of the error term. It regulates overfitting by controlling the trade-off between smooth decision boundary and classifying the training points correctly.&nbsp; Greater values of C lead to overfitting to the training data.</li><li>Degree: (integer) is a parameter used when kernel is set to “poly”. It is the degree of the polynomial used to find the hyperplane to split the data. The default value is 3. Using degree=1 is the same as using “linear” kernel.&nbsp;</li></ul>

Support Vector Machine Algorithm Explained

In this post, we will provide an example of machine learning regression algorithm using the multivariate <a href="https://financetrain.com/multiple-linear-regression/" class="rank-math-link">linear regression</a> in Python from scikit-learn library in Python. The example contains the following steps:



Step 1: Import libraries and load the data into the environment.



Step 2: Generate the features of the model that are related with some measure of volatility, price and volume.&nbsp;



Step 3: Visualize the correlation between the features and target variable with scatterplots.



Step 4: Create the train and test dataset and fit the model using the linear regression algorithm.&nbsp;



Step 5: Make predictions, obtain the performance of the model, and plot the results. 



<h3>Step 1: Import libraries and load the data into the environment.</h3>



We will first import the required libraries in our Python environment.



<pre class="wp-block-code"><code>import pandas as pd 
from datetime import datetime
import numpy as np 
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt</code></pre>



We will work with SPY data between dates 2010-01-04 to 2015-12-07. 



<div class="wp-block-file"><a href="https://financetrain.com/wp-content/uploads/SPY_regression.csv">SPY_regression Data</a><a href="https://financetrain.com/wp-content/uploads/SPY_regression.csv" class="wp-block-file__button" download>Download</a></div>



First we use the <code>read_csv()</code> method to load the csv file into the environment. Make sure to update the file path to your directory structure.



<pre class="wp-block-code"><code>SPY_data = pd.read_csv("C:/Users/FT/Documents/MachineLearningCourse/SPY_regression.csv")
 
# Change the Date column from object to datetime object 
SPY_data&#91;"Date"] = pd.to_datetime(SPY_data&#91;"Date"])
 
# Preview the data
SPY_data.head(10)</code></pre>



The data has the following structure:



<pre class="wp-block-code"><code> Date Open High Low Close Volume Adj Close
0 2015-12-07 2090.419922 2090.419922 2066.780029 2077.070068 4.043820e+09 2077.070068
1 2015-12-04 2051.239990 2093.840088 2051.239990 2091.689941 4.214910e+09 2091.689941
2 2015-12-03 2080.709961 2085.000000 2042.349976 2049.620117 4.306490e+09 2049.620117
3 2015-12-02 2101.709961 2104.270020 2077.110107 2079.510010 3.950640e+09 2079.510010
4 2015-12-01 2082.929932 2103.370117 2082.929932 2102.629883 3.712120e+09 2102.629883
5 2015-11-30 2090.949951 2093.810059 2080.409912 2080.409912 4.245030e+09 2080.409912
6 2015-11-27 2088.820068 2093.290039 2084.129883 2090.110107 1.466840e+09 2090.110107
7 2015-11-25 2089.300049 2093.000000 2086.300049 2088.870117 2.852940e+09 2088.870117
8 2015-11-24 2084.419922 2094.120117 2070.290039 2089.139893 3.884930e+09 2089.139893
9 2015-11-23 2089.409912 2095.610107 2081.389893 2086.590088 3.587980e+09 2086.590088
</code></pre>



Let's now set the Date as index and reverse the order of the dataframe in order to have oldest values at top.



<pre class="wp-block-code"><code># Set Date as index
SPY_data.set_index('Date',inplace=True)
 
# Reverse the order of the dataframe in order to have oldest values at top
SPY_data.sort_values('Date',ascending=True)</code></pre>



<h3>Step 2: Generate features of the model </h3>



We will generate the following features of the model:



<ul><li>High - Low percent change</li><li>5 periods Exponential Moving Average</li><li>Standard deviation of the price over the past 5 days </li><li> Daily volume percent change </li><li> Average volume for the past 5 days</li><li> Volume over close price ratio </li></ul>



<pre class="wp-block-code"><code>SPY_data&#91;'High-Low_pct'] = (SPY_data&#91;'High'] - SPY_data&#91;'Low']).pct_change()
SPY_data&#91;'ewm_5'] = SPY_data&#91;"Close"].ewm(span=5).mean().shift(periods=1)
SPY_data&#91;'price_std_5'] = SPY_data&#91;"Close"].rolling(center=False,window= 30).std().shift(periods=1)
 
SPY_data&#91;'volume Change'] = SPY_data&#91;'Volume'].pct_change()
SPY_data&#91;'volume_avg_5'] = SPY_data&#91;"Volume"].rolling(center=False,window=5).mean().shift(periods=1)
SPY_data&#91;'volume Close'] = SPY_data&#91;"Volume"].rolling(center=False,window=5).std().shift(periods=1)</code></pre>



<h3>Step 3: Visualize the correlation between the features and target variable</h3>



Before training the dataset, we will make some plots to observe the correlations between the features and the target variable.



<pre class="wp-block-code"><code>jet= plt.get_cmap('jet')
colors = iter(jet(np.linspace(0,1,10)))
 
def correlation(df,variables, n_rows, n_cols):
 fig = plt.figure(figsize=(8,6))
 #fig = plt.figure(figsize=(14,9))
 for i, var in enumerate(variables):
 ax = fig.add_subplot(n_rows,n_cols,i+1)
 asset = df.loc&#91;:,var]
 ax.scatter(df&#91;"Adj Close"], asset, c = next(colors))
 ax.set_xlabel("Adj Close")
 ax.set_ylabel("{}".format(var))
 ax.set_title(var +" vs price")
 fig.tight_layout() 
 plt.show()
 
# Take the name of the last 6 columns of the SPY_data which are the model features
variables = SPY_data.columns&#91;-6:] 
 
correlation(SPY_data,variables,3,3)
</code></pre>



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image15.png" alt="" class="wp-image-26793"/><figcaption>Correlations between Features and Target Variable (Adj Close)</figcaption></figure></div>



The correlation matrix between the features and the target variable has the following values:



<pre class="wp-block-code"><code>SPY_data.corr()&#91;'Adj Close'].loc&#91;variables]

High-Low_pct 	 -0.010328
ewm_5 0.998513
price_std_5 	 0.100524
volume Change 	-0.005446
volume_avg_5 	-0.485734
volume Close 	-0.241898
</code></pre>



Either the scatterplot or the correlation matrix reflects that the Exponential Moving Average for 5 periods is very highly correlated with the Adj Close variable. Secondly is possible to observe a negative correlation between Adj Close and the volume average for 5 days and with the volume to Close ratio.



<h3>Step 4: Train the Dataset and Fit the model</h3>



Due to the feature calculation, the SPY_data contains some NaN values that correspond to the first’s rows of the exponential and moving average columns. We will see how many Nan values there are in each column and then remove these rows.



<pre class="wp-block-code"><code>SPY_data.isnull().sum().loc&#91;variables]

High-Low_pct 	1
ewm_5 	1
price_std_5 30
volume Change 	 1
volume_avg_5 	5
volume Close 	5

# To train the model is necessary to drop any missing value in the dataset.

SPY_data = SPY_data.dropna(axis=0)

# Generate the train and test sets

train = SPY_data&#91;SPY_data.index &lt; datetime(year=2015, month=1, day=1)]

test = SPY_data&#91;SPY_data.index >= datetime(year=2015, month=1, day=1)]
dates = test.index</code></pre>



<h3>Step 5: Make predictions, obtain the performance of the model, and plot the results</h3>



In this step, we will fit the model with the LinearRegression classifier. We are trying to predict the Adj Close value of the Standard and Poor’s index. # So the target of the model is the "Adj Close" Column.



<pre class="wp-block-code"><code>lr = LinearRegression()
 
X_train = train&#91;&#91;"High-Low_pct","ewm_5","price_std_5","volume_avg_5","volume Change","volume Close"]]
 
Y_train = train&#91;"Adj Close"]
 
lr.fit(X_train,Y_train) 
 </code></pre>



Create the test features dataset (<code>X_test</code>) which will be used to make the predictions.



<pre class="wp-block-code"><code># Create the test features dataset (X_test) which will be used to make the predictions.

X_test = test&#91;&#91;"High-Low_pct","ewm_5","price_std_5","volume_avg_5","volume Change","volume Close"]].values 

# The labels of the model

Y_test = test&#91;"Adj Close"].values 
</code></pre>



Predict the <code>Adj Close</code> values using  the <code>X_test</code> dataframe and Compute the Mean Squared Error between the predictions and the real observations.



<pre class="wp-block-code"><code>close_predictions = lr.predict(X_test) 

mae = sum(abs(close_predictions - test&#91;"Adj Close"].values)) / test.shape&#91;0]

print(mae)

18.0904
</code></pre>



We have that the Mean Absolute Error of the model is 18.0904. This metric is more intuitive than others such as the Mean Squared Error, in terms of how close the predictions were to the real price.



Finally we will plot the error term for the last 25 days of the test dataset. This allows observing how long is the error term in each of the days, and asses the performance of the model by date. 



<pre class="wp-block-code"><code># Create a dataframe that output the Date, the Actual and the predicted values
df = pd.DataFrame({'Date':dates,'Actual': Y_test, 'Predicted': close_predictions})
df1 = df.tail(25)
 
# set the date with string format for plotting
df1&#91;'Date'] = df1&#91;'Date'].dt.strftime('%Y-%m-%d')
 
df1.set_index('Date',inplace=True)
 
error = df1&#91;'Actual'] - df1&#91;'Predicted']
 
# Plot the error term between the actual and predicted values for the last 25 days
 
error.plot(kind='bar',figsize=(8,6))
plt.grid(which='major', linestyle='-', linewidth='0.5', color='green')
plt.grid(which='minor', linestyle=':', linewidth='0.5', color='black')
plt.xticks(rotation=45)
plt.show()
</code></pre>



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image14.png" alt="Error Term - Multivariate Linear Regression in Python" class="wp-image-26794"/><figcaption>Error Term by date&nbsp;</figcaption></figure></div>



This concludes our example of Multivariate Linear Regression in Python.

Multivariate Linear Regression in Python with scikit-learn Library

In the post, we will learn about how to create a classifier model in machine learning using python. We will create a supervised classifier model that will train a dataset with a set of features and then use test data to predict price direction at day k with information only known at day k-1. Price direction can be up when the closing price at t is higher than the price at t-1, and down when the closing price at t is lower than at t-1.



For this task we create a set of features that are the lagged returns for the previous 2 days and volume percent change in each day.&nbsp; Then we train the dataset and fit different models with a set of algorithms that are the <a href="https://financetrain.com/logistic-regression/" class="rank-math-link">Logistic Regression</a>, <a href="https://financetrain.com/support-vector-machine-algorithm-explained/" class="rank-math-link">Support Vector Machine</a>, Support Vector Classifier, <a href="https://financetrain.com/random-forest-algorithm-in-python/" class="rank-math-link">Random Forest</a> and Linear Discriminant Analysis.



For each of the model we will output two metrics that are used in classification problems to assess model performance. These metrics are the Hit Rate and the Confusion Matrix.&nbsp;



<h3>Hit Rate</h3>



The Hit Rate provides a measure of the percentage of the number of times, the classifier make correct predictions (up and down). This indicator can be expressed with the following formula:



<figure class="wp-block-image size-large"><img src="https://financetrain.com/wp-content/uploads/hit-rate.png" alt="" class="wp-image-26800"/></figure>



<h3>Confusion Matrix</h3>



The Confusion Matrix gives a measure of how many times the classifier predicts up correctly and how many times did predict down correctly.&nbsp;



In a binary classification problem, the confusion matrix is a 2 x 2 contingency table that determine the False Positive Rate (Type 1 error. When incorrectly reject a True null hypothesis. UF in the contingency table) and the False Negative Rate (Type II error. When fail to reject the null hypothesis. DF in the contingency table).



<h3>UT UF DF DT </h3>



UT represents correctly classify up periods, UF represents incorrectly classify up periods (they were classified as down periods), DF represents incorrectly classify down periods (they were classified us up periods), and DT represents correctly classify down periods.



Scikit Learn library provides methods to calculate the Hit Rate and the Confusion Matrix for a classifier. The dataset of the model is the SPY data between 2015-01-01 to 2019-09-18. We will load the dataset into the environment.



<div class="wp-block-file"><a href="https://financetrain.com/wp-content/uploads/spy_data.csv">spy_data</a><a href="https://financetrain.com/wp-content/uploads/spy_data.csv" class="wp-block-file__button" download>Download</a></div>



The python example below contains the following steps:



<ul><li>Step 1: Import libraries, and load SPY data into the environment with the&nbsp; read_csv() function. This is the dataset of the model with dates&nbsp; between 2015-01-01 to 2019-09-18</li><li>Step 2: Create features and target variable with the model_variables() function</li><li>Step 3: Fit different models: Logistic Regression, Random Forest, Support Vector Machine, Linear Discriminant Analysis&nbsp;</li><li>Step 4: Obtain metrics such as Confusion Matrix and Hit Rate for each of the models.</li></ul>



<pre class="wp-block-code"><code>import pandas as pd 
import numpy as np
import datetime
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.metrics import confusion_matrix
from sklearn.svm import LinearSVC, SVC
data = pd.read_csv("C:/Users/Nicolas/Documents/Machine Learning Course/spy_data.csv",index_col='Date')
 
def model_variables(prices,lags):
 '''
 Parameters:
 prices: dataframe with historical data of SPY with the variables closes and 
 volume.
 lags: Number of lags of the closing price that will be created by the function
 to make the lagged returns features of the machine learning model
 Output:
 tsret: dataframe with index date, the independent variables(X) and the 
 dependent variable(y) of the machine learning model
 '''
 
 # Change data types of prices dataframe from object to numeric
 prices = prices.apply(pd.to_numeric)
 # Create the new lagged DataFrame
 inputs = pd.DataFrame(index=prices.index)
 
 inputs["Close"] = prices["Close"]
 inputs["Volume"] = prices["Volume"]
 # Create the shifted lag series of prior trading period close values
 for i in range(0, lags):
 tsret = pd.DataFrame(index=inputs.index)
 inputs["Lag%s" % str(i+1)] = prices["Close"].shift(i+1)
 
 #Create the returns DataFrame
 tsret["VolumeChange"] =inputs["Volume"].pct_change()
 tsret["returns"] = inputs["Close"].pct_change()*100.0
 
 # If any of the values of percentage returns equal zero, set them to
 # a small number (stops issues with QDA model in Scikit-Learn)
 for i,x in enumerate(tsret["returns"]):
 if (abs(x) &lt; 0.0001):
 tsret["returns"][i] = 0.0001
 
 # Create the lagged percentage returns columns
 for i in range(0, lags):
 tsret["Lag%s" % str(i+1)] = \
 inputs["Lag%s" % str(i+1)].pct_change()*100.0
 
 # Create the "Direction" column (+1 or -1) indicating an up/down day
 tsret = tsret.dropna()
 tsret["Direction"] = np.sign(tsret["returns"])
 
 # Convert index to datetime in order to filter the dataframe by dates when 
 # we create the train and test dataset
 tsret.index = pd.to_datetime(tsret.index)
 return tsret

# Pass the dataset(data) and the number of lags 2 as the inputs of the model_variables function
variables_data = model_variables(data,2)
 
# Use the prior two days of returns and the volume change as predictors
# values, with direction as the response
dataset = variables_data[["Lag1","Lag2","VolumeChange","Direction"]]
dataset = dataset.dropna()
 
# Create the dataset with independent variables (X) and dependent variable y
X = dataset[["Lag1","Lag2","VolumeChange"]]
y = dataset["Direction"]

# Split the train and test dataset using the date in the date_split variable
# This will create a train dataset of 4 years data and a test dataset for more than 
# 9 months data.
 
date_split = datetime.datetime(2019,1,1)
 
X_train = X[X.index &lt;= date_split]
X_test = X[X.index > date_split]
y_train = y[y.index &lt;= date_split]
y_test = y[y.index > date_split]
 
# Create the (parametrised) models
print("Hit Rates/Confusion Matrices:\n")
models = [("LR", LogisticRegression()),
 ("LDA", LDA()),
 ("LSVC", LinearSVC()),
 ("RSVM", SVC(
 C=1000000.0, cache_size=200, class_weight=None,
 coef0=0.0, degree=3, gamma=0.0001, kernel='rbf',
 max_iter=-1, probability=False, random_state=None,
 shrinking=True, tol=0.001, verbose=False)
 ),
 ("RF", RandomForestClassifier(
 n_estimators=1000, criterion='gini',
 max_depth=None, min_samples_split=2,
 min_samples_leaf=30, max_features='auto',
 bootstrap=True, oob_score=False, n_jobs=1,
 random_state=None, verbose=0)
 )]
 
 
# Iterate through the models and obtain the accuracy metrix: Hit Rate and Consusion Matrix
for m in models:
 # Train each of the models on the training set
 m[1].fit(X_train, y_train)
 # Make an array of predictions on the test set
 pred = m[1].predict(X_test)
 # Output the hit-rate and the confusion matrix for each model
 
 print("%s:\n%0.3f" % (m[0], m[1].score(X_test, y_test)))
 print("%s\n" % confusion_matrix(pred, y_test))
</code></pre>



The results of the code are the Hit Rate and the Confusion Matrix for each of the models trained. The diagonal of the matrix represent the correct predictions (up and down), and the inverse of the diagonal represents incorrect predictions (the prediction was down and the price go up, or the prediction was up and the price go down).



<pre class="wp-block-code"><code>LR:
0.583
[[28 29]
 [46 77]]

LDA:
0.589
[[28 28]
 [46 78]]

LSVC:
0.583
[[28 29]
 [46 77]]

RSVM:
0.572
[[20 23]
 [54 83]]

RF:
0.600
[[34 32]
 [40 74]]
</code></pre>



The model with the best prediction score is the Random Forest with a Hit Rate of 60%. We have changed the default parameter min_samples_leaf&nbsp; of the Random Forest Classifier&nbsp; from it default value of 1 to 30.&nbsp;



This means that each leaf node has at least 30 observations and that a split will be considered if it leaves at least 30 training samples in each left and right branches.&nbsp;&nbsp;&nbsp;All the models work well to predict down periods respect up periods, as the true positive rate for the “down” days (DTDT+UF) is significantly higher than the true positive rate for the “up” days (UTUT+DF).

Classifier Model in Machine Learning Using Python

Cross validation is a technique used to determine how the results of a machine learning model could be generalized to new, unseen data. The training error associated with a model might underestimate the test error of the model, so the Cross Validation approach provides a mechanism to get the MSE test with the current dataset without the need of finding new data to test the model.



Basically, to perform Cross Validation we need to keep aside a portion of the data that is not used to train the model.&nbsp; The goal of Cross Validation is to estimate the test error of the model, by holding a subset of the dataset in order to use them as test observations. This approach gives a more accurately estimate of the test error.



<h3>Validation Set Approach</h3>



The classical method for training and testing a dataset is called the Validation Set approach. We have used this approach in both examples of <a href="https://financetrain.com/multivariate-linear-regression-in-python-with-scikit-learn-library/" class="rank-math-link">Multivariate linear regression</a> and for the <a href="https://financetrain.com/classifier-model-in-machine-learning-using-python/" class="rank-math-link">Classifier Forecasting</a>.&nbsp; This consists of <a href="https://financetrain.com/train-test-datasets-in-machine-learning/" class="rank-math-link">splitting the dataset into a train and a test set</a>. Commonly around 80% of the data is used to train the dataset and the other 20 % of the data is used as the test set.&nbsp;



The splitting is done in chronological order, where the first two thirds represent the first two thirds of the historical data. One of the drawbacks of this method is that by choosing different length for the train and the test data, the model performance can vary significantly.



Likewise, if we have a limited amount of data, there is a possibility of high bias because we would miss some information about the data that was not used for training. If the amount of data is huge and the train and test data have the same distribution, this approach is acceptable.



The second approach to address overfitting is to train and test the model using the method called K-Fold Cross Validation. 



<h3>K-Fold Cross Validation</h3>



K-Fold Cross Validation is a more sophisticated approach that generally results in a less biased model compared to other methods. This method consists in the following steps:



<ol><li>Divides the n observations of the dataset into k mutually exclusive and equal or close-to-equal sized subsets known as “folds”.&nbsp;</li><li>Fit the model using k-1 folds as the training set and one fold (kth) as the test set. After each iteration has been finished, store the error of the model.</li><li>Repeat this process k times using one different fold every time as a test set and the remaining folds (k-1) as the training set.&nbsp;</li><li>Once all the iterations have finished, take the mean of the k models. This would be the Mean Squared Error of the model.</li></ol>



The error model in using the K-Fold cross validation has the following formula:



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/error-model.png" alt="" class="wp-image-26806"/><figcaption>Error Model Formula</figcaption></figure></div>



An important consideration of this approach is the selection of the number of folds. The choice of the number of folds should be done on the basis that each fold needs to have enough data points to provide a fair estimate of the model performance. On the other hand, the k number should not be so small such as 2, in order to have enough trained models to assess the model performance.



K-Fold Cross Validation method



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image16.png" alt="K-Fold Cross Validation method" class="wp-image-26807"/></figure></div>

Cross Validation to Avoid Overfitting in Machine Learning

In this post, we will provide an example of <a href="https://financetrain.com/cross-validation-to-avoid-overfitting-in-machine-learning/" class="rank-math-link">Cross Validation</a> using the K-Fold method with the python scikit learn library.  The K-Fold Cross Validation example would have k parameters equal to 5. By using a ‘for’ loop, we will fit each model using 4 folds for training data and 1 fold for testing data, and then we will call the accuracy_score method from scikit learn to determine the accuracy of the model.



The example is divided into the following steps:



<ul><li>Step 1: Import the libraries and load into the environment Open, High, Low, Close data for EURUSD</li><li>Step 2: Create features with the create_features() function</li><li>Step 3: Run the model with the Validation Set approach</li><li>Step 4: Run the model with the K-Fold Cross Validation approach</li></ul>



<div class="wp-block-file"><a href="https://financetrain.com/wp-content/uploads/eurusd.csv">eurusd data</a><a href="https://financetrain.com/wp-content/uploads/eurusd.csv" class="wp-block-file__button" download>Download</a></div>



<pre class="wp-block-code"><code>import pandas as pd
from sklearn.model_selection import KFold
import numpy as np
from sklearn.metrics import accuracy_score
from sklearn import tree
import matplotlib.pyplot as plt
import seaborn as sns

# Read csv file to load into the environment OHLC data from EURUSD.
eurusd_ohlc = pd.read_csv("C:/Users/Nicolas/Documents/Machine Learning Course/eurusd.csv")
 
# The create_features function, receives the eurusd_ohlc parameter and create new features to use in a machine learning model
 
def create_features(fx_data):
 '''
 Parameters:
 fx_data: has Open-High-Low-Close data for currency pair EURUSD between 2001-08-21 to 2019-09-21
 Return:
 fx_data: dataframe with original and new data with the features of the model
 target: target variable to predict, which contains the direction of the price. The values can be 1 for up direction and -1 for down direction.
 '''
 
 # Convert all columns of the stock_data data frame to numeric columns
 fx_data = fx_data.apply(pd.to_numeric)
 
 # Reverse the index to have old values at top of the dataframe 
 fx_data = fx_data.sort_values('Date')
 
 # Create features to use in the machine learning model
 fx_data&#91;'High-Low'] = fx_data&#91;'High'] - fx_data&#91;'Low']
 fx_data&#91;'pct_change'] = fx_data&#91;'Close'].pct_change()
 fx_data&#91;'ret_5'] = fx_data&#91;'pct_change'].rolling(5).mean()
 
 # Calculate RSI Indicator
 close = fx_data&#91;'Close']
 # Get the difference in price from previous step
 delta = close.diff()
 # Get rid of the first row, which is Nan since it did not have a previous 
 # row to calculate the differences
 delta = delta&#91;1:] 
 
 # Make the positive gains (up) and negative gains (down) Series
 up, down = delta.copy(), delta.copy()
 up&#91;up &lt; 0] = 0
 down&#91;down > 0] = 0
 
 # Calculate the EWMA
 roll_up = up.rolling(center=False,window=14).mean()
 roll_down = abs(down).rolling(center=False,window=14).mean()
 
 # Calculate the RSI based on EWMA
 RS = roll_up / roll_down
 RSI = 100.0 - (100.0 / (1.0 + RS))
 
 fx_data&#91;'RSI'] = RSI
 fx_data.dropna(inplace=True)
 # Create the target variable that take the values of 1 if the stock price go up or -1 if the stock price go down
 target = np.where(fx_data&#91;'Close'].shift(-1) > fx_data&#91;'Close'], 1, -1)
 
 return fx_data, target
 
features, target = create_features(eurusd_ohlc)
 
# Validation Set approach : take 80% of the data as the training set and 20 % as the test set. X is a dataframe with the input variable
X = features&#91;&#91;'High-Low', 'pct_change', 'ret_5','RSI']]
 
# Y is the target or output variable 
y = target
 
length_to_split = int(len(features) * 0.8)
# Splitting the X and y into train and test datasets
X_train, X_test = X&#91;:length_to_split], X&#91;length_to_split:]
y_train, y_test = y&#91;:length_to_split], y&#91;length_to_split:]
 
# Print the size of the train and test dataset
print(X_train.shape, X_test.shape)
print(y_train.shape, y_test.shape)
 
clf = tree.DecisionTreeClassifier(random_state=20)
 
# Create the model on train dataset
model = clf.fit(X_train, y_train)
 
# Calculate the accuracy
print(accuracy_score(y_test, model.predict(X_test), normalize=True)*100)
 
# KFold Cross Validation approach
kf = KFold(n_splits=5,shuffle=False)
kf.split(X) 
 
# Initialize the accuracy of the models to blank list. The accuracy of each model will be appended to this list
accuracy_model = &#91;]
 
# Iterate over each train-test split
for train_index, test_index in kf.split(X):
 # Split train-test
 X_train, X_test = X.iloc&#91;train_index], X.iloc&#91;test_index]
 y_train, y_test = y&#91;train_index], y&#91;test_index]
 # Train the model
 model = clf.fit(X_train, y_train)
 # Append to accuracy_model the accuracy of the model
 accuracy_model.append(accuracy_score(y_test, model.predict(X_test), normalize=True)*100)

# Print the accuracy 
print(accuracy_model)

(3989, 4) (998, 4)
(3989,) (998,)
51.4028
&#91;50.501, 52.004, 48.9468, 46.1384, 51.3541]
</code></pre>



These 4 lines above are the outputs of the <code>print() </code>messages. (3989, 4) (998, 4) are the size of the X_train and X_test dataset where 3989 is the number of observations in the train dataset and 4 is the number of features in the train dataset. 998 is the number of observations in the test dataset, and 4 is the number of features in the test dataset.



(3989,) (998,) are the size of y_train and y_test. 51.4028 is the accuracy score with the Validation set approach and [50.501, 52.004, 48.9468, 46.1384, 51.3541] is the <code>accuracy_model </code>list which show the accuracy in each iteration using the K-Fold Cross Validation method.



K-Fold Cross Validation gives a better idea of how the model will perform with new or live data, because we have used 5 different testing sets to obtain measures of the model performance. 



Finally we use a bar plot to visualize the score measure in each iteration:



<pre class="wp-block-code"><code>### Visualize accuracy for each iteration
 
scores = pd.DataFrame(accuracy_model,columns=&#91;'Scores'])
 
sns.set(style="white", rc={"lines.linewidth": 3})
sns.barplot(x=&#91;'Iter1','Iter2','Iter3','Iter4','Iter5'],y="Scores",data=scores)
plt.show()
sns.set()
</code></pre>



K-Fold Cross Validation Scores 



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image17.png" alt="K-Fold Cross Validation Scores" class="wp-image-26812"/></figure></div>

K-Fold Cross Validation Example Using Python scikit-learn

Unsupervised learning models are composed of features that are not associated with a response. This means that this type of machine learning algorithms do not have labelled data as their interest lies in the attributes of the features themselves. 



In unsupervised learning models there is no concept of training or supervising a dataset&nbsp; as the independent variables or features (x1,x2,x3,..,xn)) are not paired with a response (y).&nbsp; The goal of these problems is to model the underlying structure or distribution of the data to learn more about it.



Many machine learning problems use unsupervised techniques because sometimes it can be very expensive or time consuming to label all data or there can be cases where the data possesses high dimensionality that is not suitable for supervised learning techniques because it could lead to overfitting and have poor test performance.



Some examples of unsupervised learning problems in <a href="https://financetrain.com/quantitative-finance-the-demystification-of-stochastic-calculus/" class="rank-math-link">quantitative finance</a> are the following:



<ul><li>Portfolio/asset clustering&nbsp;</li><li>Market regime identification&nbsp;</li><li>Trading signal generation with natural language processing</li></ul>



There are two important techniques for unsupervised learning models. They are Dimensionality Reduction and Cluster.



<h2>Dimensionality Reduction</h2>



Some machine learning problems have a large number of features that are correlated in a high dimensional space. Dimensionality reduction is the process of reducing the number of features under consideration, by obtaining a set of principal features.



The most common method for Dimensionality Reduction is the Principal Component Analysis (PCA). This method reduces the features/dimensions of the data by transforming their original space into a set of linearly, uncorrelated variables called principal components. 



The principal components can be founded as the eigenvectors of the covariance matrix of the data. This procedure tries to preserve the components of the data with the higher variation and remove components with fewer variations. (The Eigenvectors represent the key information of a matrix. These are used when the features/dimensions of the data are high and it is necessary to reduce the dimension to manage the dataset.)



In quantitative finance, this technique could be applied to a large number of correlated stocks in order to reduce their dimensionality by looking at a smaller set of uncorrelated factors.



<h2>Cluster Analysis</h2>



Another important unsupervised learning technique is Cluster Analysis. Its goal is to assign a cluster label for each observation of a dataset in order to group the dataset in clusters with similar properties. In quantitative finance clustering is used to identify assets with similar characteristics which become useful to construct diversified portfolios.



A key element of the cluster technique is to define the number of clusters K in which the data will be partitioned.&nbsp; Scikit learn library provides a straightforward procedure to perform this task using the K-means algorithm. First we describe the inner steps of the algorithm and then we use and example. The algorithm performs the following steps:



<ol><li>Pick k clusters centers (centroids) randomly. The number of clusters is given by the k parameter</li><li>Assign each observation of the dataset to the nearest cluster by using the Euclidean Distance between the observation and each centroid.</li><li>Find the new centroid or new cluster mean corresponding to each observation by taking the average of all the points assigned to that cluster.</li><li>Repeat the procedure 2 and 3 until none of the cluster assignment change. This means until the clusters remain stable.</li></ol>



This algorithm can be used for more than one purpose. One good reason to use this algorithm is to identify different clusters of stocks that can support a portfolio diversification strategy among stocks in different clusters.

Unsupervised Learning Models

In this post, we will provide an example of the implementation of the K-Means algorithm in python.&nbsp; This K-Means algorithm python example consists of clustering a dataset that contains information of all the stocks that compose the Standard &amp; Poor Index.&nbsp;



This example contains the following five steps:



<ul><li>Obtain the 500 tickers for the SPY &amp; 500 by scrapping the tickers symbols from Wikipedia. The function obtain_parse_wike_snp500() conduct this task.</li><li>Obtain closes prices from last year for each of the symbols using the Quandl API</li><li>Calculate mean and variance of the returns for each stock</li><li>Choose the best k value for the cluster the dataset</li><li>Fit the model with the k number of cluster</li></ul>



<pre class="wp-block-code"><code>import pandas as pd 
import numpy as np
from math import ceil
import bs4
import requests
import quandl # need to do pip install quandl
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
 

def obtain_parse_wiki_snp500(): 
 """ Download and parse the Wikipedia list of S&amp;P500 constituents using requests and Beautiful Soup.
 """ 
 
 response = requests.get( "http://en.wikipedia.org/wiki/List_of_S%26P_500_companies" ) 
 
soup = bs4.BeautifulSoup(response.text)
 
 # This selects the first table, using CSS Selector syntax and then ignores the header row ([1:]) 
 symbolslist = soup.select('table')[0].select('tr')[1:]
 
 # Obtain the symbol information for each row in the S&amp;P500 constituent table 
 symbols = [] 
 for i, symbol in enumerate(symbolslist): 
 tds = symbol.select('td')
 symbols.append( (tds[0].select('a')[0].text, # Ticker 
 tds[1].select('a')[0].text, # Name
 tds[3].text # Sector 
 )
 ) 
 
 return symbols
 
tickers = obtain_parse_wiki_snp500()
</code></pre>



The tickers object is a list of tuples with the ticker, company name and sector of each company. To observe the structure of the tickers object we show the first 10 elements.



<pre class="wp-block-code"><code>tickers[:10]

[('MMM', '3M Company', 'Industrials'),
 ('ABT', 'Abbott Laboratories', 'Health Care'),
 ('ABBV', 'AbbVie Inc.', 'Health Care'),
 ('ABMD', 'ABIOMED Inc', 'Health Care'),
 ('ACN', 'Accenture plc', 'Information Technology'),
 ('ATVI', 'Activision Blizzard', 'Communication Services'),
 ('ADBE', 'Adobe Systems Inc', 'Information Technology'),
 ('AMD', 'Advanced Micro Devices Inc', 'Information Technology'),
 ('AAP', 'Advance Auto Parts', 'Consumer Discretionary'),
 ('AES', 'AES Corp', 'Utilities')]
</code></pre>



We will use this list in the following function to loop over each symbol of tickers and get the close price from the Quandl API. We perform the loop using a try-except block to handle errors regarding symbols that were changed or are different in the Quandl database.



<pre class="wp-block-code"><code>def get_quandl_data(symbols):
 """
 This function would loop over all the symbols from the SPY &amp; 500 and retrieve the Close price column from the WIKI database from Quandl between the dates given by the start_date and end_date parameters
 """
 
 symbols = [symbol[0] for symbol in symbols]
 stocks_info = []
 
 for symbol in symbols:
 try:
 stockdata = quandl.get("WIKI/{}".format(symbol), start_date='2018-09-20', end_date='2019-09-20',column_index=4)
 print('Downloading data from symbol {}'.format(symbol))
 stocks_info.append((symbol,stockdata.iloc[:,0]))
 except:
 print('The stock symbol {} is not on quandl database'.format(symbol))
 
 return stocks_info
 
data = get_quandl_data(tickers)
</code></pre>



The data object is a list of tuples where each tuple has 2 elements. The first element is the ticker and the second element is the Date and Close Price for each of the stocks. We need to parse this information in order to make a dataframe that has the date as index and the close prices for each stock as columns. The following lines will do this job:



<pre class="wp-block-code"><code># Get only the symbols from the data object
symbols = [data[i] for i in range(0,len(data))]
# Get the date and prices from the data object (second element of the tuple)
closes = [data[i][1] for i in range(0,len(data))]
 
# Store closes object in a dataframe and obtain the transpose of the dataframe. #With this we will have the Date as index and the Prices of each stock as columns.
closes = pd.DataFrame(closes).T
# Rename column names with the symbols list
closes.columns = symbols
#Calculating annual mean returns and standard deviation of the returns
returns = closes.pct_change().mean() * 252
std = closes.pct_change().std() * np.sqrt(252)
 
#Concatenating the returns and variances into a single data-frame
ret_var = pd.concat([returns, std], axis = 1).dropna()
ret_var.columns = ["Returns","Standard Deviation"]
</code></pre>



The <code>ret_var</code> dataframe has the following structure:



<pre class="wp-block-code"><code>ret_var.head(10)

 Returns Standard Deviation
MMM 0.301229 0.122294
ABT 0.392332 0.138839
ABBV 0.457778 0.179314
ABMD 0.545361 0.242094
ACN 0.284499 0.131696
ATVI 0.596721 0.305587
ADBE 0.554549 0.216873
AMD 0.069001 0.585976
AAP -0.460970 0.394637
AES -0.052332 0.210608
</code></pre>



In order to determine the optimal number of clusters k for the ret_var dataset, we will fit different models of the K-means algorithm while varying the k parameter in the range 2 to 14. For each model we calculate the Sum Squared Error (SSE) by using the inertia_ method of the model fitted. In each iteration we append the inertia to the sse list. Then we take the model with the less value of SSE. (Inertia tells how far away the points within a cluster are. The small the inertia value is better.)



<pre class="wp-block-code"><code>#Converting ret_var into numpy array
X = ret_var.values 
sse = []
for k in range(2,15):
 kmeans = KMeans(n_clusters = k)
 kmeans.fit(X)
 
 #SSE for each n_clusters
 sse.append(kmeans.inertia_) 
 
plt.plot(range(2,15), sse)
plt.title("Elbow Curve")
plt.show()
</code></pre>



The graph is names as Elbow Curve, and shows that the optimal value of k is 5.



Elbow Curve: Determine the optimal value of k. K-means Cluster



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image19.png" alt="" class="wp-image-26819"/></figure></div>



We chose k=5 and fit the model with this parameter-value.



<pre class="wp-block-code"><code>kmeans = KMeans(n_clusters = 5).fit(X)
centroids = kmeans.cluster_centers_
plt.scatter(X[:,0],X[:,1], c = kmeans.labels_, cmap ="rainbow")
plt.show()
</code></pre>



The different groups or cluster of the dataset are reflected in the following graph:



Clusters of the ret_var dataset. k=5



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image18.png" alt="" class="wp-image-26820"/></figure></div>



We can view the presence of outliers as only one point is on the upper right side of the graph. This outlier form its own cluster. In order to have a better categorization of the stocks within the SPY index, we would remove those stocks and fit the model another time.



<pre class="wp-block-code"><code># Find the stock with the highest value in the Standard Deviation variable 
stdOrder = ret_var.sort_values('Standard Deviation',ascending=False)
first_symbol = stdOrder.index[0]
 
# Drop the columns with the outliers values
ret_var.drop(first_symbol,inplace=True)
# Fit the model without the outliers
X = ret_var.values
kmeans =KMeans(n_clusters = 5).fit(X)
centroids = kmeans.cluster_centers_
plt.scatter(X[:,0],X[:,1], c = kmeans.labels_, cmap ="rainbow")
plt.show()
</code></pre>



Clusters of ret_var dataset without Outliers. k=5



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image21.png" alt="K-Means Algorithm Python" class="wp-image-26821"/></figure></div>



The x axis of the Figure 17, refers to the returns of the stocks and the y axis is the standard deviation of each stock. So the stocks that are in the upper-right cluster are the stocks with the higher value of returns and standard deviation.



Finally we will assign to each stock it correspondent number of cluster(1,2,3,4,and 5) and make a dataframe with this information. Having the information of cluster number for each stock, we can create a diversified portfolio in the long term, between stocks from different clusters.



<pre class="wp-block-code"><code>stocks = pd.DataFrame(ret_var.index) # the dataframe structure allow concatenation
cluster_labels = pd.DataFrame(kmeans.labels_)
stockClusters = pd.concat([stocks, cluster_labels],axis = 1)
stockClusters.columns = ['Symbol','Cluster']
</code></pre>



The structure of the stockClusters dataframe is the following:



<figure class="wp-block-image size-large"><img src="https://financetrain.com/wp-content/uploads/image20.png" alt="" class="wp-image-26822"/></figure>



These are the first rows of the stockClusters dataframe. We conclude this section with a categorization of each stock from the SPY &amp; 500 in terms of returns and risk. This could be an important tool for portfolio diversification. This concludes our K-Means algorithm python example.

K-Means Algorithm Python Example

Neural networks is an advanced technique which is within the field of Deep Learning. As we know, machine learning involves working with algorithms that try to predict a target variable or segment data to find relevant patterns without human intervention.



In contrast, in deep learning architecture there is more than one layer of these algorithms and these systems represent a network of algorithms that do not necessarily need labeled data to make predictions.&nbsp; Below we can visualize the main components of a neural network. 



Neural Network Architecture



<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://financetrain.com/wp-content/uploads/image22.png" alt="Neural Networks Architecture" class="wp-image-26828"/></figure></div>



A neural network is composed of input layers, hidden layers and output layers. The input layers are the beginning of the workflow in a neural network and receive information about the initial data or the features of a dataset.



Hidden layers identify the important information from the inputs, leaving out the redundant information. They convert the network more efficiently and pass the important patterns of data to the next layer which is the output layer.



In the middle of the hidden layer and the output layer, the network has an activation function. This activation function transforms the inputs in the desired output and can take many forms such as a sigmoid function (as in the binary classification of logistic regression) or other types of functions (RELU, Tanh, softmax among others that are beyond the scope of this post).



The goal of the activation function is to capture non-linear relationships between the inputs and on the other hand it helps to convert the inputs into more useful output.&nbsp;&nbsp; 



The figure above shows a network architecture called “feed-forward network” as the inputs’ signals flow in only one direction (from inputs to outputs). There are other types of flows such as back propagations, where the data goes back along the neural networks and inspect every connection to check how the output would behave according to a change in the weight. The key of the model is to find the optimal weights values of W that minimize the prediction error. Just like other machine learning algorithms, the neural networks also have a loss function that will be minimized.&nbsp;



One common optimization algorithm to minimize the loss function that is used in neural networks is called “gradient descent” that works on multiple iterations trying to find the weights that returns the less value of the error term.&nbsp;



The inner steps of gradient descent algorithm are the following:



<ol><li>Assign random values for the weights and pass them as parameters of the function</li><li>Calculate the change in Sum Squared Error (SSE) when the weights are changed by small amounts from their original random values</li><li>Adjust the weights to reduce the impact of the prediction errors</li><li>Use the new weights for predictions and to calculate the new SSE</li><li>Repeat steps 2 and 3 until the adjustments in weights don’t significantly reduce the error term.</li></ol>



There are some important advantages of neural networks that are pointed below:



<ul><li>They are suitable to represent and deal with not-linear relationships in the data.</li><li>These models are better choices to avoid overfitting on the model. After learning from their initial inputs and the relationship of the factors, they are able to generalize and predict on unseen data.</li><li>They do not impose restrictions on the input variables such as some distribution pattern of the data.</li><li>They are able to model heteroscedasticity data with high volatility and non-constant variance. This represents a breakthrough in the financial time series prediction, where the data has high and time dependent variance. Heteroscedasticity means that the variability or variance of the time series is not constant over time. The opposite is called homoscedasticity.&nbsp;</li></ul>



Due to their ability to model complex relationships and understand underlying factors of the data, these models are a good tool to forecast stock prices. Recent advances in the field are the usage of Long Short Term Memory (LSTM) models and Recurrent Neural Network for forecasting.&nbsp;

Neural Networks Overview

Machine Learning in Finance Using Python

This frontmatter description will appear below the title

Machine Learning in Finance Using Python

Lessons

Free Guides - Getting Started with R and Python

Accelerate your finance career with cutting-edge data skills.