Train-Test Datasets in Machine Learning

Splitting the data into training and testing set is required for Supervised Learning problems. Unsupervised Learning models don’t require a train and a test dataset. 

Classification and Regression problems would supervise or “train” a model with specific data in order to provide predictions of the target variable y. The process of training a dataset is conducted by choosing the set of relevant features or independent variables and combining these with a response y (labelled data) that is the observed value of the target variable.

In this phase, the algorithm is trained on the data and will determine the influence of each feature on the response y. Finally, we can make predictions for out-of-sample or unseen data based on the prior training experience. 

This process has two main stages that are called training and testing the model. In the training phase as we described above, we fit the model with the data and afterwards we use the test data to assess the model performance. 

The training dataset includes both features and the target variable while the testing dataset includes only features that are used to run the model and get the predictions of the target variable. The training dataset usually represent 70%-80% of the total data and the test dataset is the remaining portion of the data which is preserved to test the model accuracy.