Although, there are numerous algorithms that one can select from (such as regression, naive bayes, support vector machine, ensemble etc.,) to solve classification problems. In this example, deep neural networks were used to solve a classification problem. At the end of this example, you will understand the reason why we selected deep neural networks to solve the problem. Extent studies suggest that neural network models perform better than standard machine learning models. Neural networks are algorithms that help computers learn by means of mimicking biological neural activities of animal brains. Neuroscience suggests that animal brains consist of neurons, which are nodes that receive and transmit information to several organs. Neurons can be connected to many other. Consequently, resulting in greater complexity.
Neural networks have been applied successfully in image and speech recognition, including adaptive learning. On our case, we used neural networks for predictive modeling. Deep Neural Networks were trained and all their weights and thresholds of set of random values, then measured and controlled for change using an activation function.
The following algorithms were used to solve the same classification problem.
- Bernoulli Restricted Boltzmann Machine.
- Multi Layer Perceptron neural network.
Scikit-learn and Keras libraries were used to develop structures of neural networks.
Import Python Libraries
Underneath, key Python libraries were imported. These libraries gallantly helped us create arrays and profile tables, compute graphs and make accurate predictions.
import numpy as np
import seaborn as sns
import pandas as pd
sns.set("talk","ticks",font_scale=1,font="sans-serif",color_codes=True)
import matplotlib.pyplot as plt
%matplotlib inline
import scipy.stats as stats
import statsmodels.api as sm
from pylab import rcParams
plt.rcParams["figure.figsize"] = [10,10]
from sklearn import metrics
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split, learning_curve, GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import BernoulliRBM
from sklearn.pipeline import PipelineImport data
df = pd.read_csv(r"C:\Users\Tshepo\Desktop\MLAgortihms\Datasets\diabetes.csv")
df.head()We called the pd.read_csv() function to load obtained csv file into a pandas data frame. See underneath table to examine the data structure.
| Pregnancies | Glucose | BloodPressure | SkinThickness | Insulin | BMI | DiabetesPedigreeFunction | Age | Outcome | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 6 | 148 | 72 | 35 | 0 | 33.6 | 0.627 | 50 | 1 |
| 1 | 1 | 85 | 66 | 29 | 0 | 26.6 | 0.351 | 31 | 0 |
| 2 | 8 | 183 | 64 | 0 | 0 | 23.3 | 0.672 | 32 | 1 |
| 3 | 1 | 89 | 66 | 23 | 94 | 28.1 | 0.167 | 21 | 0 |
| 4 | 0 | 137 | 40 | 35 | 168 | 43.1 | 2.288 | 33 | 1 |
Correlation Matrix
When dealing with two or more variables, one is frequently concerned with correlation between those variables. The rule of thumb is that correlation coefficients should lie between -1 and 1.
In the correlation matrix above, correlation coefficients range from -1 to +1. There are more postive correlation coefficients than negative ones.
Based on the figure above. There are variables that are not correlated.
Covariance Matrix
Pair plot
Given that we have a vast number of variables. We opted for creating pair-plot that visually represents correlation among variables.
sns.pairplot(df)
Although a pair-plot helped us to visually express correlation among variables. Not much insight could be gained from it. To gain more insight on the phenomenon at hand, we will develop a logit model using MLE and create a profile table that shows the significance of each independent variable with the dependent variable.
Develop Logit model using MLE method
The code underneath was used to develop a logit model using MLE method. It is important note that we included a constant to the logit model. As statsmodels library does not presume that there is a constant in a model.
x = df[["Pregnancies","Glucose","BloodPressure","SkinThickness","Insulin","BMI","DiabetesPedigreeFunction","Age"]]
y = df.iloc[::,-1]
x_constant = sm.add_constant(x)
model = sm.Logit(y,x_constant).fit()
model.predict(x_constant)
model.summary()| Dep. Variable: | Outcome | No. Observations: | 768 |
|---|---|---|---|
| Model: | Logit | Df Residuals: | 759 |
| Method: | MLE | Df Model: | 8 |
| Date: | Sun, 21 Jun 2020 | Pseudo R-squ.: | 0.2718 |
| Time: | 17:59:38 | Log-Likelihood: | -361.72 |
| converged: | True | LL-Null: | -496.74 |
| Covariance Type: | nonrobust | LLR p-value: | 9.652e-54 |
| coef | std err | z | P>|z| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| const | -8.4047 | 0.717 | -11.728 | 0.000 | -9.809 | -7.000 |
| Pregnancies | 0.1232 | 0.032 | 3.840 | 0.000 | 0.060 | 0.186 |
| Glucose | 0.0352 | 0.004 | 9.481 | 0.000 | 0.028 | 0.042 |
| BloodPressure | -0.0133 | 0.005 | -2.540 | 0.011 | -0.024 | -0.003 |
| SkinThickness | 0.0006 | 0.007 | 0.090 | 0.929 | -0.013 | 0.014 |
| Insulin | -0.0012 | 0.001 | -1.322 | 0.186 | -0.003 | 0.001 |
| BMI | 0.0897 | 0.015 | 5.945 | 0.000 | 0.060 | 0.119 |
| DiabetesPedigreeFunction | 0.9452 | 0.299 | 3.160 | 0.002 | 0.359 | 1.531 |
| Age | 0.0149 | 0.009 | 1.593 | 0.111 | -0.003 | 0.033 |
There are 768 observations of our data.
SkinThickness, Insulin and Age are not good predictors of prima diabetes outcomes. These variables all have p-values greater than 0.05, we fail to reject hypotheses of these variables. There is no significant difference between these variables and prima diabetes outcomes.
Reducing variables
To reduce bias off our algorithms, we removed those variables from x array.
x = df[["Pregnancies","Glucose","BloodPressure","BMI","DiabetesPedigreeFunction"]]Split data into training, validation and test data
After finalizing our data. We splited data into training, validation and test data using train_test_split() in scikit-learn.
x_train, x_test, y_train, y_test = train_test_split(x,y,test_size=0.2,random_state=0)
x_train, x_val, y_train, y_val = train_test_split(x_train,y_train,test_size=0.1,random_state=0)Normalize data
We used the StandardScaler() function to normalize our data. This function transforms data in such a way that the mean value is 0 and the standard deviation is 1
scaler = StandardScaler()
x_train = scaler.fit_transform(x_train)
x_test = scaler.transform(x_test)Training data using Logistic Regression algorithm
It is against the background of this example to interpret results of the logistic regression model. We only trained data with logistic regression to create a pipeline for the Bernoulli Restricted Boltzmann neural network. Bare in mind, we are not going to tune any hyper-parameters. We will use default parameters on the scikit-learn library.
logreg = LogisticRegression()
logreg.fit(x_train,y_train)Create pipeline and train data using Bernoulli Restricted Boltzmann Machine algorithm
The Bernoulli Restricted Boltzmann Machine a shallow neural network that consist of two layers whereby one layer is visible and another is not visible. This model helps determine actual connection of variables.
To make predictions using the Bernoulli Restricted Boltzmann Machine algorithm, one ought to develop a pipeline.
RBM = BernoulliRBM()
classifier = Pipeline(steps=[("rbm",RBM),("logreg", logreg)])
classifier.fit(x_train,y_train)Actual values vs predicted values
The table underneath compares actual diabetes outcomes and diabetes outcome predicted Bernoulli Restricted Boltzmann Machine model.
y_predRBM = classifier.predict(x_test)
pd.DataFrame({"Actual":y_test, "Predicted":y_predRBM})| Actual | Predicted | |
|---|---|---|
| 661 | 1 | 1 |
| 122 | 0 | 0 |
| 113 | 0 | 0 |
| 14 | 1 | 1 |
| 529 | 0 | 0 |
| … | … | … |
| 476 | 1 | 0 |
| 482 | 0 | 0 |
| 230 | 1 | 1 |
| 527 | 0 | 0 |
| 380 | 0 | 0 |
154 rows × 2 columns
Classification report
Underneath is a high-level overview of the performance of the Restricted Boltzmann Machine model.
classificationRBM = pd.DataFrame(metrics.classification_report(y_test, y_predRBM, output_dict=True)).transpose()
classificationRBM| precision | recall | f1-score | support | |
|---|---|---|---|---|
| 0 | 0.897727 | 0.738318 | 0.810256 | 107.00000 |
| 1 | 0.575758 | 0.808511 | 0.672566 | 47.00000 |
| accuracy | 0.759740 | 0.759740 | 0.759740 | 0.75974 |
| macro avg | 0.736742 | 0.773414 | 0.741411 | 154.00000 |
| weighted avg | 0.799464 | 0.759740 | 0.768234 | 154.00000 |
The Restricted Boltzmann Machine model is accurate 75% of the time.
Confusion matrix
The classification report underneath gives a detail of how the Bernoulli Restricted Boltzmann Machine model performed.
| Predicted: No | Predicted: Yes | |
|---|---|---|
| Actual: No | 79 | 28 |
| Actual: Yes | 9 | 38 |
The Bernoulli Restricted Boltzmann Machine model was made up of a total of 154 predictions. Out of the 154 predictions, the classifier predicted that a patient is diabetic 28 times. In reality, 9 patients were diabetic, while 79 are not diabetic.
ROC Curve
The ROC curve underneath summarize trade-off between the true positive rate and the false positive rate of the Bernoulli Restricted Boltzmann Machine using different probability thresholds.
y_predlogreg_probaRBM = classifier.predict_proba(x_test)[::,1]
fprRBM, tprRBM, _ = metrics.roc_curve(y_test,y_predlogreg_probaRBM)
aucRBM = metrics.roc_auc_score(y_test,y_predlogreg_probaRBM)
plt.plot(fprRBM, tprRBM,label="Bernoulli Restricted Boltzmann Machine, auc: " + str(aucRBM),color="gray",alpha=0.8)
plt.plot([0,1], [0,1],color="red",alpha=0.8)
plt.xlim([0.00,1.01])
plt.ylim([0.00,1.01])
plt.title("Validation ROC Curve - Bernoulli Restricted Boltzmann Machine")
plt.xlabel("Specificity")
plt.ylabel("Sensitivity")
plt.legend(loc=4)
plt.show()
The ROC curve of the Bernoulli Restricted Boltzmann Machine model follows the left-hand border. An accurate model should have an auc of 0.84.
Precision-Recall Curve
The precision-recall curve underneath shows the tradeoff of the Bernoulli Restricted Boltzmann Machine model between precision and recall for different threshold.
The average precision score of the Bernoulli Restricted Boltzmann Machine model is 0.52.
Multi layer Perception Neural Network
Multi Layer Perceptron neural networks consist of multiple layers. There should be at least three layers, namely 1) input layer 2) hidden layer and 3 output layer.
Underneath we finalized the model. Please bare in mind that we used default parameters for the Multi layer Perception neural network model .
MLP = MLPClassifier()
MLP.fit(x_train,y_train)Actual values vs predicted
The table underneath compares actual diabetes outcomes and diabetes outcome predicted by the Multi layer Perception neural network model.
y_predMLP = classifier.predict(x_test)
pd.DataFrame({"Actual":y_test, "Predicted":y_predMLP})| Actual | Predicted | |
|---|---|---|
| 661 | 1 | 1 |
| 122 | 0 | 0 |
| 113 | 0 | 0 |
| 14 | 1 | 1 |
| 529 | 0 | 0 |
| … | … | … |
| 476 | 1 | 0 |
| 482 | 0 | 0 |
| 230 | 1 | 1 |
| 527 | 0 | 0 |
| 380 | 0 | 0 |
154 rows × 2 columns
Classification report
Underneath is an overview of the performance of the Multi Layer Perceptron neural network model in relation to accuracy, precision and recall.
classificationMLP = pd.DataFrame(metrics.classification_report(y_test, y_predMLP, output_dict=True)).transpose()
classificationMLP| precision | recall | f1-score | support | |
|---|---|---|---|---|
| 0 | 0.897727 | 0.738318 | 0.810256 | 107.00000 |
| 1 | 0.575758 | 0.808511 | 0.672566 | 47.00000 |
| accuracy | 0.759740 | 0.759740 | 0.759740 | 0.75974 |
| macro avg | 0.736742 | 0.773414 | 0.741411 | 154.00000 |
| weighted avg | 0.799464 | 0.759740 | 0.768234 | 154.00000 |
The Multi Layer Perceptron neural network model is accurate 75% of the time.
Confusion matrix
The confusion matrix underneath gives a detail of how the Multi Layer Perceptron neural network model performed.
cmatMLP = pd.DataFrame(metrics.confusion_matrix(y_test,y_predMLP), columns = ["Predicted: No", "Predicted: Yes"], index = ["Actual: No", "Actual: Yes"])
cmatMLP| Predicted: No | Predicted: Yes | |
|---|---|---|
| Actual: No | 79 | 28 |
| Actual: Yes | 9 | 38 |
The Multi Layer Perceptron neural network model was made up of a total of 154 predictions. Out of the 154 predictions, the classifier predicted that a patient is diabetic 28 times. In reality, 9 patients were diabetic, while 79 are not diabetic.
ROC Curve
The ROC curve underneath summarize trade-off between the true positive rate and the false positive rate of the Multi Layer Perceptron neural network model using different probability thresholds.
y_predlogreg_probaMLP = classifier.predict_proba(x_test)[::,1]
fprMLP, tprMLP, _ = metrics.roc_curve(y_test,y_predlogreg_probaMLP)
aucMLP = metrics.roc_auc_score(y_test,y_predlogreg_probaMLP)
plt.plot(fprMLP, tprMLP,label="Multi layer Perception Neural Network, auc: " + str(aucMLP),color="black",alpha=0.8)
plt.plot([0,1], [0,1],color="red",alpha=0.8)
plt.xlim([0.00,1.01])
plt.ylim([0.00,1.01])
plt.title("Validation ROC Curve - Multi layer Perception Neural Network")
plt.xlabel("Specificity")
plt.ylabel("Sensitivity")
plt.legend(loc=4)
plt.show()
The ROC curve of the Multi Layer Perceptron neural network model follows the left-hand border. An accurate model should have an auc of 0.84.
Precision-Recall Curve
The precision-recall curve underneath shows the tradeoff of the Multi Layer Perceptron neural network model between precision and recall for different threshold.
precisionMLP, recallMLP, thresholdMLP = metrics.precision_recall_curve(y_test,y_predMLP)
apsMLP = metrics.average_precision_score(y_test,y_predMLP)
plt.plot(precisionMLP, recallMLP,label="Multi layer Perception Neural Network , aps: " +str(apsMLP),color="black",alpha=0.8)
plt.axhline(y=0.5,color="red",alpha=0.8)
plt.xlabel("Precision")
plt.ylabel("Recall")
plt.legend(loc=4)
plt.title("Precision-Recall Curve - Multi layer Perception Neural Network")
plt.show()
The average precision score of the Multi Layer Perceptron neural network model is 0.52.
Learning Curve
trainsize, trainscore, testscore = learning_curve(MLPClassifier(),x,y,cv=10, n_jobs=-1, scoring="accuracy", train_sizes=np.linspace(0.1,1.0,50))
trainscore_mean = np.mean(trainscore,axis=1)
trainscore_std = np.std(trainscore,axis=1)
testscore_mean = np.mean(testscore,axis=1)
testscore_std = np.std(testscore,axis=1)
plt.plot(trainsize,trainscore_mean,color="red", alpha=0.8, label="Training mean score")
plt.plot(trainsize,testscore_mean,color="black", alpha=0.8, label="Cross-Valdiation mean score")
plt.legend(loc=4)
plt.title("Learning Curve - Multi layer Perception Neural Network")
plt.xlabel("Training set size")
plt.ylabel("Accuracy")
plt.show()We used the learning_curve() function to generate values for plotting the learning curve (number of samples used and mean of the training set and cross-validation set).
The training score of the logistic classifier was greater than the cross-validation score. There is a bit of convergence at some sets. Adding more training samples will likely increase generalization.
Deep Neural Network (Model 1)
Underneath we imported keras library.
from keras.models import Sequential
from keras.wrappers.scikit_learn import KerasClassifier
from keras.layers import Dense, Dropout
from keras import regularizersUsing TensorFlow backend.
Develop structure of neural network
Underneath we constructed the architecture of the deep neural network. We used reLu activation on the input layer and hidden layer and sigmoid function to control changes. We used binary cross entropy to measure loss, Adam optimizer to improve accuracy of the neural network.
def create_dnn1():
model1 = Sequential()
model1.add(Dense(11, input_dim=5, activation="sigmoid"))
model1.add(Dense(1, activation="sigmoid"))
model1.compile(loss="binary_crossentropy",optimizer="adam", metrics=["accuracy"])
return model1model1 = KerasClassifier(build_fn=create_dnn1)Hyper-parameter tuning
We created a list of numeric values that the GridSearchCV function could choose from and the best possible parameters we can tune the model with. This function helped us determine the batch size and epochs that must be configured to get optimal model performance.
batch_size = [32,64,128]
epochs = [15,30,60]
param_grid1 = {"batch_size":batch_size, "epochs":epochs}
grid_model1 = GridSearchCV(estimator=model1,param_grid=param_grid1)
grid_model1.fit(x_train,y_train, validation_data=(x_val,y_val))Determine best parameters
print("Best score", grid_model1.best_score_, "Best parameters", grid_model1.best_params_) Best score 0.760869562625885 Best parameters {'batch_size': 32, 'epochs': 60}
We found that to get optimal model performance we must train our neural network with the following parameters+ A batch size of 32 across 60 epochs.
Train data using Deep Neural Network
We configured the Deep Neural Network model using the above parameters (32 batch size and 60 epochs). Both training and validation data we incorporated in the model.
history1 = model1.fit(x_train, y_train, validation_data=(x_val,y_val), batch_size=32, epochs=60)
history1Predictions
Table underneath shows diabetes outcomes predicted by the Deep Neural Network model.
y_pred1 = model1.predict(x_test)
pd.DataFrame(y_pred1, columns = ["Predicted"])| Predicted | |
| 0 | 1 |
| 1 | 0 |
| 2 | 0 |
| 3 | 1 |
| 4 | 0 |
| … | … |
| 149 | 0 |
| 150 | 0 |
| 151 | 1 |
| 152 | 0 |
| 153 | 0 |
154 rows × 1 columns
Classification report
Underneath is an overview of the performance of the Deep Neural Network model in relation to accuracy, precision and recall.
classification1 = pd.DataFrame(metrics.classification_report(y_test,y_pred1, output_dict=True)).transpose()
classification1| precision | recall | f1-score | support | |
|---|---|---|---|---|
| 0 | 0.858407 | 0.906542 | 0.881818 | 107.000000 |
| 1 | 0.756098 | 0.659574 | 0.704545 | 47.000000 |
| accuracy | 0.831169 | 0.831169 | 0.831169 | 0.831169 |
| macro avg | 0.807252 | 0.783058 | 0.793182 | 154.000000 |
| weighted avg | 0.827183 | 0.831169 | 0.827715 | 154.000000 |
The Deep Neural Network model is accurate 83% of the time.
Confusion matrix
The confusion matrix underneath gives a detail of how the Deep Neural Network model performed.
cmat1 = pd.DataFrame(metrics.confusion_matrix(y_test,y_pred1), columns = ["Predicted: No", "Predicted: Yes"], index = ["Actual: No", "Actual: Yes"])
cmat1| Predicted: No | Predicted: Yes | |
|---|---|---|
| Actual: No | 96 | 11 |
| Actual: Yes | 16 | 3 |
The confusion matrix above clearly highlights that the Deep Neural Network model was made up of a total of 123 predictions. Out of the 123 predictions, the the Deep Neural Network model predicted “Yes” 11 times. In reality, 16 patients are diabetic, 96 are not diabetic.
ROC Curve
The ROC curve underneath summarize trade-off between the true positive rate and the false positive rate of the Deep Neural Network model using different probability thresholds.
y_predlogreg_proba1 = model1.predict_proba(x_test)[::,1]
fpr1, tpr1, _ = metrics.roc_curve(y_test,y_predlogreg_proba1)
auc1 = metrics.roc_auc_score(y_test,y_predlogreg_proba1)
plt.plot(fpr1, tpr1,label="Deep Neural Network, auc: " + str(auc1),color="orange",alpha=0.8)
plt.plot([0,1], [0,1],color="red",alpha=0.8)
plt.xlim([0.00,1.01])
plt.ylim([0.00,1.01])
plt.title("Validation ROC Curve - Deep Neural Network")
plt.xlabel("Specificity")
plt.ylabel("Sensitivity")
plt.legend(loc=4)
plt.show()
The ROC curve of the Deep Neural Network model follows the left-hand border. An accurate model should have an auc of 0.84.
Precision-Recall Curve
The precision-recall curve underneath shows the tradeoff of the Deep Neural Network model between precision and recall for different threshold.
precision1, recall1, threshold1 = metrics.precision_recall_curve(y_test,y_pred1)
aps1 = metrics.average_precision_score(y_test,y_pred1)
plt.plot(precision1, recall1,label="Deep Neural Network, aps: " +str(aps1),color="orange",alpha=0.8)
plt.axhline(y=0.5,color="red",alpha=0.8)
plt.xlabel("Precision")
plt.ylabel("Recall")
plt.legend(loc=4)
plt.title("Precision-Recall Curve - Deep Neural Network")
plt.show()
The average precision score of the Deep Neural Network model model is 0.52.
Loss across epochs
Underneath we visualized the mean loss of both training data and validation data over epochs.
plt.plot(history1.history["loss"],color="red",label="Training loss")
plt.plot(history1.history["val_loss"],color="orange",label="Validation loss")
plt.title("Training and Validation loss across epochs - Deep Neural Network")
plt.ylabel("Loss")
plt.xlabel("Epochs")
plt.legend(loc=4)
plt.show()
There is no convergence in the loss of both training data and validation. The mean loss of training data is high than that of the validation data. Optimizing our parameters might be useful.
Accuracy across epochs
Underneath we visualized the mean accuracy of both training data and validation data over epochs.
plt.plot(history1.history["accuracy"],color="red",label="Training accuracy")
plt.plot(history1.history["val_accuracy"],color="orange",label="Validation accuracy")
plt.title("Training and Validation loss accuracy epochs - Deep Neural Network")
plt.ylabel("Loss")
plt.xlabel("Epochs")
plt.legend(loc=4)
plt.show()
There is no convergence in the mean accuracy of both training data and validation. The mean accuracy of training data is high than that of the validation data. Optimizing our parameters might be useful.
Compare ROC Curves
We used a ROC to summarize trade-offs between the true positive rate and the false positive rates of all neural networks using different probability thresholds.
We found that the deep neural network constructed using Keras outperforms neural networks made available on the scikit-learn library . The ROC curve for the deep neural network is much close to the left-hand boarder.
plt.plot(fprRBM, tprRBM,label="Bernoulli Restricted Boltzmann Machine, auc: " + str(aucRBM),color="gray",alpha=0.8)
plt.plot(fprMLP, tprMLP,label="Multi layer Perception Neural Network, auc: " + str(aucMLP),color="black",alpha=0.8)
plt.plot(fpr1, tpr1,label="Deep Neural Network, auc: " + str(auc1),color="orange",alpha=0.8)
plt.plot([0,1], [0,1],color="red",alpha=0.8)
plt.xlim([0.00,1.01])
plt.ylim([0.00,1.01])
plt.title("Validation ROC Curve - All models")
plt.xlabel("Specificity")
plt.ylabel("Sensitivity")
plt.legend(loc=4)
plt.show()
The auc score of both Restricted Boltzmann Machine and multi-layer neural network is 0.83. In contrast, the deep neural network has an auc score of 0.87.
Compare Precision-Recall Curves
Now lets proceed and determine whether the average precision score of the deep neural network surpasses other neural networks’ score. Average precision are used to give a high-level overview of a model’s performance.
scores gives an idea of the performance of the neural network models.
The average precision score (aps) of both Restricted Boltzmann Machine and multi-layer neural network is 0.52. In contrast, the deep neural network has an aps of 0.60.