Hire the author: Merishna S

Image from Source


Deep Learning has introduced several technological revolutions and significant automation in the present world. One of the most life-changing contributions of Deep Learning is its use in healthcare. In this tutorial, we will look into one of the most amazing applications of Deep Learning in healthcare: diagnosing hypothyroid disease. We will be going through the code and workflow on how to diagnose hypothyroidism using neural networks, i.e. whether or not a person is suffering from Hypothyroid disease based on various metrics, as described in this GitHub project.


Hypothyroid disease or Hypothyroidism is one of the most difficult diseases to diagnose. It is a condition that causes your thyroid gland to not produce enough of certain crucial hormones. It usually may not cause noticeable symptoms in the early stages and hence becomes extremely difficult to diagnose. Applying deep learning algorithms can help assist healthcare workers in identifying signs and symptoms based on patterns shown in past cases of patients’ data.


In this tutorial, we will go through the process of using a deep neural network to classify whether or not a person is suffering from Hypothyroidism. The concept illustrated in this tutorial is primarily aimed at diagnosing hypothyroidism, however, if you wish to, you can use this concept to diagnose almost any disease you want based on the availability of data.


Deep Learning: It is a kind of machine learning technique that enables learning through the use of neural networks that mimic the human brain.

Hypothyroidism: also known as an underactive thyroid is a condition when a person’s body does not produce enough thyroid hormones.


  1. Programming knowledge in Python.
  2. Basic knowledge of Jupyter Notebook, Deep LearningKeras.

How can Deep Learning help diagnose hypothyroid disease?

The thyroid is a small butterfly-shaped gland in humans located in front of the windpipe and below the larynx. The thyroid gland is responsible for absorbing the iodine in food and generates two hormones: T3 and T4. The presence of these hormones is essential for the normal growth of the brain and neural system, especially during the first three years of life, i.e., in infants. Not having enough of these hormones is diagnosed as hypothyroidism, which is a class of thyroid disease that might even cause mental retardation in infants.

How to diagnose hypothyroid disease using Deep Learning
Image from Source

Thyroid diseases are said to be more common among women than men. Although various diagnosis methods such as Elisa, Electrochemiluminescence, etc., have been proposed for conducting hormone tests for diagnosing the type of thyroid disease, it still remains a difficult disease to diagnose.

Deep Learning tools and techniques can be effectively used in assisting healthcare experts in reducing human errors by predicting the possibility of the disease with high accuracy. We will be using neural networks, a class of deep learning algorithms that are proficient in accurately modeling the data to find hidden patterns and factors that affect the prediction or the diagnosis.

Classifying hypothyroidism using a Deep Learning model

We will now look into building a Deep Learning model to predict (diagnose) if a patient is suffering from hypothyroidism or not, based on several factors provided to it.

All the steps that we are going to discuss hereafter are available as a notebook here.

Step 1: Getting the data

For the training data, we are using the hypothyroid data from here. The dataset contains the value of different hormone levels of different patients along with our target variable, i.e., if the patient was suffering from hypothyroidism or not. To start using it, you can download the dataset and save it into a folder named “data”.

Upon reading in our data, it looks like this.

How to diagnose hypothyroid disease using Deep Learning

Step 2: Performing Data Engineering

After getting the data, cleaning, and formatting it is essential to avoid potential faults in our results due to inconsistencies or errors in our data.

We will now look into cleaning the data.

# Renaming the first column as target
dataset = dataset.rename(columns = {dataset.columns[0]:"target"})
# Check the count of data in target

The value ‘hypothyroid’ means that the patient was diagnosed with hypothyroidism and ‘‘negative” means that their hypothyroidism was not diagnosed. Also, the target value in the dataset looks imbalanced as the number of negative samples is much higher than the positive ones.

Converting categorical columns into binary

We will be mapping the value ‘‘hypothyroid” as ‘1′ and ‘‘negative” as ‘0′.

# mapping the values into binary
dataset["target"] = dataset["target"].map({"negative":0,"hypothyroid":1})

We will then look into the value counts of some other columns in the dataset and replace the categorical values with binary values.

# Displaying the categories in different columns
print("Unique categories in the column 'pregnant'", dataset['pregnant'].unique())
print("Count of categories in the column 'pregnant' \n", dataset["pregnant"].value_counts())
print("\nUnique categories in the column 'T3 measured'", dataset['T3_measured'].unique())
print("Count of categories in the column 'T3 measured' \n", dataset["T3_measured"].value_counts())
print("\nUnique categories in the column 'Gender'", dataset['Gender'].unique())
print("Count of categories in the column 'Gender' \n", dataset["Gender"].value_counts())
# Replacing the categorical values into binary values
dataset = dataset.replace({'f':0,'t':1, 'y':1, 'n':0, 'M':0, 'F':1})
# Displaying the head of the dataset

Hence, after binary conversion, the head of the dataset looks like this.

How to diagnose hypothyroid disease using Deep Learning

Now, some of the features contain ‘?’ values. So we will be replacing them with NaN values and count them.

# Replacing ? into NaN values
dataset.replace(to_replace='?', inplace=True, value=np.NaN)
# Count the number of null values

Here, we can see that the feature column “TBG” contains an extremely high number of null values. So, we will be not be using this column for our model.

# Dropping the TBG column as it contains extremely high number of null values
dataset.drop('TBG', axis = 1, inplace=True)

Converting columns into numeric values

This will help us in performing mathematical and statistical operations on the columns.

# Selecting columns with data type as 'object'
columns = dataset.columns[dataset.dtypes.eq('object')]
# Convert to numeric values
dataset[columns] = dataset[columns].apply(pd.to_numeric, errors='coerce')
# Viewing the details
How to diagnose hypothyroid disease using Deep Learning

Step 3: Exploring the data

Next, we will be looking into the relationships between different features present in our dataset by visualizing them. We will visualize the patterns in different columns (features) of the data using the hist() function of the Pandas library.

# Plot the histogram of different features
dataset.hist(figsize = (20,20));
How to diagnose hypothyroid disease using Deep Learning

Replacing null values in our data

We will now replace or impute the null values in our data with their appropriate statistical metric. The columns containing null values are

['Age', 'Gender', 'TSH', 'T3', 'TT4', 'T4U', 'FTI']

Based on the statistics plotted above, we can see that,

  • The features “Age” and “T4U” show a normal distribution, which is why we will be replacing the missing values with mean.
  • The features “TSH”, “T3”, “TT4” and “FTI” show a skewed distribution, which is why we will be replacing the missing values with the median.
  • The feature “gender” is binary and contains imbalanced data with the values “1” much greater than “0”. So, we will be replacing the missing values with “0”.
# Replacing null values by mean
dataset['Age'].fillna(dataset['Age'].mean(), inplace = True)
dataset['T4U'].fillna(dataset['T4U'].mean(), inplace = True)
# Replacing null values by median
dataset['TSH'].fillna(dataset['TSH'].mean(), inplace = True)
dataset['T3'].fillna(dataset['T3'].median(), inplace = True)
dataset['TT4'].fillna(dataset['TT4'].median(), inplace = True)
dataset['FTI'].fillna(dataset['FTI'].median(), inplace = True)
# The gender data looks to be imbalanced with 0 lesser than 1
# Replacing null values with 0
dataset['Gender'].fillna(0, inplace = True)

Let us now visualize the imputed data.

# Plot the histogram of different features
dataset.hist(figsize = (20,20));
How to diagnose hypothyroid disease using Deep Learning

Hence, we’ve successfully filled in all the missing values in our dataset. Our data is now cleaned and formatted and we are now ready to model the data for prediction using Deep Learning.

Step 4: Modeling the data

For modeling the data, we will be implementing a deep neural network using Keras. We will be using the Neural network for predicting the hypothyroidism (target) based on our input data (features).

1. Defining the features and target variables

We will be separating our dataset into features (X) and the target variable (y).

# Features
X = dataset.drop('target', axis = 1) # selecting all columns except the target
# Target variable
y = dataset['target']
# Print the shape
print(X.shape, y.shape)
view raw model_data_1.py hosted with ❤ by GitHub

2. Splitting the data into the train (80%) and test (20%) set

In order to efficiently evaluate our model, we will be training our model on 80% of the data, and keep the rest 20% for testing the model.

from sklearn.model_selection import train_test_split
# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
view raw model_data_2.py hosted with ❤ by GitHub

3. Scaling the train and test data

Scaling the data helps in normalizing the data within a particular range and also helps in speeding up the calculations in an algorithm. For scaling the data, we will be using the StandardScaler class from scikit-learn.

from sklearn.preprocessing import StandardScaler
# Initialization of the class
scaler = StandardScaler()
# Applying the scaler on test and train data
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
view raw model_data_3.py hosted with ❤ by GitHub

4. Building the neural network

The scope of our project is to design a feed-forward neural network for the classification of our target variable. Several factors determine the performance of the model such as the input, hidden and output layer configurations, and the training methodology used. Most of the neural network architectures are determined by experimentation in practice. Here, we are defining a neural network containing a sequential model with one hidden layer and one output layer.

How to diagnose hypothyroid disease using Deep Learning
Neural Network Architecture

We have 24 feature columns due to which we have specified the input dimension as input_dim=24. Here, we’ve implemented the ReLU activation function in the hidden layer, and sigmoid function for the output layer to return a binary value of output, i.e. hypothyroid (1) or negative (0).

We are then training the model with a validation split of 0.2 for 100 epochs.

# Input
model = Sequential()
# Hidden layer
model.add(Dense(64, kernel_initializer='uniform', input_dim=24, activation='relu'))
# Output layer
model.add(Dense(1, kernel_initializer='uniform', activation='sigmoid'))
# Compiling the model with 'adam' optimizer and loss function as 'binary_crossentropy'
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# Training the model
result = model.fit(X_train, y_train, epochs=100, validation_split=0.2, batch_size=40, verbose=2)
view raw model_data_4.py hosted with ❤ by GitHub

Step 5: Evaluating the model

We will now evaluate the performance of our trained model on test data to find metrics such as Accuracy, F1-score, recall, and precision.

# Get the loss and accuracy of the model by evaluation
loss, acc = model.evaluate(X_test, y_test)
# Print the loss and accuracy score for the model
print("%s: %.2f%%" % (model.metrics_names[0], loss*100))
print("%s: %.2f%%" % (model.metrics_names[1], acc*100))
# Predicting the output predictions
y_pred = model.predict(X_test).round()
# Calculating the F1 score, recall, and precision
print("%s: %.2f%%" % ("F1-score", f1_score(y_test, y_pred)*100))
print("%s: %.2f%%" % ("Recall", recall_score(y_test, y_pred)*100))
print("%s: %.2f%%" % ("Precision", precision_score(y_test, y_pred)*100))
How to diagnose hypothyroid disease using Deep Learning
How to diagnose hypothyroid disease using Deep Learning

We are getting a quite good accuracy of 98.10% from our Deep Learning model. However, our data is imbalanced and hence accuracy is not a good measure. Hence, we are using a metric called F1-score.

We are getting a quite good value of F1-score, recall, and precision as well, and hence, our model seems to be performing well.

Training and validation loss

# summarize the result and plot the training and test loss
# Set the parameters
plt.title('Deep learning model loss')
plt.legend(['train', 'test'], loc='upper right')
# Display the plots
How to diagnose hypothyroid disease using Deep Learning

We can see that both the losses have converged at a very good rate.

Step 6: Saving the model

Finally, we will be saving the trained model for future use. It can be easily loaded and used for predictions.

# Save the trained model
# Load the model
model = keras.models.load_model('saved_model.h5')
view raw save_model.py hosted with ❤ by GitHub

Learning tools and strategies

  1. Deep Learning is a type of machine learning algorithm that uses neural networks for performing its predictions. The key to learning about neural networks effectively is to learn and visualize the whole architecture of the system. By doing this, we can easily understand how the data is being processed step-by-step.
  2. Also, it is a good practice to print or log important messages and errors to help with debugging.
  3. Neural networks are very sensitive to hyperparameters. Therefore, it’s very important to tune them precisely so as to increase the model’s accuracy and improve its performance.

Reflective Analysis

Overall, this project was an interesting take on leveraging the use of Artificial Intelligence in healthcare. With the growing use of technology, its scope has as well grown ten-folds. Additionally, this project covers all the different processes from start to end of a deep learning project. Also, instead of a general machine learning algorithm, I opted to use an artificial neural network as they can be easily modified and works well with any kind of data which makes them so versatile to use and create despite their complex architectures.

Conclusions and Future Directions

In this project, we have illustrated the potential use of neural networks in diagnosing hypothyroid disease. The identification of thyroid disease identification remains to be an essential yet difficult task in both clinical diagnosis and statistical classification. The diagnosis involves the use of a large number of interrelated patient attributes along with extremely unbalanced groups that result in a complicated relationship between the input features.

Artificial neural networks show a promising result due to its flexible nature in modeling complex patterns of data for diagnosis. The architecture of the neural network implemented for this project was fairly simple for giving an overview. However, we can implement more complex architectures of neural networks with different hyperparameters comprised of several hidden layers and number of computations. Moreover, this knowledge can be further extended for many other disease diagnoses where such problems exist.


  • The dataset for training our model was derived from here.

Also, the code for this project on Diagnosing hypothyroid disease using Deep Learning is available on GitHub.

If you’re looking to do similar innovative projects in Deep Learning, you might be interested in this project on How to generate unique architectures using GANs.

Hire the author: Merishna S