---

Coursework 1:

Submitted for the partial fulfilment of the DSM150 course

By Hendrik Matthys van Rooyen

230221176

---

DLWP Flow: MedMNIST - OCTMINST¶

---

Introduction¶

For this coursework the instructions followed were as follows: ...

The Dataset¶

In preparation for this coursework, many datasets were considered, amongst them:

• Anemia Classification
• Back-order Prediction
• Heart Attack Risk Prediction
• IMDB Reccomender
• Text Generation using a corpus

Of these the Anemia and IMDB provided simple models, with minimal opportunity to satisfy the requirements for the Coursework. Both the Back-order and Heart Attack risk provided very poor quality models, barely out-preforming the baseline models. The Text Recommendation, although an interesting project does not fall into the scope of the project, and was infeasible to achieve within the Coursework limitations on the available hardware.

All this considered, the octmnist dataset form the MedMNIST library has provided ample opportunity to explore the data, balance the datasets, and leaves room for improvement on top of a basic neural network. While the simple Neural networks like these aren't normally considered for image classification, as opposed to CNN's, the DLWP book showed some success in the classification of the normal MNIST dataset, and this one was also considdered for that reason.

---

Prepare Environment¶

Load Packages¶

In [1]:

import medmnist
from medmnist import INFO, Evaluator

import pandas as pd
import numpy as np

from tensorflow.keras.callbacks import ModelCheckpoint

import medmnist from medmnist import INFO, Evaluator import pandas as pd import numpy as np from tensorflow.keras.callbacks import ModelCheckpoint

Load Dataset¶

In [2]:

data_flag = 'octmnist'
# data_flag = 'breastmnist'
download = True

NUM_EPOCHS = 3
BATCH_SIZE = 128
lr = 0.001

info = INFO[data_flag]
n_channels = info['n_channels']
n_classes = len(info['label'])

data_flag = 'octmnist' # data_flag = 'breastmnist' download = True NUM_EPOCHS = 3 BATCH_SIZE = 128 lr = 0.001 info = INFO[data_flag] n_channels = info['n_channels'] n_classes = len(info['label'])

In [3]:

info

info

Out[3]:

{'python_class': 'OCTMNIST',
 'description': 'The OCTMNIST is based on a prior dataset of 109,309 valid optical coherence tomography (OCT) images for retinal diseases. The dataset is comprised of 4 diagnosis categories, leading to a multi-class classification task. We split the source training set with a ratio of 9:1 into training and validation set, and use its source validation set as the test set. The source images are gray-scale, and their sizes are (384−1,536)×(277−512). We center-crop the images and resize them into 1×28×28.',
 'url': 'https://zenodo.org/record/6496656/files/octmnist.npz?download=1',
 'MD5': 'c68d92d5b585d8d81f7112f81e2d0842',
 'task': 'multi-class',
 'label': {'0': 'choroidal neovascularization',
  '1': 'diabetic macular edema',
  '2': 'drusen',
  '3': 'normal'},
 'n_channels': 1,
 'n_samples': {'train': 97477, 'val': 10832, 'test': 1000},
 'license': 'CC BY 4.0'}

In [4]:

DataClass = getattr(medmnist, info['python_class'])

# Load each split
train_data = DataClass(split='train', download=True)
val_data = DataClass(split='val', download=True)
test_data = DataClass(split='test', download=True)

DataClass = getattr(medmnist, info['python_class']) # Load each split train_data = DataClass(split='train', download=True) val_data = DataClass(split='val', download=True) test_data = DataClass(split='test', download=True)

Using downloaded and verified file: C:\Users\mvanr\.medmnist\octmnist.npz
Using downloaded and verified file: C:\Users\mvanr\.medmnist\octmnist.npz
Using downloaded and verified file: C:\Users\mvanr\.medmnist\octmnist.npz

---

Investigate Dataset¶

In investigating the dataset, we inspect both samples from the data, to determine if there may be any notable differences or similarities between entries, which could inform the design of the network or the handling of the data. We also investigate the distribution of the entries across classes.

In [5]:

import matplotlib.pyplot as plt

def display_images_with_label(images, labels, info, target_label, num_images, seed=None):
    # Set the random seed if provided.
    if seed is not None:
        np.random.seed(seed)
        
    # Ensure the labels are numpy arrays for boolean indexing.
    labels = np.array(labels)
    images = np.array(images)
    
    # Find the indices of all images that match the target label.
    matching_indices = np.where(labels == target_label)[0]
    
    # Select a random subset of these indices.
    if len(matching_indices) >= num_images:
        selected_indices = np.random.choice(matching_indices, size=num_images, replace=False)
    else:
        selected_indices = matching_indices  # If there aren't enough, select them all.
    
    # Set up the plot size.
    plt.figure(figsize=(2 * num_images, 2))  # You can adjust the figure size as needed.
    
    # Create a subplot for each selected image.
    for i, index in enumerate(selected_indices, 1):
        ax = plt.subplot(1, num_images, i)
        plt.imshow(images[index], cmap=plt.cm.binary)
        plt.axis('off')  # Hide the axis to put more focus on the images.
        # Set the title for the first image only.
        if i == 1:
            ax.set_title(f'{info["label"][str(target_label)]}')
    
    plt.show()
    
display_images_with_label(test_data.imgs,test_data.labels,info,0,4,42)
display_images_with_label(test_data.imgs,test_data.labels,info,1,4,42)
display_images_with_label(test_data.imgs,test_data.labels,info,2,4,42)
display_images_with_label(test_data.imgs,test_data.labels,info,3,4,42)

import matplotlib.pyplot as plt def display_images_with_label(images, labels, info, target_label, num_images, seed=None): # Set the random seed if provided. if seed is not None: np.random.seed(seed) # Ensure the labels are numpy arrays for boolean indexing. labels = np.array(labels) images = np.array(images) # Find the indices of all images that match the target label. matching_indices = np.where(labels == target_label)[0] # Select a random subset of these indices. if len(matching_indices) >= num_images: selected_indices = np.random.choice(matching_indices, size=num_images, replace=False) else: selected_indices = matching_indices # If there aren't enough, select them all. # Set up the plot size. plt.figure(figsize=(2 * num_images, 2)) # You can adjust the figure size as needed. # Create a subplot for each selected image. for i, index in enumerate(selected_indices, 1): ax = plt.subplot(1, num_images, i) plt.imshow(images[index], cmap=plt.cm.binary) plt.axis('off') # Hide the axis to put more focus on the images. # Set the title for the first image only. if i == 1: ax.set_title(f'{info["label"][str(target_label)]}') plt.show() display_images_with_label(test_data.imgs,test_data.labels,info,0,4,42) display_images_with_label(test_data.imgs,test_data.labels,info,1,4,42) display_images_with_label(test_data.imgs,test_data.labels,info,2,4,42) display_images_with_label(test_data.imgs,test_data.labels,info,3,4,42)

From the samples above, a challenge that will form a larger theme in the project becomes apparent; that being, the similarity between the drusen and the normal images.

In [6]:

import matplotlib.pyplot as plt

unique_labels, counts = np.unique(train_data.labels, return_counts=True)
label_names = [info['label'][str(label)] for label in unique_labels]

# Plotting the distribution of labels
plt.figure(figsize=(10, 6))
plt.bar(label_names, counts, color='skyblue')
plt.xlabel('Labels')
plt.ylabel('Frequency')
plt.title('Distribution of Labels in the Training Dataset')
plt.xticks(rotation=45)
plt.show()

import matplotlib.pyplot as plt unique_labels, counts = np.unique(train_data.labels, return_counts=True) label_names = [info['label'][str(label)] for label in unique_labels] # Plotting the distribution of labels plt.figure(figsize=(10, 6)) plt.bar(label_names, counts, color='skyblue') plt.xlabel('Labels') plt.ylabel('Frequency') plt.title('Distribution of Labels in the Training Dataset') plt.xticks(rotation=45) plt.show()

The above graph shows a large inequality between entries per label, with normal being the most, and drusen being the least. This increases the problem of the classifications being very similar.

In [7]:

import matplotlib.pyplot as plt

unique_labels, counts = np.unique(np.concatenate((train_data.labels, val_data.labels, test_data.labels), axis=0), return_counts=True)
label_names = [info['label'][str(label)] for label in unique_labels]

# Plotting the distribution of labels
plt.figure(figsize=(10, 6))
plt.bar(label_names, counts, color='skyblue')
plt.xlabel('Labels')
plt.ylabel('Frequency')
plt.title('Distribution of Labels in the Full Dataset')
plt.xticks(rotation=45)
plt.show()

import matplotlib.pyplot as plt unique_labels, counts = np.unique(np.concatenate((train_data.labels, val_data.labels, test_data.labels), axis=0), return_counts=True) label_names = [info['label'][str(label)] for label in unique_labels] # Plotting the distribution of labels plt.figure(figsize=(10, 6)) plt.bar(label_names, counts, color='skyblue') plt.xlabel('Labels') plt.ylabel('Frequency') plt.title('Distribution of Labels in the Full Dataset') plt.xticks(rotation=45) plt.show()

---

Prepare Data¶

In preparation of training the network we compute the class weight of each label in order to experiment with balancing the importance in training.

In order to balance the classes we augment the images of classes with lower representation.

The image pixel values, which is normally between 5 and 255 is finally divided by 255 in order to normalize the values between 0 and 1

In [8]:

from sklearn.utils.class_weight import compute_class_weight

class_weights = compute_class_weight(
    class_weight='balanced',
    classes=np.unique(train_data.labels.flatten()),
    y=train_data.labels.flatten()
)

class_weight_dict = dict(enumerate(class_weights / 4))
class_weight_dict

from sklearn.utils.class_weight import compute_class_weight class_weights = compute_class_weight( class_weight='balanced', classes=np.unique(train_data.labels.flatten()), y=train_data.labels.flatten() ) class_weight_dict = dict(enumerate(class_weights / 4)) class_weight_dict

Out[8]:

{0: 0.18194697467447138,
 1: 0.596525261921081,
 2: 0.7856993164818158,
 3: 0.13236676009212184}

In [9]:

from keras.preprocessing.image import ImageDataGenerator
import numpy as np

def augment_images_to_balance_classes(images, labels, batch_size, seed):
    
    images = images.reshape((-1, 28, 28, 1))
    
    # Find the label with the most entries
    (unique, counts) = np.unique(labels, return_counts=True)
    max_count = np.max(counts)
    class_indices = {label: np.where(labels == label)[0] for label in unique}
    max_label = unique[np.argmax(counts)]

    # Initialize the image data generator
    datagen = ImageDataGenerator(
        rotation_range=20,
        width_shift_range=0.2,
        height_shift_range=0.2,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True,
        fill_mode='nearest'
    )

    # Seed for reproducibility
    np.random.seed(seed)

    # Initialize lists for the balanced dataset
    balanced_images = list(images)
    balanced_labels = list(labels)

    # Augment data for classes with fewer images than the max_count
    for label, indices in class_indices.items():
        if label == max_label:
            continue  # Skip the class with the most samples
        num_to_augment = max_count - counts[unique.tolist().index(label)]
        
        # Augment images until the class has the same number of images as the max_count
        augmentation_factor = int(np.ceil(num_to_augment / len(indices)))
        for i in range(augmentation_factor):
            for index in indices:
                image_to_augment = images[index].reshape((1, *images[index].shape))
                iterator = datagen.flow(image_to_augment, batch_size=1, seed=seed)
                for _ in range(min(num_to_augment, batch_size)):
                    augmented_image = next(iterator)[0].astype('uint8')
                    balanced_images.append(augmented_image)
                    balanced_labels.append(label)
                    num_to_augment -= 1
                    if num_to_augment <= 0:
                        break
                if num_to_augment <= 0:
                    break

    return np.array(balanced_images), np.array(balanced_labels)

from keras.preprocessing.image import ImageDataGenerator import numpy as np def augment_images_to_balance_classes(images, labels, batch_size, seed): images = images.reshape((-1, 28, 28, 1)) # Find the label with the most entries (unique, counts) = np.unique(labels, return_counts=True) max_count = np.max(counts) class_indices = {label: np.where(labels == label)[0] for label in unique} max_label = unique[np.argmax(counts)] # Initialize the image data generator datagen = ImageDataGenerator( rotation_range=20, width_shift_range=0.2, height_shift_range=0.2, shear_range=0.2, zoom_range=0.2, horizontal_flip=True, fill_mode='nearest' ) # Seed for reproducibility np.random.seed(seed) # Initialize lists for the balanced dataset balanced_images = list(images) balanced_labels = list(labels) # Augment data for classes with fewer images than the max_count for label, indices in class_indices.items(): if label == max_label: continue # Skip the class with the most samples num_to_augment = max_count - counts[unique.tolist().index(label)] # Augment images until the class has the same number of images as the max_count augmentation_factor = int(np.ceil(num_to_augment / len(indices))) for i in range(augmentation_factor): for index in indices: image_to_augment = images[index].reshape((1, *images[index].shape)) iterator = datagen.flow(image_to_augment, batch_size=1, seed=seed) for _ in range(min(num_to_augment, batch_size)): augmented_image = next(iterator)[0].astype('uint8') balanced_images.append(augmented_image) balanced_labels.append(label) num_to_augment -= 1 if num_to_augment <= 0: break if num_to_augment <= 0: break return np.array(balanced_images), np.array(balanced_labels)

In [10]:

train_images, train_labels = augment_images_to_balance_classes(train_data.imgs, train_data.labels.flatten(), batch_size=32, seed=42)
train_images = train_images / 255.0

val_images = val_data.imgs / 255.0
val_labels = val_data.labels

test_images = test_data.imgs / 255.0
test_labels = test_data.labels

print('tensor shape:')
print('\ttraining images:', train_images.shape)
print('\ttraining labels:', train_labels.shape)
print('\tvalidation images:', val_images.shape)
print('\tvalidation images:', val_labels.shape)
print('\ttesting images:', test_images.shape)
print('\ttesting images:', test_labels.shape)

train_images, train_labels = augment_images_to_balance_classes(train_data.imgs, train_data.labels.flatten(), batch_size=32, seed=42) train_images = train_images / 255.0 val_images = val_data.imgs / 255.0 val_labels = val_data.labels test_images = test_data.imgs / 255.0 test_labels = test_data.labels print('tensor shape:') print('\ttraining images:', train_images.shape) print('\ttraining labels:', train_labels.shape) print('\tvalidation images:', val_images.shape) print('\tvalidation images:', val_labels.shape) print('\ttesting images:', test_images.shape) print('\ttesting images:', test_labels.shape)

tensor shape:
	training images: (184104, 28, 28, 1)
	training labels: (184104,)
	validation images: (10832, 28, 28)
	validation images: (10832, 1)
	testing images: (1000, 28, 28)
	testing images: (1000, 1)

In [11]:

import matplotlib.pyplot as plt

unique_labels, counts = np.unique(train_labels, return_counts=True)
label_names = [info['label'][str(label)] for label in unique_labels]

# Plotting the distribution of labels
plt.figure(figsize=(10, 6))
plt.bar(label_names, counts, color='skyblue')
plt.xlabel('Labels')
plt.ylabel('Frequency')
plt.title('Distribution of Labels in the Training Dataset after Augmentation')
plt.xticks(rotation=45)
plt.show()

import matplotlib.pyplot as plt unique_labels, counts = np.unique(train_labels, return_counts=True) label_names = [info['label'][str(label)] for label in unique_labels] # Plotting the distribution of labels plt.figure(figsize=(10, 6)) plt.bar(label_names, counts, color='skyblue') plt.xlabel('Labels') plt.ylabel('Frequency') plt.title('Distribution of Labels in the Training Dataset after Augmentation') plt.xticks(rotation=45) plt.show()

In [12]:

import matplotlib.pyplot as plt

unique_labels, counts = np.unique(val_labels, return_counts=True)
label_names = [info['label'][str(label)] for label in unique_labels]

# Plotting the distribution of labels
plt.figure(figsize=(10, 6))
plt.bar(label_names, counts, color='skyblue')
plt.xlabel('Labels')
plt.ylabel('Frequency')
plt.title('Distribution of Labels in the Training Dataset after Augmentation')
plt.xticks(rotation=45)
plt.show()

import matplotlib.pyplot as plt unique_labels, counts = np.unique(val_labels, return_counts=True) label_names = [info['label'][str(label)] for label in unique_labels] # Plotting the distribution of labels plt.figure(figsize=(10, 6)) plt.bar(label_names, counts, color='skyblue') plt.xlabel('Labels') plt.ylabel('Frequency') plt.title('Distribution of Labels in the Training Dataset after Augmentation') plt.xticks(rotation=45) plt.show()

Above it is shown that the labels is now balanced after the augmentation.

In [12]:

# Flatten images if they are not already 1D
if len(train_images.shape) > 2:
    # Flatten the image data to 1D (number of samples, image_width * image_height * channels)
    train_images = train_images.reshape((len(train_images), 28*28))
    val_images = val_images.reshape((10832, 28*28))
    test_images = test_images.reshape((1000, 28*28))

# Flatten images if they are not already 1D if len(train_images.shape) > 2: # Flatten the image data to 1D (number of samples, image_width * image_height * channels) train_images = train_images.reshape((len(train_images), 28*28)) val_images = val_images.reshape((10832, 28*28)) test_images = test_images.reshape((1000, 28*28))

In [13]:

from tensorflow.keras.utils import to_categorical

original_label = test_labels[0]
#train_labels = to_categorical(train_labels)
#test_labels = to_categorical(test_labels)
#val_labels = to_categorical(val_labels)

print('\'',original_label, '\'')
print(test_labels[0])

from tensorflow.keras.utils import to_categorical original_label = test_labels[0] #train_labels = to_categorical(train_labels) #test_labels = to_categorical(test_labels) #val_labels = to_categorical(val_labels) print('\'',original_label, '\'') print(test_labels[0])

' [3] '
[3]

Building the network¶

In [14]:

info

info

Out[14]:

{'python_class': 'OCTMNIST',
 'description': 'The OCTMNIST is based on a prior dataset of 109,309 valid optical coherence tomography (OCT) images for retinal diseases. The dataset is comprised of 4 diagnosis categories, leading to a multi-class classification task. We split the source training set with a ratio of 9:1 into training and validation set, and use its source validation set as the test set. The source images are gray-scale, and their sizes are (384−1,536)×(277−512). We center-crop the images and resize them into 1×28×28.',
 'url': 'https://zenodo.org/record/6496656/files/octmnist.npz?download=1',
 'MD5': 'c68d92d5b585d8d81f7112f81e2d0842',
 'task': 'multi-class',
 'label': {'0': 'choroidal neovascularization',
  '1': 'diabetic macular edema',
  '2': 'drusen',
  '3': 'normal'},
 'n_channels': 1,
 'n_samples': {'train': 97477, 'val': 10832, 'test': 1000},
 'license': 'CC BY 4.0'}

In [15]:

train_labels.shape

train_labels.shape

Out[15]:

(184104,)

In [16]:

from sklearn.utils.class_weight import compute_class_weight

# classes should be a list of unique labels in the dataset.
# y_train should be a list/array of your training labels.
class_weights = compute_class_weight(
    class_weight='balanced',
    classes=np.unique(train_labels.flatten()),
    y=train_labels.flatten()
)

class_weight_dict = dict(enumerate(class_weights))

from sklearn.utils.class_weight import compute_class_weight # classes should be a list of unique labels in the dataset. # y_train should be a list/array of your training labels. class_weights = compute_class_weight( class_weight='balanced', classes=np.unique(train_labels.flatten()), y=train_labels.flatten() ) class_weight_dict = dict(enumerate(class_weights))

In [17]:

class_weight_dict

class_weight_dict

Out[17]:

{0: 1.0, 1: 1.0, 2: 1.0, 3: 1.0}

In [18]:

class_weight_dict[2] += class_weight_dict[2]*1.8

class_weight_dict[2] += class_weight_dict[2]*1.8

In [19]:

class_weight_dict = {0: 1.0,1: 1.0, 2:1.0,3:1.0}

class_weight_dict = {0: 1.0,1: 1.0, 2:1.0,3:1.0}

In [20]:

class_weight_dict

class_weight_dict

Out[20]:

{0: 1.0, 1: 1.0, 2: 1.0, 3: 1.0}

Training the network¶

Helper functions is defined in order to display the model performance after training in order to evaluate the outcomes of hyperparameter tuning and network structure changes.

Checkpoints is created at each optimal epoch in order to effectively evaluate performance, and save time from having to re-run model training.

In [20]:

from tensorflow.keras.models import load_model

def plot_epochs(x,
                y_A, style_A, label_A,
                y_B, style_B, label_B,
                title, x_label, y_label):
    import matplotlib.pyplot as plt
    plt.clf()
    plt.plot(x,y_A, style_A, label = label_A)
    plt.plot(x,y_B, style_B, label = label_B)
    plt.title(title)
    plt.xlabel(x_label)
    plt.ylabel(y_label)
    plt.legend()
    plt.show()

def plot_network_summaries(history, optimal_model = None, test_images = None, test_labels = None ):
    loss = history.history['loss']
    val_loss = history.history['val_loss']
    acc = history.history['accuracy']
    val_acc = history.history['val_accuracy']

    plot_epochs(
        range(1, len(loss) + 1),
        loss, 'bo', 'Training',
        val_loss, 'b', 'Validation',
        'Training and validation loss',
        'Epochs',
        'Loss'
    )

    plot_epochs(
        range(1, len(loss) + 1),
        acc, 'bo', 'Training',
        val_acc, 'b', 'Validation',
        'Training and validation accuracy',
        'Epochs',
        'Accuracy'
    )
    
    print('Optimal point: ', np.argmin(val_loss)+1, 'epochs')
    
    if optimal_model is None:
        return
    
    network = load_model(optimal_model)
    probabilities = network.predict(test_images)
    predicted_classes = np.argmax(probabilities, axis=1)

    true_classes = test_labels.flatten()

    from sklearn.metrics import confusion_matrix
    import pandas as pd
    import seaborn as sns
    import matplotlib.pyplot as plt

    # Generate the confusion matrix
    labels = [info['label'][str(label)] for label in range(n_classes)]
    cm = confusion_matrix(true_classes, predicted_classes.flatten())

    # If you want to have a DataFrame representation
    cm_df = pd.DataFrame(cm, index=labels, columns=labels)

    plt.figure(figsize=(10,7))
    sns.heatmap(cm_df, annot=True, fmt='g')
    plt.title('Confusion Matrix')
    plt.ylabel('Actual Labels')
    plt.xlabel('Predicted Labels')
    plt.xticks(rotation=45)
    plt.show()
    
    test_loss, test_acc = network.evaluate(test_images, test_labels)
    print('Test accuracy:', test_acc)

from tensorflow.keras.models import load_model def plot_epochs(x, y_A, style_A, label_A, y_B, style_B, label_B, title, x_label, y_label): import matplotlib.pyplot as plt plt.clf() plt.plot(x,y_A, style_A, label = label_A) plt.plot(x,y_B, style_B, label = label_B) plt.title(title) plt.xlabel(x_label) plt.ylabel(y_label) plt.legend() plt.show() def plot_network_summaries(history, optimal_model = None, test_images = None, test_labels = None ): loss = history.history['loss'] val_loss = history.history['val_loss'] acc = history.history['accuracy'] val_acc = history.history['val_accuracy'] plot_epochs( range(1, len(loss) + 1), loss, 'bo', 'Training', val_loss, 'b', 'Validation', 'Training and validation loss', 'Epochs', 'Loss' ) plot_epochs( range(1, len(loss) + 1), acc, 'bo', 'Training', val_acc, 'b', 'Validation', 'Training and validation accuracy', 'Epochs', 'Accuracy' ) print('Optimal point: ', np.argmin(val_loss)+1, 'epochs') if optimal_model is None: return network = load_model(optimal_model) probabilities = network.predict(test_images) predicted_classes = np.argmax(probabilities, axis=1) true_classes = test_labels.flatten() from sklearn.metrics import confusion_matrix import pandas as pd import seaborn as sns import matplotlib.pyplot as plt # Generate the confusion matrix labels = [info['label'][str(label)] for label in range(n_classes)] cm = confusion_matrix(true_classes, predicted_classes.flatten()) # If you want to have a DataFrame representation cm_df = pd.DataFrame(cm, index=labels, columns=labels) plt.figure(figsize=(10,7)) sns.heatmap(cm_df, annot=True, fmt='g') plt.title('Confusion Matrix') plt.ylabel('Actual Labels') plt.xlabel('Predicted Labels') plt.xticks(rotation=45) plt.show() test_loss, test_acc = network.evaluate(test_images, test_labels) print('Test accuracy:', test_acc)

Network A¶

This fist model has been left for reference, as it was trained prior to the balancing of the labels through augmentation.

The code for balancing is also still present, but was later moved to the data preparation stage.

In [60]:

from tensorflow.keras import models, layers

class_weight_dict = {0: 1.0,1: 1.0, 2:1.0,3:1.0}

checkpoint = ModelCheckpoint(
    'best_model_A.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

network = models.Sequential()
network.add(layers.Dense(512, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='RMSprop', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=25, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers class_weight_dict = {0: 1.0,1: 1.0, 2:1.0,3:1.0} checkpoint = ModelCheckpoint( 'best_model_A.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) network = models.Sequential() network.add(layers.Dense(512, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='RMSprop', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=25, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/25
1439/1439 [==============================] - 5s 3ms/step - loss: 1.1059 - accuracy: 0.5091 - val_loss: 0.9769 - val_accuracy: 0.6251
Epoch 2/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.9600 - accuracy: 0.5969 - val_loss: 0.9124 - val_accuracy: 0.6350
Epoch 3/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.8895 - accuracy: 0.6327 - val_loss: 0.8386 - val_accuracy: 0.6753
Epoch 4/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.8473 - accuracy: 0.6555 - val_loss: 0.7633 - val_accuracy: 0.7324
Epoch 5/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.8130 - accuracy: 0.6719 - val_loss: 0.7707 - val_accuracy: 0.7156
Epoch 6/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.7888 - accuracy: 0.6848 - val_loss: 0.7031 - val_accuracy: 0.7425
Epoch 7/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.7681 - accuracy: 0.6934 - val_loss: 0.7001 - val_accuracy: 0.7426
Epoch 8/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.7504 - accuracy: 0.7019 - val_loss: 0.7299 - val_accuracy: 0.7326
Epoch 9/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.7358 - accuracy: 0.7075 - val_loss: 0.6502 - val_accuracy: 0.7639
Epoch 10/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.7218 - accuracy: 0.7149 - val_loss: 0.7189 - val_accuracy: 0.7274
Epoch 11/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.7104 - accuracy: 0.7192 - val_loss: 0.6426 - val_accuracy: 0.7673
Epoch 12/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.7010 - accuracy: 0.7237 - val_loss: 0.7120 - val_accuracy: 0.7644
Epoch 13/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.6902 - accuracy: 0.7281 - val_loss: 0.8474 - val_accuracy: 0.7147
Epoch 14/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.6802 - accuracy: 0.7325 - val_loss: 0.6103 - val_accuracy: 0.7894
Epoch 15/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.6717 - accuracy: 0.7377 - val_loss: 0.6696 - val_accuracy: 0.7811
Epoch 16/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.6645 - accuracy: 0.7405 - val_loss: 0.6502 - val_accuracy: 0.7635
Epoch 17/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.6589 - accuracy: 0.7427 - val_loss: 0.6142 - val_accuracy: 0.7864
Epoch 18/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.6505 - accuracy: 0.7460 - val_loss: 0.5890 - val_accuracy: 0.7971
Epoch 19/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.6445 - accuracy: 0.7496 - val_loss: 0.5686 - val_accuracy: 0.7981
Epoch 20/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.6383 - accuracy: 0.7517 - val_loss: 0.5983 - val_accuracy: 0.7898
Epoch 21/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.6323 - accuracy: 0.7538 - val_loss: 0.6732 - val_accuracy: 0.7751
Epoch 22/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.6265 - accuracy: 0.7567 - val_loss: 0.6332 - val_accuracy: 0.7745
Epoch 23/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.6221 - accuracy: 0.7592 - val_loss: 0.6671 - val_accuracy: 0.7590
Epoch 24/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.6171 - accuracy: 0.7617 - val_loss: 0.5664 - val_accuracy: 0.8058
Epoch 25/25
1439/1439 [==============================] - 5s 3ms/step - loss: 0.6120 - accuracy: 0.7644 - val_loss: 0.5863 - val_accuracy: 0.7994

In [64]:

plot_network_summaries(history, 'best_model_A.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_A.hdf5', val_images, val_labels)

Optimal point:  23 epochs
339/339 [==============================] - 1s 1ms/step

339/339 [==============================] - 1s 2ms/step - loss: 0.5664 - accuracy: 0.8058
Test accuracy: 0.8057606816291809

From the above we can see that the model at the optimal point results in an .8 accuracy, which is a good start, however, it is also clear that the drusen label is much less accurate.

Having an unbalanced validation set (as provided by the medmnist library) does not provide a reliable indication of performance.

This, combined with the similarity between drusen and normal would cause a very inaccurate final test.

Reload and balance the dataset¶

In [15]:

DataClass = getattr(medmnist, info['python_class'])

# Load each split
train_data = DataClass(split='train', download=True)
val_data = DataClass(split='val', download=True)

import numpy as np
from sklearn.model_selection import train_test_split

# Combine the datasets
all_imgs = np.concatenate((train_data.imgs, val_data.imgs), axis=0)
all_labels = np.concatenate((train_data.labels, val_data.labels), axis=0)

# Shuffle the dataset
permutation = np.random.permutation(len(all_labels))
shuffled_imgs = all_imgs[permutation]
shuffled_labels = all_labels[permutation]

# Define the number of samples per label for the validation set
validation_count_per_label = 750
unique_labels = [0, 1, 2, 3]  # Defined set of unique labels

# Initialize lists to hold the split data
validation_imgs = []
validation_labels = []
train_imgs = []
train_labels = []

for label in unique_labels:
    label_indices = np.where(shuffled_labels == label)[0]
    
    # Check if there are enough samples for the label
    if len(label_indices) < validation_count_per_label:
        raise ValueError(f"Not enough samples for label {label}.")
    
    validation_indices = label_indices[:validation_count_per_label]
    train_indices = label_indices[validation_count_per_label:]
    
    validation_imgs.append(shuffled_imgs[validation_indices])
    validation_labels.append(shuffled_labels[validation_indices])
    
    train_imgs.append(shuffled_imgs[train_indices])
    train_labels.append(shuffled_labels[train_indices])

# Reassemble the training and validation sets
train_imgs = np.concatenate(train_imgs, axis=0)
train_labels = np.concatenate(train_labels, axis=0)
validation_imgs = np.concatenate(validation_imgs, axis=0)
validation_labels = np.concatenate(validation_labels, axis=0)

DataClass = getattr(medmnist, info['python_class']) # Load each split train_data = DataClass(split='train', download=True) val_data = DataClass(split='val', download=True) import numpy as np from sklearn.model_selection import train_test_split # Combine the datasets all_imgs = np.concatenate((train_data.imgs, val_data.imgs), axis=0) all_labels = np.concatenate((train_data.labels, val_data.labels), axis=0) # Shuffle the dataset permutation = np.random.permutation(len(all_labels)) shuffled_imgs = all_imgs[permutation] shuffled_labels = all_labels[permutation] # Define the number of samples per label for the validation set validation_count_per_label = 750 unique_labels = [0, 1, 2, 3] # Defined set of unique labels # Initialize lists to hold the split data validation_imgs = [] validation_labels = [] train_imgs = [] train_labels = [] for label in unique_labels: label_indices = np.where(shuffled_labels == label)[0] # Check if there are enough samples for the label if len(label_indices) < validation_count_per_label: raise ValueError(f"Not enough samples for label {label}.") validation_indices = label_indices[:validation_count_per_label] train_indices = label_indices[validation_count_per_label:] validation_imgs.append(shuffled_imgs[validation_indices]) validation_labels.append(shuffled_labels[validation_indices]) train_imgs.append(shuffled_imgs[train_indices]) train_labels.append(shuffled_labels[train_indices]) # Reassemble the training and validation sets train_imgs = np.concatenate(train_imgs, axis=0) train_labels = np.concatenate(train_labels, axis=0) validation_imgs = np.concatenate(validation_imgs, axis=0) validation_labels = np.concatenate(validation_labels, axis=0)

Using downloaded and verified file: C:\Users\mvanr\.medmnist\octmnist.npz
Using downloaded and verified file: C:\Users\mvanr\.medmnist\octmnist.npz

In [16]:

import matplotlib.pyplot as plt

unique_labels, counts = np.unique(validation_labels, return_counts=True)
label_names = [info['label'][str(label)] for label in unique_labels]

# Plotting the distribution of labels
plt.figure(figsize=(10, 6))
plt.bar(label_names, counts, color='skyblue')
plt.xlabel('Labels')
plt.ylabel('Frequency')
plt.title('New Distribution of Labels in the Validation Dataset')
plt.xticks(rotation=45)
plt.show()

import matplotlib.pyplot as plt unique_labels, counts = np.unique(validation_labels, return_counts=True) label_names = [info['label'][str(label)] for label in unique_labels] # Plotting the distribution of labels plt.figure(figsize=(10, 6)) plt.bar(label_names, counts, color='skyblue') plt.xlabel('Labels') plt.ylabel('Frequency') plt.title('New Distribution of Labels in the Validation Dataset') plt.xticks(rotation=45) plt.show()

In [17]:

train_images, train_labels = augment_images_to_balance_classes(train_imgs, train_labels.flatten(), batch_size=32, seed=42)
train_images = train_images / 255.0

val_images = validation_imgs / 255.0
val_labels = validation_labels

train_images, train_labels = augment_images_to_balance_classes(train_imgs, train_labels.flatten(), batch_size=32, seed=42) train_images = train_images / 255.0 val_images = validation_imgs / 255.0 val_labels = validation_labels

In [18]:

train_images = train_images.reshape((len(train_images), 28*28))
val_images = val_images.reshape((len(val_images), 28*28))

train_images = train_images.reshape((len(train_images), 28*28)) val_images = val_images.reshape((len(val_images), 28*28))

Rerun initial training on rebalanced data¶

The initial training is re-done on the balanced dataset.

In [96]:

from tensorflow.keras import models, layers

class_weight_dict = {0: 1.0,1: 1.0, 2:1.0,3:1.0}

checkpoint = ModelCheckpoint(
    'best_model_A.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

network = models.Sequential()
network.add(layers.Dense(512, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='RMSprop', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=25, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers class_weight_dict = {0: 1.0,1: 1.0, 2:1.0,3:1.0} checkpoint = ModelCheckpoint( 'best_model_A.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) network = models.Sequential() network.add(layers.Dense(512, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='RMSprop', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=25, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/25
1575/1575 [==============================] - 5s 3ms/step - loss: 1.0901 - accuracy: 0.5191 - val_loss: 1.2743 - val_accuracy: 0.4537
Epoch 2/25
  47/1575 [..............................] - ETA: 5s - loss: 0.9853 - accuracy: 0.5836

c:\Projects\UoL\(DSM150-2023-OCT) Neural networks\CW1\.venv\Lib\site-packages\keras\src\engine\training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

1575/1575 [==============================] - 5s 3ms/step - loss: 0.9377 - accuracy: 0.6096 - val_loss: 1.2898 - val_accuracy: 0.5053
Epoch 3/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.8661 - accuracy: 0.6445 - val_loss: 1.2058 - val_accuracy: 0.5437
Epoch 4/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.8238 - accuracy: 0.6670 - val_loss: 0.9648 - val_accuracy: 0.5910
Epoch 5/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.7930 - accuracy: 0.6824 - val_loss: 1.0018 - val_accuracy: 0.5837
Epoch 6/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.7685 - accuracy: 0.6939 - val_loss: 0.9446 - val_accuracy: 0.6123
Epoch 7/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.7482 - accuracy: 0.7033 - val_loss: 1.0299 - val_accuracy: 0.6000
Epoch 8/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.7329 - accuracy: 0.7111 - val_loss: 1.1116 - val_accuracy: 0.5720
Epoch 9/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.7186 - accuracy: 0.7168 - val_loss: 1.0408 - val_accuracy: 0.6023
Epoch 10/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.7056 - accuracy: 0.7219 - val_loss: 0.9184 - val_accuracy: 0.6320
Epoch 11/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6943 - accuracy: 0.7273 - val_loss: 0.9852 - val_accuracy: 0.6243
Epoch 12/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6837 - accuracy: 0.7322 - val_loss: 0.8860 - val_accuracy: 0.6517
Epoch 13/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6748 - accuracy: 0.7369 - val_loss: 0.9100 - val_accuracy: 0.6400
Epoch 14/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6667 - accuracy: 0.7396 - val_loss: 0.9385 - val_accuracy: 0.6340
Epoch 15/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6588 - accuracy: 0.7442 - val_loss: 0.9421 - val_accuracy: 0.6327
Epoch 16/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6502 - accuracy: 0.7478 - val_loss: 1.1047 - val_accuracy: 0.5987
Epoch 17/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6433 - accuracy: 0.7496 - val_loss: 0.9460 - val_accuracy: 0.6450
Epoch 18/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6380 - accuracy: 0.7540 - val_loss: 0.9888 - val_accuracy: 0.6107
Epoch 19/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6332 - accuracy: 0.7553 - val_loss: 0.9338 - val_accuracy: 0.6543
Epoch 20/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6268 - accuracy: 0.7587 - val_loss: 0.9359 - val_accuracy: 0.6593
Epoch 21/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6220 - accuracy: 0.7609 - val_loss: 1.0371 - val_accuracy: 0.6330
Epoch 22/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6158 - accuracy: 0.7633 - val_loss: 0.9155 - val_accuracy: 0.6537
Epoch 23/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6114 - accuracy: 0.7653 - val_loss: 0.9885 - val_accuracy: 0.6500
Epoch 24/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6062 - accuracy: 0.7673 - val_loss: 0.9171 - val_accuracy: 0.6693
Epoch 25/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6028 - accuracy: 0.7690 - val_loss: 0.9647 - val_accuracy: 0.6523

In [97]:

plot_network_summaries(history, 'best_model_A.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_A.hdf5', val_images, val_labels)

Optimal point:  11 epochs
94/94 [==============================] - 0s 1ms/step

94/94 [==============================] - 0s 2ms/step - loss: 0.8860 - accuracy: 0.6517
Test accuracy: 0.6516666412353516

From the above, we can see that the drusen label still under-performs, but this is now also correctly reflected in the accuracy

Network A1 - Adam optimizer¶

The optimizer is changed to the "Adam" optimizer, resulting in some accuracy gain while the rest of the network stays constant.

In [99]:

from tensorflow.keras import models, layers
from tensorflow.keras.callbacks import ModelCheckpoint

checkpoint = ModelCheckpoint(
    'best_model_A1.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

class_weight_dict = {0: 1.0,1: 1.0, 2:1.0,3:1.0}

network = models.Sequential()
network.add(layers.Dense(512, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=25, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers from tensorflow.keras.callbacks import ModelCheckpoint checkpoint = ModelCheckpoint( 'best_model_A1.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) class_weight_dict = {0: 1.0,1: 1.0, 2:1.0,3:1.0} network = models.Sequential() network.add(layers.Dense(512, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=25, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/25
1575/1575 [==============================] - 6s 3ms/step - loss: 1.0616 - accuracy: 0.5352 - val_loss: 1.2005 - val_accuracy: 0.4893
Epoch 2/25
  31/1575 [..............................] - ETA: 5s - loss: 0.9705 - accuracy: 0.5905

c:\Projects\UoL\(DSM150-2023-OCT) Neural networks\CW1\.venv\Lib\site-packages\keras\src\engine\training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

1575/1575 [==============================] - 5s 3ms/step - loss: 0.9107 - accuracy: 0.6216 - val_loss: 1.0246 - val_accuracy: 0.5607
Epoch 3/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.8432 - accuracy: 0.6561 - val_loss: 1.1047 - val_accuracy: 0.5610
Epoch 4/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.8035 - accuracy: 0.6761 - val_loss: 0.9832 - val_accuracy: 0.5857
Epoch 5/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.7743 - accuracy: 0.6892 - val_loss: 1.0746 - val_accuracy: 0.5710
Epoch 6/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.7526 - accuracy: 0.7003 - val_loss: 0.9447 - val_accuracy: 0.6100
Epoch 7/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.7336 - accuracy: 0.7086 - val_loss: 0.9457 - val_accuracy: 0.6107
Epoch 8/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.7159 - accuracy: 0.7158 - val_loss: 0.9243 - val_accuracy: 0.6090
Epoch 9/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.7029 - accuracy: 0.7225 - val_loss: 0.8479 - val_accuracy: 0.6460
Epoch 10/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6895 - accuracy: 0.7276 - val_loss: 0.9313 - val_accuracy: 0.6197
Epoch 11/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6766 - accuracy: 0.7333 - val_loss: 0.8911 - val_accuracy: 0.6290
Epoch 12/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6677 - accuracy: 0.7385 - val_loss: 0.9064 - val_accuracy: 0.6277
Epoch 13/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6587 - accuracy: 0.7415 - val_loss: 0.9324 - val_accuracy: 0.6327
Epoch 14/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6488 - accuracy: 0.7460 - val_loss: 0.8695 - val_accuracy: 0.6463
Epoch 15/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6405 - accuracy: 0.7499 - val_loss: 0.8224 - val_accuracy: 0.6617
Epoch 16/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6348 - accuracy: 0.7520 - val_loss: 0.9299 - val_accuracy: 0.6313
Epoch 17/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6242 - accuracy: 0.7559 - val_loss: 0.8899 - val_accuracy: 0.6433
Epoch 18/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6213 - accuracy: 0.7586 - val_loss: 0.8571 - val_accuracy: 0.6487
Epoch 19/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6136 - accuracy: 0.7604 - val_loss: 1.0085 - val_accuracy: 0.6290
Epoch 20/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6087 - accuracy: 0.7630 - val_loss: 0.8252 - val_accuracy: 0.6600
Epoch 21/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.6044 - accuracy: 0.7648 - val_loss: 0.8945 - val_accuracy: 0.6470
Epoch 22/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.5986 - accuracy: 0.7658 - val_loss: 0.8578 - val_accuracy: 0.6533
Epoch 23/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.5932 - accuracy: 0.7696 - val_loss: 0.8237 - val_accuracy: 0.6663
Epoch 24/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.5885 - accuracy: 0.7714 - val_loss: 0.8735 - val_accuracy: 0.6530
Epoch 25/25
1575/1575 [==============================] - 5s 3ms/step - loss: 0.5829 - accuracy: 0.7739 - val_loss: 0.8437 - val_accuracy: 0.6597

In [100]:

plot_network_summaries(history, 'best_model_A1.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_A1.hdf5', val_images, val_labels)

Optimal point:  14 epochs
94/94 [==============================] - 0s 1ms/step

94/94 [==============================] - 0s 1ms/step - loss: 0.8224 - accuracy: 0.6617
Test accuracy: 0.6616666913032532

Network B¶

A new, deeper network is constructed in order to determine an over-fitted baseline.

This network consists of dense layers with 1024, 512, 256, 64 neurons respectively.

In [103]:

from tensorflow.keras import models, layers

checkpoint = ModelCheckpoint(
    'best_model_B.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

network = models.Sequential()
network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(512, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(256, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(64, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=50, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers checkpoint = ModelCheckpoint( 'best_model_B.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) network = models.Sequential() network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(512, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(256, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(64, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=50, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/50
1575/1575 [==============================] - 14s 9ms/step - loss: 1.0325 - accuracy: 0.5440 - val_loss: 1.2274 - val_accuracy: 0.4887
Epoch 2/50
  13/1575 [..............................] - ETA: 13s - loss: 0.9147 - accuracy: 0.6166

c:\Projects\UoL\(DSM150-2023-OCT) Neural networks\CW1\.venv\Lib\site-packages\keras\src\engine\training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

1575/1575 [==============================] - 14s 9ms/step - loss: 0.8553 - accuracy: 0.6451 - val_loss: 1.0513 - val_accuracy: 0.5647
Epoch 3/50
1575/1575 [==============================] - 13s 9ms/step - loss: 0.7802 - accuracy: 0.6816 - val_loss: 1.0006 - val_accuracy: 0.5827
Epoch 4/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.7341 - accuracy: 0.7035 - val_loss: 0.9759 - val_accuracy: 0.5947
Epoch 5/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.6995 - accuracy: 0.7208 - val_loss: 0.9221 - val_accuracy: 0.6070
Epoch 6/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.6736 - accuracy: 0.7313 - val_loss: 1.1278 - val_accuracy: 0.5887
Epoch 7/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.6496 - accuracy: 0.7431 - val_loss: 0.8857 - val_accuracy: 0.6290
Epoch 8/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.6272 - accuracy: 0.7517 - val_loss: 0.8575 - val_accuracy: 0.6403
Epoch 9/50
1575/1575 [==============================] - 13s 9ms/step - loss: 0.6078 - accuracy: 0.7614 - val_loss: 0.8940 - val_accuracy: 0.6373
Epoch 10/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.5902 - accuracy: 0.7670 - val_loss: 0.8627 - val_accuracy: 0.6350
Epoch 11/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.5721 - accuracy: 0.7749 - val_loss: 0.8257 - val_accuracy: 0.6613
Epoch 12/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.5562 - accuracy: 0.7819 - val_loss: 0.8317 - val_accuracy: 0.6640
Epoch 13/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.5409 - accuracy: 0.7876 - val_loss: 0.8179 - val_accuracy: 0.6683
Epoch 14/50
1575/1575 [==============================] - 13s 8ms/step - loss: 0.5247 - accuracy: 0.7948 - val_loss: 0.8111 - val_accuracy: 0.6723
Epoch 15/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.5109 - accuracy: 0.8002 - val_loss: 0.7991 - val_accuracy: 0.6757
Epoch 16/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.4964 - accuracy: 0.8056 - val_loss: 0.8666 - val_accuracy: 0.6643
Epoch 17/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.4846 - accuracy: 0.8103 - val_loss: 0.7998 - val_accuracy: 0.6790
Epoch 18/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.4715 - accuracy: 0.8153 - val_loss: 0.8028 - val_accuracy: 0.6883
Epoch 19/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.4579 - accuracy: 0.8213 - val_loss: 0.8011 - val_accuracy: 0.6837
Epoch 20/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.4465 - accuracy: 0.8268 - val_loss: 0.8272 - val_accuracy: 0.6893
Epoch 21/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.4361 - accuracy: 0.8293 - val_loss: 0.7911 - val_accuracy: 0.6957
Epoch 22/50
1575/1575 [==============================] - 13s 8ms/step - loss: 0.4241 - accuracy: 0.8347 - val_loss: 0.7752 - val_accuracy: 0.7060
Epoch 23/50
1575/1575 [==============================] - 13s 8ms/step - loss: 0.4143 - accuracy: 0.8384 - val_loss: 0.9113 - val_accuracy: 0.6707
Epoch 24/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.4035 - accuracy: 0.8427 - val_loss: 0.7716 - val_accuracy: 0.7117
Epoch 25/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.3931 - accuracy: 0.8461 - val_loss: 0.7725 - val_accuracy: 0.7100
Epoch 26/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.3823 - accuracy: 0.8510 - val_loss: 0.8435 - val_accuracy: 0.7013
Epoch 27/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.3740 - accuracy: 0.8542 - val_loss: 0.8173 - val_accuracy: 0.7033
Epoch 28/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.3615 - accuracy: 0.8591 - val_loss: 0.7721 - val_accuracy: 0.7317
Epoch 29/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.3571 - accuracy: 0.8609 - val_loss: 0.7857 - val_accuracy: 0.7197
Epoch 30/50
1575/1575 [==============================] - 15s 9ms/step - loss: 0.3469 - accuracy: 0.8649 - val_loss: 0.8514 - val_accuracy: 0.7063
Epoch 31/50
1575/1575 [==============================] - 15s 9ms/step - loss: 0.3395 - accuracy: 0.8680 - val_loss: 0.8294 - val_accuracy: 0.7173
Epoch 32/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.3296 - accuracy: 0.8717 - val_loss: 0.8244 - val_accuracy: 0.7243
Epoch 33/50
1575/1575 [==============================] - 15s 9ms/step - loss: 0.3231 - accuracy: 0.8744 - val_loss: 0.8070 - val_accuracy: 0.7227
Epoch 34/50
1575/1575 [==============================] - 15s 9ms/step - loss: 0.3171 - accuracy: 0.8763 - val_loss: 0.8838 - val_accuracy: 0.7187
Epoch 35/50
1575/1575 [==============================] - 15s 9ms/step - loss: 0.3095 - accuracy: 0.8807 - val_loss: 0.8386 - val_accuracy: 0.7250
Epoch 36/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.3023 - accuracy: 0.8832 - val_loss: 0.8307 - val_accuracy: 0.7270
Epoch 37/50
1575/1575 [==============================] - 15s 9ms/step - loss: 0.2950 - accuracy: 0.8855 - val_loss: 0.8800 - val_accuracy: 0.7163
Epoch 38/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.2897 - accuracy: 0.8880 - val_loss: 0.8694 - val_accuracy: 0.7223
Epoch 39/50
1575/1575 [==============================] - 15s 9ms/step - loss: 0.2821 - accuracy: 0.8916 - val_loss: 0.8591 - val_accuracy: 0.7347
Epoch 40/50
1575/1575 [==============================] - 14s 9ms/step - loss: 0.2763 - accuracy: 0.8934 - val_loss: 0.8840 - val_accuracy: 0.7340
Epoch 41/50
1575/1575 [==============================] - 15s 9ms/step - loss: 0.2700 - accuracy: 0.8960 - val_loss: 0.8825 - val_accuracy: 0.7250
Epoch 42/50
1575/1575 [==============================] - 15s 9ms/step - loss: 0.2624 - accuracy: 0.8987 - val_loss: 0.8277 - val_accuracy: 0.7410
Epoch 43/50
1575/1575 [==============================] - 15s 10ms/step - loss: 0.2594 - accuracy: 0.9008 - val_loss: 0.8232 - val_accuracy: 0.7523
Epoch 44/50
1575/1575 [==============================] - 15s 9ms/step - loss: 0.2522 - accuracy: 0.9023 - val_loss: 0.9202 - val_accuracy: 0.7350
Epoch 45/50
1575/1575 [==============================] - 15s 9ms/step - loss: 0.2497 - accuracy: 0.9036 - val_loss: 0.9083 - val_accuracy: 0.7393
Epoch 46/50
1575/1575 [==============================] - 15s 9ms/step - loss: 0.2436 - accuracy: 0.9065 - val_loss: 0.9109 - val_accuracy: 0.7403
Epoch 47/50
1575/1575 [==============================] - 15s 10ms/step - loss: 0.2394 - accuracy: 0.9072 - val_loss: 0.8838 - val_accuracy: 0.7427
Epoch 48/50
1575/1575 [==============================] - 15s 10ms/step - loss: 0.2330 - accuracy: 0.9106 - val_loss: 0.9397 - val_accuracy: 0.7337
Epoch 49/50
1575/1575 [==============================] - 15s 9ms/step - loss: 0.2285 - accuracy: 0.9121 - val_loss: 0.9029 - val_accuracy: 0.7517
Epoch 50/50
1575/1575 [==============================] - 15s 10ms/step - loss: 0.2236 - accuracy: 0.9139 - val_loss: 0.9403 - val_accuracy: 0.7363

In [108]:

plot_network_summaries(history, 'best_model_B.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_B.hdf5', val_images, val_labels)

Optimal point:  24 epochs
94/94 [==============================] - 0s 3ms/step

94/94 [==============================] - 0s 3ms/step - loss: 0.7716 - accuracy: 0.7117
Test accuracy: 0.7116666436195374

This new network improves the accuracy further, but overfits at 24 epochs.

Drusen is still frequently predicted as normal.

B1 - Add Dropout¶

For this iteration of the network a dropout of 0.2 is added to each layer in order to reduce overfiting.

In [21]:

from tensorflow.keras import models, layers

checkpoint = ModelCheckpoint(
    'best_model_B1.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

network = models.Sequential()
network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, )))
network.add(layers.Dropout(0.2))
network.add(layers.Dense(512, activation='relu'))
network.add(layers.Dropout(0.2))
network.add(layers.Dense(256, activation='relu'))
network.add(layers.Dropout(0.2))
network.add(layers.Dense(64, activation='relu'))
network.add(layers.Dropout(0.2))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=150, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers checkpoint = ModelCheckpoint( 'best_model_B1.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) network = models.Sequential() network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, ))) network.add(layers.Dropout(0.2)) network.add(layers.Dense(512, activation='relu')) network.add(layers.Dropout(0.2)) network.add(layers.Dense(256, activation='relu')) network.add(layers.Dropout(0.2)) network.add(layers.Dense(64, activation='relu')) network.add(layers.Dropout(0.2)) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=150, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/150
1573/1575 [============================>.] - ETA: 0s - loss: 0.4369 - accuracy: 0.3745

c:\Projects\UoL\(DSM150-2023-OCT) Neural networks\CW1\.venv\Lib\site-packages\keras\src\engine\training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

1575/1575 [==============================] - 21s 12ms/step - loss: 0.4369 - accuracy: 0.3745 - val_loss: 1.2823 - val_accuracy: 0.4060
Epoch 2/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.3974 - accuracy: 0.4553 - val_loss: 1.2131 - val_accuracy: 0.4667
Epoch 3/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.3791 - accuracy: 0.4947 - val_loss: 1.1473 - val_accuracy: 0.5053
Epoch 4/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.3659 - accuracy: 0.5159 - val_loss: 1.0950 - val_accuracy: 0.5390
Epoch 5/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.3571 - accuracy: 0.5320 - val_loss: 1.0361 - val_accuracy: 0.5680
Epoch 6/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.3503 - accuracy: 0.5411 - val_loss: 1.0273 - val_accuracy: 0.5707
Epoch 7/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.3445 - accuracy: 0.5512 - val_loss: 1.0736 - val_accuracy: 0.5527
Epoch 8/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.3394 - accuracy: 0.5593 - val_loss: 1.0486 - val_accuracy: 0.5590
Epoch 9/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.3349 - accuracy: 0.5667 - val_loss: 1.0103 - val_accuracy: 0.5730
Epoch 10/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.3309 - accuracy: 0.5758 - val_loss: 0.9633 - val_accuracy: 0.5963
Epoch 11/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.3273 - accuracy: 0.5822 - val_loss: 0.9562 - val_accuracy: 0.6110
Epoch 12/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.3238 - accuracy: 0.5872 - val_loss: 0.9739 - val_accuracy: 0.6047
Epoch 13/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.3211 - accuracy: 0.5907 - val_loss: 0.9788 - val_accuracy: 0.5933
Epoch 14/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.3183 - accuracy: 0.5925 - val_loss: 0.9540 - val_accuracy: 0.6107
Epoch 15/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.3161 - accuracy: 0.5972 - val_loss: 0.9327 - val_accuracy: 0.6287
Epoch 16/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.3136 - accuracy: 0.6004 - val_loss: 0.9274 - val_accuracy: 0.6250
Epoch 17/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.3118 - accuracy: 0.6043 - val_loss: 0.9385 - val_accuracy: 0.6227
Epoch 18/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.3096 - accuracy: 0.6066 - val_loss: 0.9347 - val_accuracy: 0.6173
Epoch 19/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.3069 - accuracy: 0.6102 - val_loss: 0.9138 - val_accuracy: 0.6250
Epoch 20/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.3062 - accuracy: 0.6119 - val_loss: 0.8822 - val_accuracy: 0.6483
Epoch 21/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.3050 - accuracy: 0.6158 - val_loss: 0.9034 - val_accuracy: 0.6333
Epoch 22/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.3025 - accuracy: 0.6181 - val_loss: 0.9013 - val_accuracy: 0.6293
Epoch 23/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.3008 - accuracy: 0.6180 - val_loss: 0.8847 - val_accuracy: 0.6447
Epoch 24/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2997 - accuracy: 0.6214 - val_loss: 0.8720 - val_accuracy: 0.6487
Epoch 25/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.2988 - accuracy: 0.6228 - val_loss: 0.9381 - val_accuracy: 0.6200
Epoch 26/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2968 - accuracy: 0.6235 - val_loss: 0.8703 - val_accuracy: 0.6400
Epoch 27/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.2965 - accuracy: 0.6249 - val_loss: 0.8518 - val_accuracy: 0.6477
Epoch 28/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2943 - accuracy: 0.6287 - val_loss: 0.8714 - val_accuracy: 0.6427
Epoch 29/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2932 - accuracy: 0.6302 - val_loss: 0.8950 - val_accuracy: 0.6320
Epoch 30/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2930 - accuracy: 0.6320 - val_loss: 0.8622 - val_accuracy: 0.6543
Epoch 31/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2912 - accuracy: 0.6342 - val_loss: 0.8445 - val_accuracy: 0.6603
Epoch 32/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.2904 - accuracy: 0.6341 - val_loss: 0.8677 - val_accuracy: 0.6483
Epoch 33/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2891 - accuracy: 0.6366 - val_loss: 0.8577 - val_accuracy: 0.6520
Epoch 34/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2885 - accuracy: 0.6380 - val_loss: 0.8670 - val_accuracy: 0.6443
Epoch 35/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2874 - accuracy: 0.6399 - val_loss: 0.8623 - val_accuracy: 0.6550
Epoch 36/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2860 - accuracy: 0.6401 - val_loss: 0.8464 - val_accuracy: 0.6637
Epoch 37/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2856 - accuracy: 0.6410 - val_loss: 0.8690 - val_accuracy: 0.6520
Epoch 38/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2842 - accuracy: 0.6429 - val_loss: 0.8402 - val_accuracy: 0.6623
Epoch 39/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2838 - accuracy: 0.6437 - val_loss: 0.8466 - val_accuracy: 0.6540
Epoch 40/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2824 - accuracy: 0.6449 - val_loss: 0.8414 - val_accuracy: 0.6680
Epoch 41/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.2809 - accuracy: 0.6468 - val_loss: 0.8636 - val_accuracy: 0.6460
Epoch 42/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2818 - accuracy: 0.6467 - val_loss: 0.8262 - val_accuracy: 0.6597
Epoch 43/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.2807 - accuracy: 0.6484 - val_loss: 0.8486 - val_accuracy: 0.6633
Epoch 44/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2796 - accuracy: 0.6503 - val_loss: 0.8196 - val_accuracy: 0.6787
Epoch 45/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.2793 - accuracy: 0.6493 - val_loss: 0.8793 - val_accuracy: 0.6527
Epoch 46/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2777 - accuracy: 0.6510 - val_loss: 0.8523 - val_accuracy: 0.6670
Epoch 47/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2780 - accuracy: 0.6513 - val_loss: 0.8304 - val_accuracy: 0.6633
Epoch 48/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2779 - accuracy: 0.6512 - val_loss: 0.8484 - val_accuracy: 0.6623
Epoch 49/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2757 - accuracy: 0.6549 - val_loss: 0.8480 - val_accuracy: 0.6590
Epoch 50/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2760 - accuracy: 0.6542 - val_loss: 0.8300 - val_accuracy: 0.6627
Epoch 51/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2758 - accuracy: 0.6556 - val_loss: 0.8097 - val_accuracy: 0.6793
Epoch 52/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.2741 - accuracy: 0.6564 - val_loss: 0.8627 - val_accuracy: 0.6487
Epoch 53/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2733 - accuracy: 0.6575 - val_loss: 0.8122 - val_accuracy: 0.6803
Epoch 54/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2730 - accuracy: 0.6573 - val_loss: 0.8283 - val_accuracy: 0.6667
Epoch 55/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2722 - accuracy: 0.6591 - val_loss: 0.8148 - val_accuracy: 0.6697
Epoch 56/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2731 - accuracy: 0.6561 - val_loss: 0.8473 - val_accuracy: 0.6570
Epoch 57/150
1575/1575 [==============================] - 20s 12ms/step - loss: 0.2705 - accuracy: 0.6627 - val_loss: 0.8563 - val_accuracy: 0.6593
Epoch 58/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2709 - accuracy: 0.6610 - val_loss: 0.8490 - val_accuracy: 0.6623
Epoch 59/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2701 - accuracy: 0.6625 - val_loss: 0.8607 - val_accuracy: 0.6390
Epoch 60/150
1575/1575 [==============================] - 21s 13ms/step - loss: 0.2693 - accuracy: 0.6627 - val_loss: 0.8610 - val_accuracy: 0.6520
Epoch 61/150
1575/1575 [==============================] - 20s 12ms/step - loss: 0.2700 - accuracy: 0.6606 - val_loss: 0.8002 - val_accuracy: 0.6767
Epoch 62/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2692 - accuracy: 0.6623 - val_loss: 0.7986 - val_accuracy: 0.6757
Epoch 63/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2684 - accuracy: 0.6638 - val_loss: 0.8121 - val_accuracy: 0.6747
Epoch 64/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2686 - accuracy: 0.6646 - val_loss: 0.8252 - val_accuracy: 0.6667
Epoch 65/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2666 - accuracy: 0.6667 - val_loss: 0.7880 - val_accuracy: 0.6907
Epoch 66/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2668 - accuracy: 0.6678 - val_loss: 0.8129 - val_accuracy: 0.6777
Epoch 67/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2673 - accuracy: 0.6668 - val_loss: 0.8444 - val_accuracy: 0.6687
Epoch 68/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2660 - accuracy: 0.6689 - val_loss: 0.7808 - val_accuracy: 0.6967
Epoch 69/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2662 - accuracy: 0.6685 - val_loss: 0.8183 - val_accuracy: 0.6623
Epoch 70/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2655 - accuracy: 0.6689 - val_loss: 0.8294 - val_accuracy: 0.6680
Epoch 71/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2646 - accuracy: 0.6685 - val_loss: 0.7910 - val_accuracy: 0.6890
Epoch 72/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2651 - accuracy: 0.6696 - val_loss: 0.7909 - val_accuracy: 0.6883
Epoch 73/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2642 - accuracy: 0.6703 - val_loss: 0.8102 - val_accuracy: 0.6657
Epoch 74/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2636 - accuracy: 0.6706 - val_loss: 0.8216 - val_accuracy: 0.6747
Epoch 75/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2623 - accuracy: 0.6730 - val_loss: 0.8134 - val_accuracy: 0.6773
Epoch 76/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2626 - accuracy: 0.6702 - val_loss: 0.8186 - val_accuracy: 0.6570
Epoch 77/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2625 - accuracy: 0.6724 - val_loss: 0.7775 - val_accuracy: 0.6983
Epoch 78/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.2624 - accuracy: 0.6723 - val_loss: 0.8231 - val_accuracy: 0.6700
Epoch 79/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2626 - accuracy: 0.6712 - val_loss: 0.7681 - val_accuracy: 0.6997
Epoch 80/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.2603 - accuracy: 0.6756 - val_loss: 0.8165 - val_accuracy: 0.6750
Epoch 81/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.2609 - accuracy: 0.6747 - val_loss: 0.7855 - val_accuracy: 0.6833
Epoch 82/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2607 - accuracy: 0.6739 - val_loss: 0.7629 - val_accuracy: 0.6917
Epoch 83/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2606 - accuracy: 0.6750 - val_loss: 0.8008 - val_accuracy: 0.6770
Epoch 84/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2605 - accuracy: 0.6741 - val_loss: 0.8234 - val_accuracy: 0.6807
Epoch 85/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2604 - accuracy: 0.6735 - val_loss: 0.8043 - val_accuracy: 0.6860
Epoch 86/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2595 - accuracy: 0.6775 - val_loss: 0.7817 - val_accuracy: 0.6847
Epoch 87/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2591 - accuracy: 0.6780 - val_loss: 0.7983 - val_accuracy: 0.6723
Epoch 88/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2595 - accuracy: 0.6753 - val_loss: 0.8509 - val_accuracy: 0.6593
Epoch 89/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2582 - accuracy: 0.6769 - val_loss: 0.7877 - val_accuracy: 0.6817
Epoch 90/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2579 - accuracy: 0.6788 - val_loss: 0.7990 - val_accuracy: 0.6803
Epoch 91/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2581 - accuracy: 0.6780 - val_loss: 0.7676 - val_accuracy: 0.6813
Epoch 92/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2576 - accuracy: 0.6784 - val_loss: 0.7897 - val_accuracy: 0.6880
Epoch 93/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2574 - accuracy: 0.6812 - val_loss: 0.7663 - val_accuracy: 0.6873
Epoch 94/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2565 - accuracy: 0.6809 - val_loss: 0.7960 - val_accuracy: 0.6713
Epoch 95/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2565 - accuracy: 0.6807 - val_loss: 0.7741 - val_accuracy: 0.6900
Epoch 96/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2558 - accuracy: 0.6805 - val_loss: 0.8148 - val_accuracy: 0.6663
Epoch 97/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2559 - accuracy: 0.6795 - val_loss: 0.7776 - val_accuracy: 0.6930
Epoch 98/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2562 - accuracy: 0.6805 - val_loss: 0.7813 - val_accuracy: 0.6933
Epoch 99/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2554 - accuracy: 0.6821 - val_loss: 0.7974 - val_accuracy: 0.6733
Epoch 100/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2552 - accuracy: 0.6830 - val_loss: 0.7746 - val_accuracy: 0.6937
Epoch 101/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2543 - accuracy: 0.6822 - val_loss: 0.7838 - val_accuracy: 0.6910
Epoch 102/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2548 - accuracy: 0.6822 - val_loss: 0.7711 - val_accuracy: 0.6867
Epoch 103/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2540 - accuracy: 0.6832 - val_loss: 0.7735 - val_accuracy: 0.6820
Epoch 104/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2544 - accuracy: 0.6821 - val_loss: 0.8034 - val_accuracy: 0.6860
Epoch 105/150
1575/1575 [==============================] - 20s 12ms/step - loss: 0.2545 - accuracy: 0.6823 - val_loss: 0.7592 - val_accuracy: 0.7007
Epoch 106/150
1575/1575 [==============================] - 17s 11ms/step - loss: 0.2533 - accuracy: 0.6850 - val_loss: 0.8154 - val_accuracy: 0.6673
Epoch 107/150
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2541 - accuracy: 0.6836 - val_loss: 0.8109 - val_accuracy: 0.6630
Epoch 108/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2533 - accuracy: 0.6861 - val_loss: 0.7971 - val_accuracy: 0.6733
Epoch 109/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2524 - accuracy: 0.6853 - val_loss: 0.7665 - val_accuracy: 0.7017
Epoch 110/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2521 - accuracy: 0.6856 - val_loss: 0.7914 - val_accuracy: 0.6797
Epoch 111/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2528 - accuracy: 0.6838 - val_loss: 0.7771 - val_accuracy: 0.6810
Epoch 112/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2512 - accuracy: 0.6858 - val_loss: 0.7801 - val_accuracy: 0.6937
Epoch 113/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2519 - accuracy: 0.6859 - val_loss: 0.7816 - val_accuracy: 0.6750
Epoch 114/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2519 - accuracy: 0.6865 - val_loss: 0.8193 - val_accuracy: 0.6693
Epoch 115/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2509 - accuracy: 0.6854 - val_loss: 0.7862 - val_accuracy: 0.6800
Epoch 116/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2511 - accuracy: 0.6851 - val_loss: 0.7871 - val_accuracy: 0.6820
Epoch 117/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2510 - accuracy: 0.6864 - val_loss: 0.7825 - val_accuracy: 0.6930
Epoch 118/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2506 - accuracy: 0.6867 - val_loss: 0.8163 - val_accuracy: 0.6703
Epoch 119/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2512 - accuracy: 0.6864 - val_loss: 0.8317 - val_accuracy: 0.6627
Epoch 120/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2497 - accuracy: 0.6879 - val_loss: 0.7637 - val_accuracy: 0.6947
Epoch 121/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2507 - accuracy: 0.6874 - val_loss: 0.7760 - val_accuracy: 0.6967
Epoch 122/150
1575/1575 [==============================] - 21s 13ms/step - loss: 0.2508 - accuracy: 0.6891 - val_loss: 0.7947 - val_accuracy: 0.6883
Epoch 123/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2500 - accuracy: 0.6874 - val_loss: 0.7703 - val_accuracy: 0.6937
Epoch 124/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2495 - accuracy: 0.6892 - val_loss: 0.7830 - val_accuracy: 0.6940
Epoch 125/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2494 - accuracy: 0.6890 - val_loss: 0.7497 - val_accuracy: 0.7067
Epoch 126/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2485 - accuracy: 0.6914 - val_loss: 0.7757 - val_accuracy: 0.6930
Epoch 127/150
1575/1575 [==============================] - 18s 12ms/step - loss: 0.2488 - accuracy: 0.6910 - val_loss: 0.7857 - val_accuracy: 0.6900
Epoch 128/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2480 - accuracy: 0.6902 - val_loss: 0.7629 - val_accuracy: 0.6900
Epoch 129/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2481 - accuracy: 0.6905 - val_loss: 0.7853 - val_accuracy: 0.6900
Epoch 130/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2485 - accuracy: 0.6894 - val_loss: 0.7842 - val_accuracy: 0.6863
Epoch 131/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2473 - accuracy: 0.6910 - val_loss: 0.7947 - val_accuracy: 0.6697
Epoch 132/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2480 - accuracy: 0.6913 - val_loss: 0.8054 - val_accuracy: 0.6850
Epoch 133/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2472 - accuracy: 0.6934 - val_loss: 0.7764 - val_accuracy: 0.6940
Epoch 134/150
1575/1575 [==============================] - 21s 13ms/step - loss: 0.2473 - accuracy: 0.6916 - val_loss: 0.7576 - val_accuracy: 0.7027
Epoch 135/150
1575/1575 [==============================] - 21s 13ms/step - loss: 0.2472 - accuracy: 0.6919 - val_loss: 0.7805 - val_accuracy: 0.6793
Epoch 136/150
1575/1575 [==============================] - 21s 13ms/step - loss: 0.2472 - accuracy: 0.6920 - val_loss: 0.7447 - val_accuracy: 0.7107
Epoch 137/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2473 - accuracy: 0.6918 - val_loss: 0.7502 - val_accuracy: 0.7063
Epoch 138/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2465 - accuracy: 0.6920 - val_loss: 0.7873 - val_accuracy: 0.6860
Epoch 139/150
1575/1575 [==============================] - 20s 12ms/step - loss: 0.2458 - accuracy: 0.6928 - val_loss: 0.7468 - val_accuracy: 0.7043
Epoch 140/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2452 - accuracy: 0.6946 - val_loss: 0.8321 - val_accuracy: 0.6537
Epoch 141/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2460 - accuracy: 0.6923 - val_loss: 0.7536 - val_accuracy: 0.7010
Epoch 142/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2451 - accuracy: 0.6940 - val_loss: 0.7764 - val_accuracy: 0.6823
Epoch 143/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2458 - accuracy: 0.6937 - val_loss: 0.7885 - val_accuracy: 0.6753
Epoch 144/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2456 - accuracy: 0.6938 - val_loss: 0.7727 - val_accuracy: 0.6940
Epoch 145/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2465 - accuracy: 0.6923 - val_loss: 0.7695 - val_accuracy: 0.6933
Epoch 146/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2450 - accuracy: 0.6947 - val_loss: 0.7799 - val_accuracy: 0.6953
Epoch 147/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2441 - accuracy: 0.6941 - val_loss: 0.7937 - val_accuracy: 0.6723
Epoch 148/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2459 - accuracy: 0.6936 - val_loss: 0.7560 - val_accuracy: 0.6977
Epoch 149/150
1575/1575 [==============================] - 19s 12ms/step - loss: 0.2455 - accuracy: 0.6943 - val_loss: 0.7659 - val_accuracy: 0.6937
Epoch 150/150
1575/1575 [==============================] - 20s 13ms/step - loss: 0.2452 - accuracy: 0.6941 - val_loss: 0.7243 - val_accuracy: 0.7177

In [24]:

plot_network_summaries(history, 'best_model_B1.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_B1.hdf5', val_images, val_labels)

Optimal point:  150 epochs
94/94 [==============================] - 0s 3ms/step

94/94 [==============================] - 0s 3ms/step - loss: 0.7243 - accuracy: 0.7177
Test accuracy: 0.7176666855812073

This iteration does not seem to reach a phase of overfitting within 150 epochs, but does not present clear accuracy benefit over that of the "B" model.

This model does however present some benefit in the classification of the drusen label.

B2 - No Dropout, with square layers¶

The dropout is removed, and three layers of 1024 neurons each is used.

In [30]:

from tensorflow.keras import models, layers

checkpoint = ModelCheckpoint(
    'best_model_B1-2.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

network = models.Sequential()
network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(1024, activation='relu'))
network.add(layers.Dense(1024, activation='relu'))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=50, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers checkpoint = ModelCheckpoint( 'best_model_B1-2.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) network = models.Sequential() network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(1024, activation='relu')) network.add(layers.Dense(1024, activation='relu')) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=50, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/50
1575/1575 [==============================] - 29s 18ms/step - loss: 0.4091 - accuracy: 0.4340 - val_loss: 1.2176 - val_accuracy: 0.4643
Epoch 2/50
   1/1575 [..............................] - ETA: 31s - loss: 0.3991 - accuracy: 0.4219

c:\Projects\UoL\(DSM150-2023-OCT) Neural networks\CW1\.venv\Lib\site-packages\keras\src\engine\training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

1575/1575 [==============================] - 27s 17ms/step - loss: 0.3555 - accuracy: 0.5359 - val_loss: 1.0603 - val_accuracy: 0.5490
Epoch 3/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.3269 - accuracy: 0.5825 - val_loss: 0.9937 - val_accuracy: 0.5763
Epoch 4/50
1575/1575 [==============================] - 27s 17ms/step - loss: 0.3085 - accuracy: 0.6121 - val_loss: 0.9808 - val_accuracy: 0.5960
Epoch 5/50
1575/1575 [==============================] - 27s 17ms/step - loss: 0.2949 - accuracy: 0.6314 - val_loss: 0.8779 - val_accuracy: 0.6400
Epoch 6/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.2843 - accuracy: 0.6475 - val_loss: 0.9046 - val_accuracy: 0.6300
Epoch 7/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.2748 - accuracy: 0.6602 - val_loss: 0.8724 - val_accuracy: 0.6520
Epoch 8/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.2658 - accuracy: 0.6741 - val_loss: 0.8488 - val_accuracy: 0.6637
Epoch 9/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.2580 - accuracy: 0.6833 - val_loss: 0.8204 - val_accuracy: 0.6730
Epoch 10/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.2499 - accuracy: 0.6945 - val_loss: 0.8076 - val_accuracy: 0.6853
Epoch 11/50
1575/1575 [==============================] - 27s 17ms/step - loss: 0.2426 - accuracy: 0.7036 - val_loss: 0.7974 - val_accuracy: 0.6860
Epoch 12/50
1575/1575 [==============================] - 27s 17ms/step - loss: 0.2350 - accuracy: 0.7124 - val_loss: 0.7832 - val_accuracy: 0.6970
Epoch 13/50
1575/1575 [==============================] - 27s 17ms/step - loss: 0.2272 - accuracy: 0.7241 - val_loss: 0.7871 - val_accuracy: 0.7053
Epoch 14/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.2204 - accuracy: 0.7314 - val_loss: 0.7893 - val_accuracy: 0.6953
Epoch 15/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.2130 - accuracy: 0.7403 - val_loss: 0.7879 - val_accuracy: 0.7027
Epoch 16/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.2054 - accuracy: 0.7500 - val_loss: 0.7333 - val_accuracy: 0.7230
Epoch 17/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.1976 - accuracy: 0.7591 - val_loss: 0.7250 - val_accuracy: 0.7303
Epoch 18/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.1906 - accuracy: 0.7675 - val_loss: 0.7878 - val_accuracy: 0.7130
Epoch 19/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.1839 - accuracy: 0.7754 - val_loss: 0.7472 - val_accuracy: 0.7340
Epoch 20/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.1769 - accuracy: 0.7848 - val_loss: 0.8115 - val_accuracy: 0.7283
Epoch 21/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.1696 - accuracy: 0.7929 - val_loss: 0.7622 - val_accuracy: 0.7323
Epoch 22/50
1575/1575 [==============================] - 30s 19ms/step - loss: 0.1637 - accuracy: 0.8000 - val_loss: 0.7864 - val_accuracy: 0.7270
Epoch 23/50
1575/1575 [==============================] - 29s 18ms/step - loss: 0.1577 - accuracy: 0.8083 - val_loss: 0.7823 - val_accuracy: 0.7337
Epoch 24/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.1505 - accuracy: 0.8157 - val_loss: 0.8279 - val_accuracy: 0.7250
Epoch 25/50
1575/1575 [==============================] - 29s 18ms/step - loss: 0.1430 - accuracy: 0.8251 - val_loss: 0.8375 - val_accuracy: 0.7357
Epoch 26/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.1383 - accuracy: 0.8305 - val_loss: 0.8097 - val_accuracy: 0.7477
Epoch 27/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.1323 - accuracy: 0.8370 - val_loss: 0.8050 - val_accuracy: 0.7477
Epoch 28/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.1279 - accuracy: 0.8427 - val_loss: 0.8300 - val_accuracy: 0.7547
Epoch 29/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.1223 - accuracy: 0.8488 - val_loss: 0.8360 - val_accuracy: 0.7523
Epoch 30/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.1168 - accuracy: 0.8559 - val_loss: 0.8667 - val_accuracy: 0.7430
Epoch 31/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.1125 - accuracy: 0.8602 - val_loss: 0.8800 - val_accuracy: 0.7513
Epoch 32/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.1086 - accuracy: 0.8645 - val_loss: 0.8644 - val_accuracy: 0.7587
Epoch 33/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.1069 - accuracy: 0.8672 - val_loss: 0.8515 - val_accuracy: 0.7637
Epoch 34/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.0999 - accuracy: 0.8755 - val_loss: 0.9155 - val_accuracy: 0.7560
Epoch 35/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.0975 - accuracy: 0.8784 - val_loss: 0.8807 - val_accuracy: 0.7550
Epoch 36/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.0958 - accuracy: 0.8814 - val_loss: 0.9323 - val_accuracy: 0.7560
Epoch 37/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.0921 - accuracy: 0.8849 - val_loss: 0.9534 - val_accuracy: 0.7517
Epoch 38/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.0886 - accuracy: 0.8898 - val_loss: 0.9513 - val_accuracy: 0.7690
Epoch 39/50
1575/1575 [==============================] - 29s 18ms/step - loss: 0.0875 - accuracy: 0.8915 - val_loss: 0.9516 - val_accuracy: 0.7613
Epoch 40/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.0836 - accuracy: 0.8950 - val_loss: 0.9981 - val_accuracy: 0.7550
Epoch 41/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.0821 - accuracy: 0.8984 - val_loss: 1.0031 - val_accuracy: 0.7623
Epoch 42/50
1575/1575 [==============================] - 28s 18ms/step - loss: 0.0780 - accuracy: 0.9022 - val_loss: 1.0392 - val_accuracy: 0.7540
Epoch 43/50
1575/1575 [==============================] - 29s 18ms/step - loss: 0.0772 - accuracy: 0.9036 - val_loss: 1.0496 - val_accuracy: 0.7653
Epoch 44/50
1575/1575 [==============================] - 29s 18ms/step - loss: 0.0755 - accuracy: 0.9061 - val_loss: 1.1087 - val_accuracy: 0.7597
Epoch 45/50
1575/1575 [==============================] - 29s 18ms/step - loss: 0.0758 - accuracy: 0.9058 - val_loss: 1.0854 - val_accuracy: 0.7637
Epoch 46/50
1575/1575 [==============================] - 29s 18ms/step - loss: 0.0688 - accuracy: 0.9137 - val_loss: 1.1022 - val_accuracy: 0.7567
Epoch 47/50
1575/1575 [==============================] - 29s 18ms/step - loss: 0.0719 - accuracy: 0.9105 - val_loss: 1.1392 - val_accuracy: 0.7593
Epoch 48/50
1575/1575 [==============================] - 29s 18ms/step - loss: 0.0676 - accuracy: 0.9156 - val_loss: 1.1376 - val_accuracy: 0.7540
Epoch 49/50
1575/1575 [==============================] - 29s 18ms/step - loss: 0.0685 - accuracy: 0.9148 - val_loss: 1.0964 - val_accuracy: 0.7740
Epoch 50/50
1575/1575 [==============================] - 29s 18ms/step - loss: 0.0653 - accuracy: 0.9193 - val_loss: 1.1931 - val_accuracy: 0.7617

In [31]:

plot_network_summaries(history, 'best_model_B1-2.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_B1-2.hdf5', val_images, val_labels)

Optimal point:  17 epochs
94/94 [==============================] - 0s 2ms/step

94/94 [==============================] - 0s 3ms/step - loss: 0.7250 - accuracy: 0.7303
Test accuracy: 0.7303333282470703

The model overfits fairly quickly again, but provides a greater accuracy than the earlier models.

B3 - Decrease Layers¶

One of the dense layers is removed in order to evaluate the effect of simplification of the network.

In [22]:

from tensorflow.keras import models, layers
from tensorflow.keras import regularizers

checkpoint = ModelCheckpoint(
    'best_model_B1-3.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

network = models.Sequential()
network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(1024, activation='relu'))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=50, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers from tensorflow.keras import regularizers checkpoint = ModelCheckpoint( 'best_model_B1-3.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) network = models.Sequential() network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(1024, activation='relu')) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=50, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/50
1575/1575 [==============================] - 18s 11ms/step - loss: 0.4071 - accuracy: 0.4427 - val_loss: 1.1304 - val_accuracy: 0.5063
Epoch 2/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.3539 - accuracy: 0.5386 - val_loss: 1.0639 - val_accuracy: 0.5520
Epoch 3/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.3306 - accuracy: 0.5791 - val_loss: 0.9754 - val_accuracy: 0.5973
Epoch 4/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.3135 - accuracy: 0.6064 - val_loss: 0.8839 - val_accuracy: 0.6387
Epoch 5/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.3022 - accuracy: 0.6255 - val_loss: 0.8857 - val_accuracy: 0.6363
Epoch 6/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.2921 - accuracy: 0.6402 - val_loss: 0.8513 - val_accuracy: 0.6487
Epoch 7/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.2843 - accuracy: 0.6516 - val_loss: 0.8336 - val_accuracy: 0.6597
Epoch 8/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.2774 - accuracy: 0.6619 - val_loss: 0.9047 - val_accuracy: 0.6367
Epoch 9/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.2708 - accuracy: 0.6715 - val_loss: 0.8121 - val_accuracy: 0.6637
Epoch 10/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.2653 - accuracy: 0.6789 - val_loss: 0.7943 - val_accuracy: 0.6720
Epoch 11/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.2604 - accuracy: 0.6844 - val_loss: 0.8013 - val_accuracy: 0.6843
Epoch 12/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.2546 - accuracy: 0.6933 - val_loss: 0.7676 - val_accuracy: 0.6893
Epoch 13/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.2493 - accuracy: 0.7010 - val_loss: 0.7932 - val_accuracy: 0.6853
Epoch 14/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.2449 - accuracy: 0.7064 - val_loss: 0.7962 - val_accuracy: 0.6870
Epoch 15/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.2406 - accuracy: 0.7112 - val_loss: 0.7903 - val_accuracy: 0.6870
Epoch 16/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.2370 - accuracy: 0.7162 - val_loss: 0.7817 - val_accuracy: 0.6913
Epoch 17/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.2325 - accuracy: 0.7234 - val_loss: 0.7354 - val_accuracy: 0.6997
Epoch 18/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.2285 - accuracy: 0.7281 - val_loss: 0.7478 - val_accuracy: 0.7020
Epoch 19/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.2245 - accuracy: 0.7329 - val_loss: 0.8097 - val_accuracy: 0.6930
Epoch 20/50
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2210 - accuracy: 0.7362 - val_loss: 0.7327 - val_accuracy: 0.7073
Epoch 21/50
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2172 - accuracy: 0.7405 - val_loss: 0.7470 - val_accuracy: 0.7053
Epoch 22/50
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2132 - accuracy: 0.7464 - val_loss: 0.7857 - val_accuracy: 0.7013
Epoch 23/50
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2116 - accuracy: 0.7476 - val_loss: 0.7029 - val_accuracy: 0.7243
Epoch 24/50
1575/1575 [==============================] - 18s 11ms/step - loss: 0.2074 - accuracy: 0.7532 - val_loss: 0.7362 - val_accuracy: 0.7083
Epoch 25/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.2039 - accuracy: 0.7567 - val_loss: 0.7248 - val_accuracy: 0.7167
Epoch 26/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.2005 - accuracy: 0.7614 - val_loss: 0.7495 - val_accuracy: 0.7133
Epoch 27/50
1575/1575 [==============================] - 18s 11ms/step - loss: 0.1978 - accuracy: 0.7646 - val_loss: 0.7292 - val_accuracy: 0.7193
Epoch 28/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1949 - accuracy: 0.7666 - val_loss: 0.7300 - val_accuracy: 0.7237
Epoch 29/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1921 - accuracy: 0.7715 - val_loss: 0.7288 - val_accuracy: 0.7307
Epoch 30/50
1575/1575 [==============================] - 18s 11ms/step - loss: 0.1901 - accuracy: 0.7726 - val_loss: 0.7002 - val_accuracy: 0.7387
Epoch 31/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1869 - accuracy: 0.7771 - val_loss: 0.7313 - val_accuracy: 0.7183
Epoch 32/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1840 - accuracy: 0.7801 - val_loss: 0.7192 - val_accuracy: 0.7190
Epoch 33/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1806 - accuracy: 0.7844 - val_loss: 0.7133 - val_accuracy: 0.7277
Epoch 34/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.1801 - accuracy: 0.7851 - val_loss: 0.7270 - val_accuracy: 0.7283
Epoch 35/50
1575/1575 [==============================] - 16s 10ms/step - loss: 0.1767 - accuracy: 0.7896 - val_loss: 0.7616 - val_accuracy: 0.7130
Epoch 36/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1751 - accuracy: 0.7905 - val_loss: 0.7367 - val_accuracy: 0.7280
Epoch 37/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1725 - accuracy: 0.7934 - val_loss: 0.7451 - val_accuracy: 0.7300
Epoch 38/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1703 - accuracy: 0.7966 - val_loss: 0.7380 - val_accuracy: 0.7320
Epoch 39/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1676 - accuracy: 0.7995 - val_loss: 0.7548 - val_accuracy: 0.7290
Epoch 40/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1661 - accuracy: 0.8006 - val_loss: 0.7161 - val_accuracy: 0.7443
Epoch 41/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1641 - accuracy: 0.8039 - val_loss: 0.7686 - val_accuracy: 0.7280
Epoch 42/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1621 - accuracy: 0.8058 - val_loss: 0.7457 - val_accuracy: 0.7283
Epoch 43/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1600 - accuracy: 0.8073 - val_loss: 0.8058 - val_accuracy: 0.7260
Epoch 44/50
1575/1575 [==============================] - 18s 12ms/step - loss: 0.1579 - accuracy: 0.8102 - val_loss: 0.7093 - val_accuracy: 0.7470
Epoch 45/50
1575/1575 [==============================] - 19s 12ms/step - loss: 0.1560 - accuracy: 0.8128 - val_loss: 0.7578 - val_accuracy: 0.7337
Epoch 46/50
1575/1575 [==============================] - 18s 12ms/step - loss: 0.1542 - accuracy: 0.8141 - val_loss: 0.7587 - val_accuracy: 0.7340
Epoch 47/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1521 - accuracy: 0.8162 - val_loss: 0.7192 - val_accuracy: 0.7423
Epoch 48/50
1575/1575 [==============================] - 17s 11ms/step - loss: 0.1504 - accuracy: 0.8180 - val_loss: 0.7411 - val_accuracy: 0.7353
Epoch 49/50
1575/1575 [==============================] - 18s 11ms/step - loss: 0.1493 - accuracy: 0.8198 - val_loss: 0.7390 - val_accuracy: 0.7307
Epoch 50/50
1575/1575 [==============================] - 18s 11ms/step - loss: 0.1473 - accuracy: 0.8216 - val_loss: 0.7299 - val_accuracy: 0.7437

In [25]:

plot_network_summaries(history, 'best_model_B1-3.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_B1-3.hdf5', val_images, val_labels)

Optimal point:  30 epochs
94/94 [==============================] - 0s 2ms/step

94/94 [==============================] - 0s 2ms/step - loss: 0.7002 - accuracy: 0.7387
Test accuracy: 0.7386666536331177

The simplification resulted in some further accuracy gain, but also addresses some of the drastic overfitting of the earlier model.

B4 - Decreased Neurons¶

The Neurons is decreased from 1028 in each layer to 512

In [26]:

from tensorflow.keras import models, layers
from tensorflow.keras import regularizers

checkpoint = ModelCheckpoint(
    'best_model_B1-4.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

network = models.Sequential()
network.add(layers.Dense(512, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(512, activation='relu'))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=50, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers from tensorflow.keras import regularizers checkpoint = ModelCheckpoint( 'best_model_B1-4.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) network = models.Sequential() network.add(layers.Dense(512, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(512, activation='relu')) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=50, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.4072 - accuracy: 0.4440 - val_loss: 1.1974 - val_accuracy: 0.4847
Epoch 2/50
  12/1575 [..............................] - ETA: 7s - loss: 0.3676 - accuracy: 0.5286

c:\Projects\UoL\(DSM150-2023-OCT) Neural networks\CW1\.venv\Lib\site-packages\keras\src\engine\training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

1575/1575 [==============================] - 7s 5ms/step - loss: 0.3534 - accuracy: 0.5436 - val_loss: 1.0586 - val_accuracy: 0.5437
Epoch 3/50
1575/1575 [==============================] - 7s 5ms/step - loss: 0.3293 - accuracy: 0.5822 - val_loss: 0.9676 - val_accuracy: 0.5963
Epoch 4/50
1575/1575 [==============================] - 7s 5ms/step - loss: 0.3132 - accuracy: 0.6078 - val_loss: 0.9127 - val_accuracy: 0.6180
Epoch 5/50
1575/1575 [==============================] - 7s 5ms/step - loss: 0.3017 - accuracy: 0.6233 - val_loss: 0.9337 - val_accuracy: 0.6217
Epoch 6/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.2919 - accuracy: 0.6396 - val_loss: 0.8527 - val_accuracy: 0.6567
Epoch 7/50
1575/1575 [==============================] - 7s 5ms/step - loss: 0.2846 - accuracy: 0.6499 - val_loss: 0.8622 - val_accuracy: 0.6450
Epoch 8/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.2773 - accuracy: 0.6627 - val_loss: 0.8470 - val_accuracy: 0.6463
Epoch 9/50
1575/1575 [==============================] - 7s 5ms/step - loss: 0.2712 - accuracy: 0.6704 - val_loss: 0.8431 - val_accuracy: 0.6590
Epoch 10/50
1575/1575 [==============================] - 7s 5ms/step - loss: 0.2659 - accuracy: 0.6773 - val_loss: 0.8068 - val_accuracy: 0.6797
Epoch 11/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.2610 - accuracy: 0.6843 - val_loss: 0.8135 - val_accuracy: 0.6733
Epoch 12/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.2554 - accuracy: 0.6926 - val_loss: 0.7684 - val_accuracy: 0.6913
Epoch 13/50
1575/1575 [==============================] - 7s 5ms/step - loss: 0.2517 - accuracy: 0.6964 - val_loss: 0.8006 - val_accuracy: 0.6870
Epoch 14/50
1575/1575 [==============================] - 7s 5ms/step - loss: 0.2470 - accuracy: 0.7040 - val_loss: 0.8077 - val_accuracy: 0.6797
Epoch 15/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.2438 - accuracy: 0.7077 - val_loss: 0.7799 - val_accuracy: 0.6820
Epoch 16/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.2393 - accuracy: 0.7126 - val_loss: 0.7808 - val_accuracy: 0.6920
Epoch 17/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.2353 - accuracy: 0.7177 - val_loss: 0.7914 - val_accuracy: 0.6847
Epoch 18/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.2314 - accuracy: 0.7228 - val_loss: 0.7570 - val_accuracy: 0.7013
Epoch 19/50
1575/1575 [==============================] - 7s 5ms/step - loss: 0.2287 - accuracy: 0.7265 - val_loss: 0.7841 - val_accuracy: 0.6947
Epoch 20/50
1575/1575 [==============================] - 7s 5ms/step - loss: 0.2259 - accuracy: 0.7291 - val_loss: 0.7235 - val_accuracy: 0.7027
Epoch 21/50
1575/1575 [==============================] - 7s 5ms/step - loss: 0.2223 - accuracy: 0.7350 - val_loss: 0.7535 - val_accuracy: 0.7043
Epoch 22/50
1575/1575 [==============================] - 7s 5ms/step - loss: 0.2187 - accuracy: 0.7392 - val_loss: 0.7712 - val_accuracy: 0.7017
Epoch 23/50
1575/1575 [==============================] - 7s 5ms/step - loss: 0.2158 - accuracy: 0.7423 - val_loss: 0.7503 - val_accuracy: 0.7067
Epoch 24/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.2130 - accuracy: 0.7462 - val_loss: 0.7496 - val_accuracy: 0.7033
Epoch 25/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.2109 - accuracy: 0.7472 - val_loss: 0.7224 - val_accuracy: 0.7273
Epoch 26/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.2075 - accuracy: 0.7525 - val_loss: 0.7284 - val_accuracy: 0.7223
Epoch 27/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.2042 - accuracy: 0.7550 - val_loss: 0.7237 - val_accuracy: 0.7140
Epoch 28/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.2021 - accuracy: 0.7590 - val_loss: 0.7791 - val_accuracy: 0.7130
Epoch 29/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.2000 - accuracy: 0.7611 - val_loss: 0.7593 - val_accuracy: 0.7143
Epoch 30/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.1975 - accuracy: 0.7645 - val_loss: 0.7329 - val_accuracy: 0.7180
Epoch 31/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.1951 - accuracy: 0.7662 - val_loss: 0.7539 - val_accuracy: 0.7190
Epoch 32/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1935 - accuracy: 0.7697 - val_loss: 0.7720 - val_accuracy: 0.7100
Epoch 33/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1911 - accuracy: 0.7719 - val_loss: 0.7468 - val_accuracy: 0.7220
Epoch 34/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1886 - accuracy: 0.7756 - val_loss: 0.7409 - val_accuracy: 0.7330
Epoch 35/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1874 - accuracy: 0.7760 - val_loss: 0.7533 - val_accuracy: 0.7187
Epoch 36/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.1847 - accuracy: 0.7790 - val_loss: 0.7359 - val_accuracy: 0.7230
Epoch 37/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1826 - accuracy: 0.7817 - val_loss: 0.7257 - val_accuracy: 0.7233
Epoch 38/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1810 - accuracy: 0.7833 - val_loss: 0.7633 - val_accuracy: 0.7223
Epoch 39/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.1794 - accuracy: 0.7855 - val_loss: 0.7535 - val_accuracy: 0.7277
Epoch 40/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1770 - accuracy: 0.7880 - val_loss: 0.7497 - val_accuracy: 0.7307
Epoch 41/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1753 - accuracy: 0.7896 - val_loss: 0.7424 - val_accuracy: 0.7270
Epoch 42/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.1738 - accuracy: 0.7912 - val_loss: 0.7731 - val_accuracy: 0.7260
Epoch 43/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1716 - accuracy: 0.7946 - val_loss: 0.7695 - val_accuracy: 0.7250
Epoch 44/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1701 - accuracy: 0.7960 - val_loss: 0.7335 - val_accuracy: 0.7340
Epoch 45/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1686 - accuracy: 0.7982 - val_loss: 0.7721 - val_accuracy: 0.7307
Epoch 46/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1671 - accuracy: 0.7993 - val_loss: 0.7569 - val_accuracy: 0.7277
Epoch 47/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1659 - accuracy: 0.8002 - val_loss: 0.7926 - val_accuracy: 0.7277
Epoch 48/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1637 - accuracy: 0.8031 - val_loss: 0.7605 - val_accuracy: 0.7340
Epoch 49/50
1575/1575 [==============================] - 8s 5ms/step - loss: 0.1627 - accuracy: 0.8046 - val_loss: 0.7793 - val_accuracy: 0.7297
Epoch 50/50
1575/1575 [==============================] - 9s 5ms/step - loss: 0.1620 - accuracy: 0.8052 - val_loss: 0.7617 - val_accuracy: 0.7227

In [28]:

plot_network_summaries(history, 'best_model_B1-4.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_B1-4.hdf5', val_images, val_labels)

Optimal point:  25 epochs
94/94 [==============================] - 0s 2ms/step

94/94 [==============================] - 0s 2ms/step - loss: 0.7224 - accuracy: 0.7273
Test accuracy: 0.7273333072662354

The less complex network does have a decreased accuracy, but warrants some further experimentation.

B5 - Structure experimentation¶

Instead of a square structure, two layers, one of 1024, and one of 256 neurons is used.

In [29]:

from tensorflow.keras import models, layers
from tensorflow.keras import regularizers

checkpoint = ModelCheckpoint(
    'best_model_B1-5.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

network = models.Sequential()
network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(256, activation='relu'))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=50, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers from tensorflow.keras import regularizers checkpoint = ModelCheckpoint( 'best_model_B1-5.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) network = models.Sequential() network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(256, activation='relu')) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=50, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.4109 - accuracy: 0.4372 - val_loss: 1.2270 - val_accuracy: 0.4780
Epoch 2/50
  28/1575 [..............................] - ETA: 8s - loss: 0.3769 - accuracy: 0.5114

c:\Projects\UoL\(DSM150-2023-OCT) Neural networks\CW1\.venv\Lib\site-packages\keras\src\engine\training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

1575/1575 [==============================] - 10s 6ms/step - loss: 0.3582 - accuracy: 0.5344 - val_loss: 1.0384 - val_accuracy: 0.5717
Epoch 3/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.3333 - accuracy: 0.5760 - val_loss: 0.9807 - val_accuracy: 0.5870
Epoch 4/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.3163 - accuracy: 0.6039 - val_loss: 0.9492 - val_accuracy: 0.5960
Epoch 5/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.3027 - accuracy: 0.6234 - val_loss: 0.9137 - val_accuracy: 0.6213
Epoch 6/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.2917 - accuracy: 0.6414 - val_loss: 0.8235 - val_accuracy: 0.6640
Epoch 7/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2838 - accuracy: 0.6522 - val_loss: 0.8563 - val_accuracy: 0.6557
Epoch 8/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2756 - accuracy: 0.6647 - val_loss: 0.7996 - val_accuracy: 0.6690
Epoch 9/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2696 - accuracy: 0.6732 - val_loss: 0.8272 - val_accuracy: 0.6530
Epoch 10/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2639 - accuracy: 0.6822 - val_loss: 0.8077 - val_accuracy: 0.6577
Epoch 11/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2584 - accuracy: 0.6882 - val_loss: 0.7858 - val_accuracy: 0.6810
Epoch 12/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2536 - accuracy: 0.6944 - val_loss: 0.7820 - val_accuracy: 0.6877
Epoch 13/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2490 - accuracy: 0.7023 - val_loss: 0.7838 - val_accuracy: 0.6853
Epoch 14/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2454 - accuracy: 0.7066 - val_loss: 0.7771 - val_accuracy: 0.6857
Epoch 15/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2408 - accuracy: 0.7124 - val_loss: 0.7749 - val_accuracy: 0.6927
Epoch 16/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.2371 - accuracy: 0.7160 - val_loss: 0.7558 - val_accuracy: 0.6947
Epoch 17/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2330 - accuracy: 0.7216 - val_loss: 0.7591 - val_accuracy: 0.7027
Epoch 18/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2305 - accuracy: 0.7249 - val_loss: 0.7104 - val_accuracy: 0.7070
Epoch 19/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.2272 - accuracy: 0.7289 - val_loss: 0.7789 - val_accuracy: 0.6887
Epoch 20/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.2231 - accuracy: 0.7334 - val_loss: 0.7260 - val_accuracy: 0.7110
Epoch 21/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2205 - accuracy: 0.7367 - val_loss: 0.7293 - val_accuracy: 0.7107
Epoch 22/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2179 - accuracy: 0.7400 - val_loss: 0.7276 - val_accuracy: 0.7137
Epoch 23/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2139 - accuracy: 0.7453 - val_loss: 0.7486 - val_accuracy: 0.7067
Epoch 24/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2118 - accuracy: 0.7475 - val_loss: 0.7845 - val_accuracy: 0.7103
Epoch 25/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2093 - accuracy: 0.7507 - val_loss: 0.7429 - val_accuracy: 0.7113
Epoch 26/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2066 - accuracy: 0.7533 - val_loss: 0.7079 - val_accuracy: 0.7243
Epoch 27/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2042 - accuracy: 0.7567 - val_loss: 0.7194 - val_accuracy: 0.7147
Epoch 28/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2013 - accuracy: 0.7599 - val_loss: 0.7032 - val_accuracy: 0.7300
Epoch 29/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.1990 - accuracy: 0.7630 - val_loss: 0.7312 - val_accuracy: 0.7150
Epoch 30/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.1972 - accuracy: 0.7647 - val_loss: 0.7198 - val_accuracy: 0.7187
Epoch 31/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.1951 - accuracy: 0.7670 - val_loss: 0.7220 - val_accuracy: 0.7220
Epoch 32/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.1922 - accuracy: 0.7698 - val_loss: 0.7633 - val_accuracy: 0.7060
Epoch 33/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1898 - accuracy: 0.7734 - val_loss: 0.7173 - val_accuracy: 0.7200
Epoch 34/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1873 - accuracy: 0.7761 - val_loss: 0.6924 - val_accuracy: 0.7373
Epoch 35/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1853 - accuracy: 0.7795 - val_loss: 0.6976 - val_accuracy: 0.7333
Epoch 36/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1842 - accuracy: 0.7794 - val_loss: 0.6954 - val_accuracy: 0.7353
Epoch 37/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.1820 - accuracy: 0.7821 - val_loss: 0.6996 - val_accuracy: 0.7437
Epoch 38/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1799 - accuracy: 0.7847 - val_loss: 0.7005 - val_accuracy: 0.7403
Epoch 39/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1778 - accuracy: 0.7873 - val_loss: 0.7001 - val_accuracy: 0.7330
Epoch 40/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1757 - accuracy: 0.7890 - val_loss: 0.7403 - val_accuracy: 0.7230
Epoch 41/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1747 - accuracy: 0.7903 - val_loss: 0.7018 - val_accuracy: 0.7383
Epoch 42/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1728 - accuracy: 0.7936 - val_loss: 0.6960 - val_accuracy: 0.7447
Epoch 43/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1704 - accuracy: 0.7958 - val_loss: 0.7125 - val_accuracy: 0.7410
Epoch 44/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1699 - accuracy: 0.7959 - val_loss: 0.7504 - val_accuracy: 0.7250
Epoch 45/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1677 - accuracy: 0.7987 - val_loss: 0.7344 - val_accuracy: 0.7360
Epoch 46/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1654 - accuracy: 0.8016 - val_loss: 0.7183 - val_accuracy: 0.7450
Epoch 47/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1651 - accuracy: 0.8020 - val_loss: 0.7245 - val_accuracy: 0.7457
Epoch 48/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1623 - accuracy: 0.8047 - val_loss: 0.7392 - val_accuracy: 0.7403
Epoch 49/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1613 - accuracy: 0.8052 - val_loss: 0.7425 - val_accuracy: 0.7350
Epoch 50/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1605 - accuracy: 0.8079 - val_loss: 0.7425 - val_accuracy: 0.7327

In [30]:

plot_network_summaries(history, 'best_model_B1-5.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_B1-5.hdf5', val_images, val_labels)

Optimal point:  34 epochs
94/94 [==============================] - 0s 2ms/step

94/94 [==============================] - 0s 2ms/step - loss: 0.6924 - accuracy: 0.7373
Test accuracy: 0.737333357334137

This network once again shows improvement, indicating that the first 1024 neuron dense layer may be important in the composition.

B6 - Further experimentation¶

The neurons of the second dense layer is decreased further.

In [31]:

from tensorflow.keras import models, layers
from tensorflow.keras import regularizers

checkpoint = ModelCheckpoint(
    'best_model_B1-6.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

network = models.Sequential()
network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(128, activation='relu'))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=50, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers from tensorflow.keras import regularizers checkpoint = ModelCheckpoint( 'best_model_B1-6.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) network = models.Sequential() network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(128, activation='relu')) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=50, batch_size=128, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.4124 - accuracy: 0.4338 - val_loss: 1.1469 - val_accuracy: 0.5033
Epoch 2/50
   1/1575 [..............................] - ETA: 9s - loss: 0.3376 - accuracy: 0.6094

c:\Projects\UoL\(DSM150-2023-OCT) Neural networks\CW1\.venv\Lib\site-packages\keras\src\engine\training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

1575/1575 [==============================] - 10s 6ms/step - loss: 0.3593 - accuracy: 0.5346 - val_loss: 1.0447 - val_accuracy: 0.5587
Epoch 3/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.3343 - accuracy: 0.5786 - val_loss: 1.0391 - val_accuracy: 0.5573
Epoch 4/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.3176 - accuracy: 0.6048 - val_loss: 1.0037 - val_accuracy: 0.5840
Epoch 5/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.3054 - accuracy: 0.6231 - val_loss: 0.8744 - val_accuracy: 0.6350
Epoch 6/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.2964 - accuracy: 0.6382 - val_loss: 0.8871 - val_accuracy: 0.6433
Epoch 7/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2884 - accuracy: 0.6473 - val_loss: 0.8537 - val_accuracy: 0.6490
Epoch 8/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2812 - accuracy: 0.6590 - val_loss: 0.8397 - val_accuracy: 0.6613
Epoch 9/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2748 - accuracy: 0.6670 - val_loss: 0.8221 - val_accuracy: 0.6660
Epoch 10/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.2693 - accuracy: 0.6742 - val_loss: 0.8233 - val_accuracy: 0.6677
Epoch 11/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.2643 - accuracy: 0.6818 - val_loss: 0.7917 - val_accuracy: 0.6853
Epoch 12/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.2601 - accuracy: 0.6877 - val_loss: 0.7798 - val_accuracy: 0.6820
Epoch 13/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2557 - accuracy: 0.6924 - val_loss: 0.8025 - val_accuracy: 0.6727
Epoch 14/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2520 - accuracy: 0.6977 - val_loss: 0.7934 - val_accuracy: 0.6793
Epoch 15/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2482 - accuracy: 0.7020 - val_loss: 0.7862 - val_accuracy: 0.6767
Epoch 16/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2443 - accuracy: 0.7074 - val_loss: 0.7830 - val_accuracy: 0.6853
Epoch 17/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2408 - accuracy: 0.7126 - val_loss: 0.7641 - val_accuracy: 0.6843
Epoch 18/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.2376 - accuracy: 0.7148 - val_loss: 0.7567 - val_accuracy: 0.6893
Epoch 19/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2348 - accuracy: 0.7196 - val_loss: 0.7879 - val_accuracy: 0.6877
Epoch 20/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2317 - accuracy: 0.7232 - val_loss: 0.7433 - val_accuracy: 0.6963
Epoch 21/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2292 - accuracy: 0.7281 - val_loss: 0.7759 - val_accuracy: 0.6917
Epoch 22/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.2262 - accuracy: 0.7310 - val_loss: 0.7660 - val_accuracy: 0.6920
Epoch 23/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.2236 - accuracy: 0.7343 - val_loss: 0.7473 - val_accuracy: 0.6930
Epoch 24/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.2212 - accuracy: 0.7368 - val_loss: 0.7373 - val_accuracy: 0.7060
Epoch 25/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2186 - accuracy: 0.7401 - val_loss: 0.7356 - val_accuracy: 0.6967
Epoch 26/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2166 - accuracy: 0.7424 - val_loss: 0.7476 - val_accuracy: 0.7117
Epoch 27/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2133 - accuracy: 0.7463 - val_loss: 0.7431 - val_accuracy: 0.7017
Epoch 28/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2113 - accuracy: 0.7486 - val_loss: 0.7343 - val_accuracy: 0.7073
Epoch 29/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2090 - accuracy: 0.7509 - val_loss: 0.7342 - val_accuracy: 0.7187
Epoch 30/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2070 - accuracy: 0.7536 - val_loss: 0.7798 - val_accuracy: 0.6810
Epoch 31/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2051 - accuracy: 0.7556 - val_loss: 0.7113 - val_accuracy: 0.7150
Epoch 32/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.2029 - accuracy: 0.7582 - val_loss: 0.7107 - val_accuracy: 0.7173
Epoch 33/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.2013 - accuracy: 0.7601 - val_loss: 0.7229 - val_accuracy: 0.7130
Epoch 34/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1992 - accuracy: 0.7627 - val_loss: 0.7130 - val_accuracy: 0.7177
Epoch 35/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1968 - accuracy: 0.7658 - val_loss: 0.7359 - val_accuracy: 0.7127
Epoch 36/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1949 - accuracy: 0.7673 - val_loss: 0.7262 - val_accuracy: 0.7167
Epoch 37/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1936 - accuracy: 0.7693 - val_loss: 0.7439 - val_accuracy: 0.7087
Epoch 38/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1924 - accuracy: 0.7710 - val_loss: 0.7345 - val_accuracy: 0.7073
Epoch 39/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1904 - accuracy: 0.7733 - val_loss: 0.7217 - val_accuracy: 0.7187
Epoch 40/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1885 - accuracy: 0.7759 - val_loss: 0.7576 - val_accuracy: 0.7060
Epoch 41/50
1575/1575 [==============================] - 10s 7ms/step - loss: 0.1865 - accuracy: 0.7776 - val_loss: 0.7100 - val_accuracy: 0.7187
Epoch 42/50
1575/1575 [==============================] - 10s 7ms/step - loss: 0.1862 - accuracy: 0.7778 - val_loss: 0.7560 - val_accuracy: 0.7107
Epoch 43/50
1575/1575 [==============================] - 10s 7ms/step - loss: 0.1839 - accuracy: 0.7805 - val_loss: 0.7320 - val_accuracy: 0.7103
Epoch 44/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1826 - accuracy: 0.7816 - val_loss: 0.7189 - val_accuracy: 0.7223
Epoch 45/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1811 - accuracy: 0.7850 - val_loss: 0.6973 - val_accuracy: 0.7300
Epoch 46/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.1792 - accuracy: 0.7856 - val_loss: 0.7198 - val_accuracy: 0.7193
Epoch 47/50
1575/1575 [==============================] - 9s 6ms/step - loss: 0.1790 - accuracy: 0.7867 - val_loss: 0.7155 - val_accuracy: 0.7220
Epoch 48/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1764 - accuracy: 0.7889 - val_loss: 0.7080 - val_accuracy: 0.7263
Epoch 49/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1757 - accuracy: 0.7889 - val_loss: 0.7025 - val_accuracy: 0.7293
Epoch 50/50
1575/1575 [==============================] - 10s 6ms/step - loss: 0.1737 - accuracy: 0.7921 - val_loss: 0.7486 - val_accuracy: 0.7237

In [32]:

plot_network_summaries(history, 'best_model_B1-6.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_B1-6.hdf5', val_images, val_labels)

Optimal point:  45 epochs
94/94 [==============================] - 0s 1ms/step

94/94 [==============================] - 0s 2ms/step - loss: 0.6973 - accuracy: 0.7300
Test accuracy: 0.7300000190734863

The model has a bit less accuracy that the earlier model, but trains faster, and has a good distribution of confusion between the drusen and normal labels

B7 - Regularizer¶

A regularizer of 0.001 is added, and the epochs is increased in order to evaluate the effect.

In [37]:

from tensorflow.keras import models, layers
from tensorflow.keras import regularizers

checkpoint = ModelCheckpoint(
    'best_model_B1-7.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

network = models.Sequential()
network.add(layers.Dense(1024, activation='relu',kernel_regularizer=regularizers.l2(0.001), input_shape = (28*28, )))
network.add(layers.Dense(128, activation='relu'))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=75, batch_size=512, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers from tensorflow.keras import regularizers checkpoint = ModelCheckpoint( 'best_model_B1-7.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) network = models.Sequential() network.add(layers.Dense(1024, activation='relu',kernel_regularizer=regularizers.l2(0.001), input_shape = (28*28, ))) network.add(layers.Dense(128, activation='relu')) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=75, batch_size=512, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/75
394/394 [==============================] - 4s 8ms/step - loss: 0.5400 - accuracy: 0.3592 - val_loss: 1.4048 - val_accuracy: 0.3910
Epoch 2/75
394/394 [==============================] - 3s 8ms/step - loss: 0.4246 - accuracy: 0.4393 - val_loss: 1.2981 - val_accuracy: 0.4263
Epoch 3/75
394/394 [==============================] - 3s 8ms/step - loss: 0.4066 - accuracy: 0.4749 - val_loss: 1.2123 - val_accuracy: 0.4733
Epoch 4/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3954 - accuracy: 0.4974 - val_loss: 1.2086 - val_accuracy: 0.4713
Epoch 5/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3885 - accuracy: 0.5117 - val_loss: 1.1554 - val_accuracy: 0.5073
Epoch 6/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3818 - accuracy: 0.5266 - val_loss: 1.1432 - val_accuracy: 0.5083
Epoch 7/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3754 - accuracy: 0.5375 - val_loss: 1.1098 - val_accuracy: 0.5217
Epoch 8/75
394/394 [==============================] - 3s 7ms/step - loss: 0.3695 - accuracy: 0.5495 - val_loss: 1.1200 - val_accuracy: 0.5137
Epoch 9/75
394/394 [==============================] - 3s 7ms/step - loss: 0.3646 - accuracy: 0.5599 - val_loss: 1.0825 - val_accuracy: 0.5557
Epoch 10/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3599 - accuracy: 0.5684 - val_loss: 1.0487 - val_accuracy: 0.5407
Epoch 11/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3577 - accuracy: 0.5745 - val_loss: 1.0229 - val_accuracy: 0.5750
Epoch 12/75
394/394 [==============================] - 3s 7ms/step - loss: 0.3537 - accuracy: 0.5801 - val_loss: 1.0048 - val_accuracy: 0.5733
Epoch 13/75
394/394 [==============================] - 3s 7ms/step - loss: 0.3515 - accuracy: 0.5839 - val_loss: 0.9520 - val_accuracy: 0.6110
Epoch 14/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3503 - accuracy: 0.5874 - val_loss: 0.9405 - val_accuracy: 0.6087
Epoch 15/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3454 - accuracy: 0.5964 - val_loss: 1.0090 - val_accuracy: 0.5890
Epoch 16/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3443 - accuracy: 0.5968 - val_loss: 0.9517 - val_accuracy: 0.6133
Epoch 17/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3448 - accuracy: 0.5963 - val_loss: 0.9730 - val_accuracy: 0.6040
Epoch 18/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3420 - accuracy: 0.6005 - val_loss: 0.9668 - val_accuracy: 0.6073
Epoch 19/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3388 - accuracy: 0.6055 - val_loss: 0.9505 - val_accuracy: 0.6007
Epoch 20/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3381 - accuracy: 0.6074 - val_loss: 0.9083 - val_accuracy: 0.6250
Epoch 21/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3373 - accuracy: 0.6094 - val_loss: 0.9324 - val_accuracy: 0.6127
Epoch 22/75
394/394 [==============================] - 3s 7ms/step - loss: 0.3368 - accuracy: 0.6091 - val_loss: 0.9435 - val_accuracy: 0.6127
Epoch 23/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3348 - accuracy: 0.6118 - val_loss: 1.0298 - val_accuracy: 0.5893
Epoch 24/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3349 - accuracy: 0.6126 - val_loss: 0.9945 - val_accuracy: 0.5940
Epoch 25/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3349 - accuracy: 0.6118 - val_loss: 1.0013 - val_accuracy: 0.5823
Epoch 26/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3329 - accuracy: 0.6155 - val_loss: 0.9534 - val_accuracy: 0.5907
Epoch 27/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3302 - accuracy: 0.6195 - val_loss: 0.9329 - val_accuracy: 0.6157
Epoch 28/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3322 - accuracy: 0.6147 - val_loss: 0.9372 - val_accuracy: 0.6283
Epoch 29/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3309 - accuracy: 0.6176 - val_loss: 0.8894 - val_accuracy: 0.6433
Epoch 30/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3301 - accuracy: 0.6180 - val_loss: 0.9663 - val_accuracy: 0.5980
Epoch 31/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3292 - accuracy: 0.6195 - val_loss: 0.9008 - val_accuracy: 0.6290
Epoch 32/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3282 - accuracy: 0.6225 - val_loss: 0.9095 - val_accuracy: 0.6307
Epoch 33/75
394/394 [==============================] - 3s 9ms/step - loss: 0.3263 - accuracy: 0.6249 - val_loss: 0.8894 - val_accuracy: 0.6297
Epoch 34/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3290 - accuracy: 0.6212 - val_loss: 0.9663 - val_accuracy: 0.5963
Epoch 35/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3262 - accuracy: 0.6255 - val_loss: 0.8751 - val_accuracy: 0.6410
Epoch 36/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3269 - accuracy: 0.6252 - val_loss: 0.8846 - val_accuracy: 0.6430
Epoch 37/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3249 - accuracy: 0.6270 - val_loss: 0.9545 - val_accuracy: 0.6003
Epoch 38/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3255 - accuracy: 0.6269 - val_loss: 0.8753 - val_accuracy: 0.6423
Epoch 39/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3252 - accuracy: 0.6271 - val_loss: 0.9853 - val_accuracy: 0.6110
Epoch 40/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3251 - accuracy: 0.6282 - val_loss: 0.9074 - val_accuracy: 0.6247
Epoch 41/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3254 - accuracy: 0.6280 - val_loss: 0.9243 - val_accuracy: 0.6207
Epoch 42/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3224 - accuracy: 0.6313 - val_loss: 0.9452 - val_accuracy: 0.6053
Epoch 43/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3230 - accuracy: 0.6307 - val_loss: 0.8901 - val_accuracy: 0.6390
Epoch 44/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3220 - accuracy: 0.6333 - val_loss: 0.9627 - val_accuracy: 0.6117
Epoch 45/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3228 - accuracy: 0.6315 - val_loss: 0.9432 - val_accuracy: 0.6227
Epoch 46/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3209 - accuracy: 0.6338 - val_loss: 0.9614 - val_accuracy: 0.6037
Epoch 47/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3232 - accuracy: 0.6297 - val_loss: 0.8962 - val_accuracy: 0.6277
Epoch 48/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3229 - accuracy: 0.6308 - val_loss: 0.9051 - val_accuracy: 0.6357
Epoch 49/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3217 - accuracy: 0.6330 - val_loss: 0.9185 - val_accuracy: 0.6290
Epoch 50/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3203 - accuracy: 0.6359 - val_loss: 0.9491 - val_accuracy: 0.6247
Epoch 51/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3211 - accuracy: 0.6334 - val_loss: 0.9211 - val_accuracy: 0.6273
Epoch 52/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3203 - accuracy: 0.6351 - val_loss: 0.8919 - val_accuracy: 0.6343
Epoch 53/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3219 - accuracy: 0.6329 - val_loss: 0.8861 - val_accuracy: 0.6397
Epoch 54/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3193 - accuracy: 0.6358 - val_loss: 0.9072 - val_accuracy: 0.6273
Epoch 55/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3196 - accuracy: 0.6353 - val_loss: 0.9479 - val_accuracy: 0.6177
Epoch 56/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3192 - accuracy: 0.6371 - val_loss: 0.9702 - val_accuracy: 0.6083
Epoch 57/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3215 - accuracy: 0.6342 - val_loss: 0.9151 - val_accuracy: 0.6127
Epoch 58/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3190 - accuracy: 0.6379 - val_loss: 0.9698 - val_accuracy: 0.6063
Epoch 59/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3192 - accuracy: 0.6368 - val_loss: 0.8744 - val_accuracy: 0.6467
Epoch 60/75
394/394 [==============================] - 3s 7ms/step - loss: 0.3199 - accuracy: 0.6352 - val_loss: 0.9177 - val_accuracy: 0.6317
Epoch 61/75
394/394 [==============================] - 3s 7ms/step - loss: 0.3177 - accuracy: 0.6380 - val_loss: 0.9334 - val_accuracy: 0.6143
Epoch 62/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3183 - accuracy: 0.6380 - val_loss: 0.9418 - val_accuracy: 0.6110
Epoch 63/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3177 - accuracy: 0.6396 - val_loss: 0.9471 - val_accuracy: 0.6153
Epoch 64/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3181 - accuracy: 0.6394 - val_loss: 0.9090 - val_accuracy: 0.6243
Epoch 65/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3176 - accuracy: 0.6390 - val_loss: 0.9003 - val_accuracy: 0.6310
Epoch 66/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3188 - accuracy: 0.6368 - val_loss: 0.8996 - val_accuracy: 0.6417
Epoch 67/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3166 - accuracy: 0.6410 - val_loss: 0.8762 - val_accuracy: 0.6510
Epoch 68/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3194 - accuracy: 0.6364 - val_loss: 0.8930 - val_accuracy: 0.6407
Epoch 69/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3182 - accuracy: 0.6399 - val_loss: 0.9059 - val_accuracy: 0.6413
Epoch 70/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3191 - accuracy: 0.6375 - val_loss: 0.9700 - val_accuracy: 0.6023
Epoch 71/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3178 - accuracy: 0.6390 - val_loss: 0.8677 - val_accuracy: 0.6570
Epoch 72/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3192 - accuracy: 0.6369 - val_loss: 0.8830 - val_accuracy: 0.6303
Epoch 73/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3183 - accuracy: 0.6379 - val_loss: 0.8970 - val_accuracy: 0.6453
Epoch 74/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3173 - accuracy: 0.6378 - val_loss: 0.9130 - val_accuracy: 0.6267
Epoch 75/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3188 - accuracy: 0.6374 - val_loss: 0.8955 - val_accuracy: 0.6327

In [38]:

plot_network_summaries(history, 'best_model_B1-7.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_B1-7.hdf5', val_images, val_labels)

Optimal point:  71 epochs
94/94 [==============================] - 0s 2ms/step

94/94 [==============================] - 0s 2ms/step - loss: 0.8677 - accuracy: 0.6570
Test accuracy: 0.6570000052452087

The accuracy of the new model is much lower, while quickly exhausting the data in the training set.

B8 - Regularizer removed, epoch kept¶

The regularizer is removed, but a test is made with the increased epoch, to see if overfitting can be reached.

In [39]:

from tensorflow.keras import models, layers
from tensorflow.keras import regularizers

checkpoint = ModelCheckpoint(
    'best_model_B1-8.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

network = models.Sequential()
network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(128, activation='relu'))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=75, batch_size=512, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers from tensorflow.keras import regularizers checkpoint = ModelCheckpoint( 'best_model_B1-8.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) network = models.Sequential() network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(128, activation='relu')) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=75, batch_size=512, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/75
394/394 [==============================] - 3s 8ms/step - loss: 0.4381 - accuracy: 0.3731 - val_loss: 1.2854 - val_accuracy: 0.4463
Epoch 2/75
 23/394 [>.............................] - ETA: 2s - loss: 0.4002 - accuracy: 0.4671

c:\Projects\UoL\(DSM150-2023-OCT) Neural networks\CW1\.venv\Lib\site-packages\keras\src\engine\training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

394/394 [==============================] - 3s 8ms/step - loss: 0.3782 - accuracy: 0.5013 - val_loss: 1.1033 - val_accuracy: 0.5157
Epoch 3/75
394/394 [==============================] - 3s 8ms/step - loss: 0.3540 - accuracy: 0.5429 - val_loss: 1.0455 - val_accuracy: 0.5577
Epoch 4/75
394/394 [==============================] - 3s 7ms/step - loss: 0.3358 - accuracy: 0.5731 - val_loss: 1.0173 - val_accuracy: 0.5533
Epoch 5/75
394/394 [==============================] - 3s 7ms/step - loss: 0.3206 - accuracy: 0.5987 - val_loss: 0.9098 - val_accuracy: 0.6143
Epoch 6/75
394/394 [==============================] - 3s 7ms/step - loss: 0.3083 - accuracy: 0.6178 - val_loss: 0.9127 - val_accuracy: 0.6280
Epoch 7/75
394/394 [==============================] - 3s 7ms/step - loss: 0.3006 - accuracy: 0.6302 - val_loss: 0.8781 - val_accuracy: 0.6307
Epoch 8/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2922 - accuracy: 0.6437 - val_loss: 0.8553 - val_accuracy: 0.6497
Epoch 9/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2847 - accuracy: 0.6539 - val_loss: 0.8676 - val_accuracy: 0.6433
Epoch 10/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2780 - accuracy: 0.6634 - val_loss: 0.7972 - val_accuracy: 0.6737
Epoch 11/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2726 - accuracy: 0.6716 - val_loss: 0.7737 - val_accuracy: 0.6800
Epoch 12/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2671 - accuracy: 0.6797 - val_loss: 0.8264 - val_accuracy: 0.6713
Epoch 13/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2628 - accuracy: 0.6858 - val_loss: 0.7676 - val_accuracy: 0.6963
Epoch 14/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2575 - accuracy: 0.6925 - val_loss: 0.8614 - val_accuracy: 0.6553
Epoch 15/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2542 - accuracy: 0.6966 - val_loss: 0.7588 - val_accuracy: 0.6893
Epoch 16/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2490 - accuracy: 0.7052 - val_loss: 0.7628 - val_accuracy: 0.6970
Epoch 17/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2465 - accuracy: 0.7080 - val_loss: 0.7765 - val_accuracy: 0.6960
Epoch 18/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2422 - accuracy: 0.7132 - val_loss: 0.7598 - val_accuracy: 0.6923
Epoch 19/75
394/394 [==============================] - 3s 8ms/step - loss: 0.2388 - accuracy: 0.7174 - val_loss: 0.7529 - val_accuracy: 0.6913
Epoch 20/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2374 - accuracy: 0.7187 - val_loss: 0.7062 - val_accuracy: 0.7187
Epoch 21/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2324 - accuracy: 0.7255 - val_loss: 0.7651 - val_accuracy: 0.6993
Epoch 22/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2310 - accuracy: 0.7272 - val_loss: 0.7154 - val_accuracy: 0.7183
Epoch 23/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2265 - accuracy: 0.7328 - val_loss: 0.7408 - val_accuracy: 0.6983
Epoch 24/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2244 - accuracy: 0.7355 - val_loss: 0.7141 - val_accuracy: 0.7150
Epoch 25/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2216 - accuracy: 0.7393 - val_loss: 0.7123 - val_accuracy: 0.7203
Epoch 26/75
394/394 [==============================] - 3s 8ms/step - loss: 0.2179 - accuracy: 0.7435 - val_loss: 0.7061 - val_accuracy: 0.7120
Epoch 27/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2173 - accuracy: 0.7443 - val_loss: 0.7029 - val_accuracy: 0.7137
Epoch 28/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2140 - accuracy: 0.7486 - val_loss: 0.7025 - val_accuracy: 0.7240
Epoch 29/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2113 - accuracy: 0.7521 - val_loss: 0.6994 - val_accuracy: 0.7287
Epoch 30/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2092 - accuracy: 0.7537 - val_loss: 0.7154 - val_accuracy: 0.7097
Epoch 31/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2066 - accuracy: 0.7560 - val_loss: 0.7032 - val_accuracy: 0.7240
Epoch 32/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2031 - accuracy: 0.7600 - val_loss: 0.6955 - val_accuracy: 0.7230
Epoch 33/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2025 - accuracy: 0.7621 - val_loss: 0.6893 - val_accuracy: 0.7283
Epoch 34/75
394/394 [==============================] - 3s 7ms/step - loss: 0.2003 - accuracy: 0.7638 - val_loss: 0.7037 - val_accuracy: 0.7217
Epoch 35/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1973 - accuracy: 0.7682 - val_loss: 0.7011 - val_accuracy: 0.7273
Epoch 36/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1956 - accuracy: 0.7699 - val_loss: 0.7159 - val_accuracy: 0.7193
Epoch 37/75
394/394 [==============================] - 3s 8ms/step - loss: 0.1933 - accuracy: 0.7723 - val_loss: 0.6748 - val_accuracy: 0.7327
Epoch 38/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1913 - accuracy: 0.7752 - val_loss: 0.6964 - val_accuracy: 0.7287
Epoch 39/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1895 - accuracy: 0.7766 - val_loss: 0.6943 - val_accuracy: 0.7287
Epoch 40/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1884 - accuracy: 0.7782 - val_loss: 0.7020 - val_accuracy: 0.7243
Epoch 41/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1856 - accuracy: 0.7816 - val_loss: 0.6900 - val_accuracy: 0.7293
Epoch 42/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1843 - accuracy: 0.7822 - val_loss: 0.7223 - val_accuracy: 0.7233
Epoch 43/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1822 - accuracy: 0.7860 - val_loss: 0.7143 - val_accuracy: 0.7287
Epoch 44/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1807 - accuracy: 0.7870 - val_loss: 0.7206 - val_accuracy: 0.7270
Epoch 45/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1794 - accuracy: 0.7892 - val_loss: 0.6887 - val_accuracy: 0.7407
Epoch 46/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1773 - accuracy: 0.7911 - val_loss: 0.6817 - val_accuracy: 0.7433
Epoch 47/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1756 - accuracy: 0.7930 - val_loss: 0.7056 - val_accuracy: 0.7347
Epoch 48/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1730 - accuracy: 0.7952 - val_loss: 0.7263 - val_accuracy: 0.7280
Epoch 49/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1727 - accuracy: 0.7968 - val_loss: 0.6733 - val_accuracy: 0.7440
Epoch 50/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1705 - accuracy: 0.7995 - val_loss: 0.6864 - val_accuracy: 0.7437
Epoch 51/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1680 - accuracy: 0.8010 - val_loss: 0.6588 - val_accuracy: 0.7453
Epoch 52/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1682 - accuracy: 0.8010 - val_loss: 0.7241 - val_accuracy: 0.7240
Epoch 53/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1662 - accuracy: 0.8038 - val_loss: 0.7147 - val_accuracy: 0.7263
Epoch 54/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1660 - accuracy: 0.8038 - val_loss: 0.7162 - val_accuracy: 0.7327
Epoch 55/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1638 - accuracy: 0.8060 - val_loss: 0.7204 - val_accuracy: 0.7220
Epoch 56/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1613 - accuracy: 0.8088 - val_loss: 0.7056 - val_accuracy: 0.7393
Epoch 57/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1595 - accuracy: 0.8102 - val_loss: 0.6857 - val_accuracy: 0.7350
Epoch 58/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1578 - accuracy: 0.8136 - val_loss: 0.7310 - val_accuracy: 0.7433
Epoch 59/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1579 - accuracy: 0.8133 - val_loss: 0.6680 - val_accuracy: 0.7540
Epoch 60/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1574 - accuracy: 0.8133 - val_loss: 0.6647 - val_accuracy: 0.7517
Epoch 61/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1551 - accuracy: 0.8151 - val_loss: 0.6899 - val_accuracy: 0.7453
Epoch 62/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1529 - accuracy: 0.8181 - val_loss: 0.6757 - val_accuracy: 0.7453
Epoch 63/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1533 - accuracy: 0.8184 - val_loss: 0.7406 - val_accuracy: 0.7367
Epoch 64/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1513 - accuracy: 0.8196 - val_loss: 0.7293 - val_accuracy: 0.7310
Epoch 65/75
394/394 [==============================] - 3s 7ms/step - loss: 0.1492 - accuracy: 0.8221 - val_loss: 0.6826 - val_accuracy: 0.7517
Epoch 66/75
394/394 [==============================] - 3s 8ms/step - loss: 0.1473 - accuracy: 0.8241 - val_loss: 0.6866 - val_accuracy: 0.7450
Epoch 67/75
394/394 [==============================] - 3s 8ms/step - loss: 0.1465 - accuracy: 0.8249 - val_loss: 0.7000 - val_accuracy: 0.7470
Epoch 68/75
394/394 [==============================] - 3s 8ms/step - loss: 0.1452 - accuracy: 0.8264 - val_loss: 0.6971 - val_accuracy: 0.7503
Epoch 69/75
394/394 [==============================] - 3s 8ms/step - loss: 0.1458 - accuracy: 0.8251 - val_loss: 0.7139 - val_accuracy: 0.7507
Epoch 70/75
394/394 [==============================] - 3s 8ms/step - loss: 0.1434 - accuracy: 0.8291 - val_loss: 0.6814 - val_accuracy: 0.7507
Epoch 71/75
394/394 [==============================] - 3s 8ms/step - loss: 0.1431 - accuracy: 0.8291 - val_loss: 0.7002 - val_accuracy: 0.7423
Epoch 72/75
394/394 [==============================] - 3s 8ms/step - loss: 0.1400 - accuracy: 0.8321 - val_loss: 0.7193 - val_accuracy: 0.7460
Epoch 73/75
394/394 [==============================] - 3s 8ms/step - loss: 0.1385 - accuracy: 0.8344 - val_loss: 0.6905 - val_accuracy: 0.7487
Epoch 74/75
394/394 [==============================] - 3s 8ms/step - loss: 0.1380 - accuracy: 0.8345 - val_loss: 0.7115 - val_accuracy: 0.7350
Epoch 75/75
394/394 [==============================] - 3s 8ms/step - loss: 0.1373 - accuracy: 0.8345 - val_loss: 0.7126 - val_accuracy: 0.7470

In [40]:

plot_network_summaries(history, 'best_model_B1-8.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_B1-8.hdf5', val_images, val_labels)

Optimal point:  51 epochs
94/94 [==============================] - 0s 2ms/step

94/94 [==============================] - 0s 2ms/step - loss: 0.6588 - accuracy: 0.7453
Test accuracy: 0.7453333139419556

Increasing the epochs further improved the accuracy, overfitting is still uncertain.

B9 - Further Epoch increase¶

Epochs is increased to 150 to ensure that the point of overfitting is reached. Also experiment with bach size increase.

In [21]:

from tensorflow.keras import models, layers
from tensorflow.keras import regularizers

checkpoint = ModelCheckpoint(
    'best_model_B1-9.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

network = models.Sequential()
network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(128, activation='relu'))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=150, batch_size=1024, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers from tensorflow.keras import regularizers checkpoint = ModelCheckpoint( 'best_model_B1-9.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) network = models.Sequential() network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(128, activation='relu')) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=150, batch_size=1024, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/150
196/197 [============================>.] - ETA: 0s - loss: 0.4455 - accuracy: 0.3582

c:\Projects\UoL\(DSM150-2023-OCT) Neural networks\CW1\.venv\Lib\site-packages\keras\src\engine\training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

197/197 [==============================] - 4s 15ms/step - loss: 0.4454 - accuracy: 0.3586 - val_loss: 1.3722 - val_accuracy: 0.4003
Epoch 2/150
197/197 [==============================] - 3s 14ms/step - loss: 0.3909 - accuracy: 0.4741 - val_loss: 1.2253 - val_accuracy: 0.4527
Epoch 3/150
197/197 [==============================] - 3s 14ms/step - loss: 0.3648 - accuracy: 0.5192 - val_loss: 1.1246 - val_accuracy: 0.5217
Epoch 4/150
197/197 [==============================] - 3s 17ms/step - loss: 0.3464 - accuracy: 0.5505 - val_loss: 1.0561 - val_accuracy: 0.5550
Epoch 5/150
197/197 [==============================] - 3s 13ms/step - loss: 0.3327 - accuracy: 0.5742 - val_loss: 1.0728 - val_accuracy: 0.5530
Epoch 6/150
197/197 [==============================] - 3s 15ms/step - loss: 0.3199 - accuracy: 0.5951 - val_loss: 1.0162 - val_accuracy: 0.5707
Epoch 7/150
197/197 [==============================] - 5s 26ms/step - loss: 0.3098 - accuracy: 0.6125 - val_loss: 1.0113 - val_accuracy: 0.6000
Epoch 8/150
197/197 [==============================] - 5s 27ms/step - loss: 0.3009 - accuracy: 0.6256 - val_loss: 0.9358 - val_accuracy: 0.6103
Epoch 9/150
197/197 [==============================] - 3s 14ms/step - loss: 0.2926 - accuracy: 0.6388 - val_loss: 0.9432 - val_accuracy: 0.6200
Epoch 10/150
197/197 [==============================] - 3s 15ms/step - loss: 0.2847 - accuracy: 0.6502 - val_loss: 0.8572 - val_accuracy: 0.6483
Epoch 11/150
197/197 [==============================] - 3s 14ms/step - loss: 0.2779 - accuracy: 0.6608 - val_loss: 0.8608 - val_accuracy: 0.6473
Epoch 12/150
197/197 [==============================] - 3s 14ms/step - loss: 0.2726 - accuracy: 0.6691 - val_loss: 0.8742 - val_accuracy: 0.6567
Epoch 13/150
197/197 [==============================] - 3s 15ms/step - loss: 0.2677 - accuracy: 0.6750 - val_loss: 0.8589 - val_accuracy: 0.6570
Epoch 14/150
197/197 [==============================] - 3s 16ms/step - loss: 0.2627 - accuracy: 0.6819 - val_loss: 0.8246 - val_accuracy: 0.6660
Epoch 15/150
197/197 [==============================] - 3s 17ms/step - loss: 0.2600 - accuracy: 0.6853 - val_loss: 0.8538 - val_accuracy: 0.6603
Epoch 16/150
197/197 [==============================] - 3s 15ms/step - loss: 0.2556 - accuracy: 0.6914 - val_loss: 0.8746 - val_accuracy: 0.6577
Epoch 17/150
197/197 [==============================] - 4s 19ms/step - loss: 0.2506 - accuracy: 0.6984 - val_loss: 0.7961 - val_accuracy: 0.6830
Epoch 18/150
197/197 [==============================] - 3s 15ms/step - loss: 0.2463 - accuracy: 0.7039 - val_loss: 0.7829 - val_accuracy: 0.6870
Epoch 19/150
197/197 [==============================] - 3s 15ms/step - loss: 0.2439 - accuracy: 0.7066 - val_loss: 0.7938 - val_accuracy: 0.6787
Epoch 20/150
197/197 [==============================] - 3s 15ms/step - loss: 0.2398 - accuracy: 0.7128 - val_loss: 0.7595 - val_accuracy: 0.7013
Epoch 21/150
197/197 [==============================] - 3s 14ms/step - loss: 0.2365 - accuracy: 0.7167 - val_loss: 0.7824 - val_accuracy: 0.6870
Epoch 22/150
197/197 [==============================] - 3s 14ms/step - loss: 0.2345 - accuracy: 0.7191 - val_loss: 0.7659 - val_accuracy: 0.6927
Epoch 23/150
197/197 [==============================] - 3s 14ms/step - loss: 0.2297 - accuracy: 0.7254 - val_loss: 0.7742 - val_accuracy: 0.6883
Epoch 24/150
197/197 [==============================] - 3s 14ms/step - loss: 0.2273 - accuracy: 0.7280 - val_loss: 0.7951 - val_accuracy: 0.6887
Epoch 25/150
197/197 [==============================] - 3s 15ms/step - loss: 0.2242 - accuracy: 0.7324 - val_loss: 0.7757 - val_accuracy: 0.6943
Epoch 26/150
197/197 [==============================] - 3s 14ms/step - loss: 0.2223 - accuracy: 0.7351 - val_loss: 0.7470 - val_accuracy: 0.7103
Epoch 27/150
197/197 [==============================] - 3s 13ms/step - loss: 0.2195 - accuracy: 0.7393 - val_loss: 0.8306 - val_accuracy: 0.6883
Epoch 28/150
197/197 [==============================] - 3s 17ms/step - loss: 0.2168 - accuracy: 0.7429 - val_loss: 0.7387 - val_accuracy: 0.7180
Epoch 29/150
197/197 [==============================] - 3s 16ms/step - loss: 0.2138 - accuracy: 0.7447 - val_loss: 0.7240 - val_accuracy: 0.7167
Epoch 30/150
197/197 [==============================] - 3s 13ms/step - loss: 0.2128 - accuracy: 0.7470 - val_loss: 0.7878 - val_accuracy: 0.7010
Epoch 31/150
197/197 [==============================] - 3s 13ms/step - loss: 0.2094 - accuracy: 0.7511 - val_loss: 0.7285 - val_accuracy: 0.7203
Epoch 32/150
197/197 [==============================] - 2s 13ms/step - loss: 0.2059 - accuracy: 0.7549 - val_loss: 0.7519 - val_accuracy: 0.7050
Epoch 33/150
197/197 [==============================] - 3s 14ms/step - loss: 0.2048 - accuracy: 0.7562 - val_loss: 0.7432 - val_accuracy: 0.7120
Epoch 34/150
197/197 [==============================] - 3s 14ms/step - loss: 0.2023 - accuracy: 0.7599 - val_loss: 0.7379 - val_accuracy: 0.7137
Epoch 35/150
197/197 [==============================] - 4s 18ms/step - loss: 0.2001 - accuracy: 0.7620 - val_loss: 0.7197 - val_accuracy: 0.7310
Epoch 36/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1994 - accuracy: 0.7633 - val_loss: 0.7619 - val_accuracy: 0.7043
Epoch 37/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1975 - accuracy: 0.7640 - val_loss: 0.7490 - val_accuracy: 0.7093
Epoch 38/150
197/197 [==============================] - 4s 21ms/step - loss: 0.1948 - accuracy: 0.7673 - val_loss: 0.7026 - val_accuracy: 0.7273
Epoch 39/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1905 - accuracy: 0.7736 - val_loss: 0.7127 - val_accuracy: 0.7300
Epoch 40/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1897 - accuracy: 0.7744 - val_loss: 0.6983 - val_accuracy: 0.7293
Epoch 41/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1876 - accuracy: 0.7755 - val_loss: 0.6916 - val_accuracy: 0.7377
Epoch 42/150
197/197 [==============================] - 2s 12ms/step - loss: 0.1854 - accuracy: 0.7795 - val_loss: 0.7075 - val_accuracy: 0.7290
Epoch 43/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1829 - accuracy: 0.7826 - val_loss: 0.6945 - val_accuracy: 0.7310
Epoch 44/150
197/197 [==============================] - 3s 16ms/step - loss: 0.1814 - accuracy: 0.7828 - val_loss: 0.6862 - val_accuracy: 0.7417
Epoch 45/150
197/197 [==============================] - 2s 12ms/step - loss: 0.1788 - accuracy: 0.7872 - val_loss: 0.6956 - val_accuracy: 0.7377
Epoch 46/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1779 - accuracy: 0.7876 - val_loss: 0.6998 - val_accuracy: 0.7260
Epoch 47/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1764 - accuracy: 0.7894 - val_loss: 0.7048 - val_accuracy: 0.7377
Epoch 48/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1745 - accuracy: 0.7909 - val_loss: 0.7016 - val_accuracy: 0.7367
Epoch 49/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1717 - accuracy: 0.7936 - val_loss: 0.7096 - val_accuracy: 0.7347
Epoch 50/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1705 - accuracy: 0.7959 - val_loss: 0.7196 - val_accuracy: 0.7313
Epoch 51/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1686 - accuracy: 0.7984 - val_loss: 0.7141 - val_accuracy: 0.7407
Epoch 52/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1678 - accuracy: 0.7992 - val_loss: 0.6961 - val_accuracy: 0.7423
Epoch 53/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1644 - accuracy: 0.8030 - val_loss: 0.7061 - val_accuracy: 0.7450
Epoch 54/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1643 - accuracy: 0.8022 - val_loss: 0.7058 - val_accuracy: 0.7443
Epoch 55/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1623 - accuracy: 0.8055 - val_loss: 0.7133 - val_accuracy: 0.7330
Epoch 56/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1598 - accuracy: 0.8080 - val_loss: 0.7189 - val_accuracy: 0.7323
Epoch 57/150
197/197 [==============================] - 3s 16ms/step - loss: 0.1598 - accuracy: 0.8079 - val_loss: 0.7247 - val_accuracy: 0.7410
Epoch 58/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1589 - accuracy: 0.8088 - val_loss: 0.7090 - val_accuracy: 0.7463
Epoch 59/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1562 - accuracy: 0.8114 - val_loss: 0.6919 - val_accuracy: 0.7407
Epoch 60/150
197/197 [==============================] - 3s 16ms/step - loss: 0.1548 - accuracy: 0.8138 - val_loss: 0.6965 - val_accuracy: 0.7503
Epoch 61/150
197/197 [==============================] - 3s 16ms/step - loss: 0.1515 - accuracy: 0.8177 - val_loss: 0.6837 - val_accuracy: 0.7503
Epoch 62/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1506 - accuracy: 0.8186 - val_loss: 0.7148 - val_accuracy: 0.7413
Epoch 63/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1488 - accuracy: 0.8210 - val_loss: 0.6940 - val_accuracy: 0.7463
Epoch 64/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1480 - accuracy: 0.8199 - val_loss: 0.6797 - val_accuracy: 0.7550
Epoch 65/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1490 - accuracy: 0.8202 - val_loss: 0.7060 - val_accuracy: 0.7403
Epoch 66/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1447 - accuracy: 0.8249 - val_loss: 0.6829 - val_accuracy: 0.7553
Epoch 67/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1416 - accuracy: 0.8280 - val_loss: 0.7116 - val_accuracy: 0.7437
Epoch 68/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1432 - accuracy: 0.8260 - val_loss: 0.7052 - val_accuracy: 0.7560
Epoch 69/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1409 - accuracy: 0.8292 - val_loss: 0.6887 - val_accuracy: 0.7447
Epoch 70/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1406 - accuracy: 0.8301 - val_loss: 0.6834 - val_accuracy: 0.7600
Epoch 71/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1391 - accuracy: 0.8311 - val_loss: 0.6860 - val_accuracy: 0.7543
Epoch 72/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1355 - accuracy: 0.8362 - val_loss: 0.6976 - val_accuracy: 0.7540
Epoch 73/150
197/197 [==============================] - 3s 16ms/step - loss: 0.1372 - accuracy: 0.8335 - val_loss: 0.7073 - val_accuracy: 0.7517
Epoch 74/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1352 - accuracy: 0.8353 - val_loss: 0.7185 - val_accuracy: 0.7500
Epoch 75/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1339 - accuracy: 0.8366 - val_loss: 0.6923 - val_accuracy: 0.7617
Epoch 76/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1334 - accuracy: 0.8379 - val_loss: 0.7033 - val_accuracy: 0.7533
Epoch 77/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1304 - accuracy: 0.8406 - val_loss: 0.7223 - val_accuracy: 0.7507
Epoch 78/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1289 - accuracy: 0.8433 - val_loss: 0.6989 - val_accuracy: 0.7480
Epoch 79/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1286 - accuracy: 0.8429 - val_loss: 0.7029 - val_accuracy: 0.7553
Epoch 80/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1278 - accuracy: 0.8430 - val_loss: 0.7106 - val_accuracy: 0.7530
Epoch 81/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1269 - accuracy: 0.8444 - val_loss: 0.7701 - val_accuracy: 0.7327
Epoch 82/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1253 - accuracy: 0.8462 - val_loss: 0.7105 - val_accuracy: 0.7573
Epoch 83/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1252 - accuracy: 0.8459 - val_loss: 0.7354 - val_accuracy: 0.7450
Epoch 84/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1235 - accuracy: 0.8482 - val_loss: 0.7511 - val_accuracy: 0.7460
Epoch 85/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1210 - accuracy: 0.8519 - val_loss: 0.7709 - val_accuracy: 0.7457
Epoch 86/150
197/197 [==============================] - 3s 17ms/step - loss: 0.1221 - accuracy: 0.8497 - val_loss: 0.7670 - val_accuracy: 0.7453
Epoch 87/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1189 - accuracy: 0.8533 - val_loss: 0.7134 - val_accuracy: 0.7563
Epoch 88/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1186 - accuracy: 0.8539 - val_loss: 0.7162 - val_accuracy: 0.7607
Epoch 89/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1178 - accuracy: 0.8552 - val_loss: 0.7248 - val_accuracy: 0.7603
Epoch 90/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1152 - accuracy: 0.8580 - val_loss: 0.7328 - val_accuracy: 0.7510
Epoch 91/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1135 - accuracy: 0.8604 - val_loss: 0.7198 - val_accuracy: 0.7610
Epoch 92/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1147 - accuracy: 0.8579 - val_loss: 0.7362 - val_accuracy: 0.7573
Epoch 93/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1137 - accuracy: 0.8588 - val_loss: 0.7104 - val_accuracy: 0.7547
Epoch 94/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1123 - accuracy: 0.8603 - val_loss: 0.7553 - val_accuracy: 0.7490
Epoch 95/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1106 - accuracy: 0.8626 - val_loss: 0.7417 - val_accuracy: 0.7623
Epoch 96/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1113 - accuracy: 0.8620 - val_loss: 0.7608 - val_accuracy: 0.7483
Epoch 97/150
197/197 [==============================] - 3s 13ms/step - loss: 0.1118 - accuracy: 0.8608 - val_loss: 0.7583 - val_accuracy: 0.7503
Epoch 98/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1083 - accuracy: 0.8653 - val_loss: 0.7401 - val_accuracy: 0.7620
Epoch 99/150
197/197 [==============================] - 3s 16ms/step - loss: 0.1092 - accuracy: 0.8642 - val_loss: 0.7554 - val_accuracy: 0.7567
Epoch 100/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1051 - accuracy: 0.8685 - val_loss: 0.7589 - val_accuracy: 0.7580
Epoch 101/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1074 - accuracy: 0.8661 - val_loss: 0.7340 - val_accuracy: 0.7603
Epoch 102/150
197/197 [==============================] - 3s 16ms/step - loss: 0.1090 - accuracy: 0.8643 - val_loss: 0.7329 - val_accuracy: 0.7597
Epoch 103/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1049 - accuracy: 0.8688 - val_loss: 0.7626 - val_accuracy: 0.7543
Epoch 104/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1030 - accuracy: 0.8712 - val_loss: 0.8069 - val_accuracy: 0.7477
Epoch 105/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1044 - accuracy: 0.8697 - val_loss: 0.7783 - val_accuracy: 0.7577
Epoch 106/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1002 - accuracy: 0.8745 - val_loss: 0.7523 - val_accuracy: 0.7657
Epoch 107/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0992 - accuracy: 0.8754 - val_loss: 0.7360 - val_accuracy: 0.7707
Epoch 108/150
197/197 [==============================] - 3s 14ms/step - loss: 0.1013 - accuracy: 0.8722 - val_loss: 0.7586 - val_accuracy: 0.7580
Epoch 109/150
197/197 [==============================] - 3s 15ms/step - loss: 0.1032 - accuracy: 0.8712 - val_loss: 0.7629 - val_accuracy: 0.7650
Epoch 110/150
197/197 [==============================] - 3s 15ms/step - loss: 0.0970 - accuracy: 0.8779 - val_loss: 0.7546 - val_accuracy: 0.7637
Epoch 111/150
197/197 [==============================] - 3s 15ms/step - loss: 0.0961 - accuracy: 0.8792 - val_loss: 0.7628 - val_accuracy: 0.7683
Epoch 112/150
197/197 [==============================] - 3s 16ms/step - loss: 0.0973 - accuracy: 0.8773 - val_loss: 0.7869 - val_accuracy: 0.7630
Epoch 113/150
197/197 [==============================] - 3s 15ms/step - loss: 0.0972 - accuracy: 0.8780 - val_loss: 0.8215 - val_accuracy: 0.7550
Epoch 114/150
197/197 [==============================] - 3s 15ms/step - loss: 0.0946 - accuracy: 0.8810 - val_loss: 0.7504 - val_accuracy: 0.7630
Epoch 115/150
197/197 [==============================] - 3s 16ms/step - loss: 0.0957 - accuracy: 0.8789 - val_loss: 0.7813 - val_accuracy: 0.7623
Epoch 116/150
197/197 [==============================] - 3s 15ms/step - loss: 0.0957 - accuracy: 0.8795 - val_loss: 0.8099 - val_accuracy: 0.7590
Epoch 117/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0949 - accuracy: 0.8802 - val_loss: 0.7915 - val_accuracy: 0.7673
Epoch 118/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0919 - accuracy: 0.8837 - val_loss: 0.8019 - val_accuracy: 0.7673
Epoch 119/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0941 - accuracy: 0.8813 - val_loss: 0.7964 - val_accuracy: 0.7580
Epoch 120/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0903 - accuracy: 0.8852 - val_loss: 0.8172 - val_accuracy: 0.7483
Epoch 121/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0933 - accuracy: 0.8822 - val_loss: 0.8247 - val_accuracy: 0.7607
Epoch 122/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0889 - accuracy: 0.8864 - val_loss: 0.8079 - val_accuracy: 0.7617
Epoch 123/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0873 - accuracy: 0.8890 - val_loss: 0.8135 - val_accuracy: 0.7670
Epoch 124/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0898 - accuracy: 0.8866 - val_loss: 0.7839 - val_accuracy: 0.7630
Epoch 125/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0879 - accuracy: 0.8877 - val_loss: 0.8104 - val_accuracy: 0.7677
Epoch 126/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0881 - accuracy: 0.8876 - val_loss: 0.7947 - val_accuracy: 0.7710
Epoch 127/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0862 - accuracy: 0.8906 - val_loss: 0.8456 - val_accuracy: 0.7660
Epoch 128/150
197/197 [==============================] - 3s 15ms/step - loss: 0.0878 - accuracy: 0.8878 - val_loss: 0.8522 - val_accuracy: 0.7603
Epoch 129/150
197/197 [==============================] - 3s 15ms/step - loss: 0.0874 - accuracy: 0.8893 - val_loss: 0.7905 - val_accuracy: 0.7680
Epoch 130/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0839 - accuracy: 0.8927 - val_loss: 0.7910 - val_accuracy: 0.7737
Epoch 131/150
197/197 [==============================] - 3s 15ms/step - loss: 0.0834 - accuracy: 0.8933 - val_loss: 0.8091 - val_accuracy: 0.7660
Epoch 132/150
197/197 [==============================] - 3s 15ms/step - loss: 0.0841 - accuracy: 0.8924 - val_loss: 0.8171 - val_accuracy: 0.7750
Epoch 133/150
197/197 [==============================] - 3s 15ms/step - loss: 0.0853 - accuracy: 0.8911 - val_loss: 0.8397 - val_accuracy: 0.7633
Epoch 134/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0856 - accuracy: 0.8904 - val_loss: 0.8222 - val_accuracy: 0.7697
Epoch 135/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0792 - accuracy: 0.8983 - val_loss: 0.8356 - val_accuracy: 0.7613
Epoch 136/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0802 - accuracy: 0.8964 - val_loss: 0.8111 - val_accuracy: 0.7683
Epoch 137/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0811 - accuracy: 0.8963 - val_loss: 0.8303 - val_accuracy: 0.7710
Epoch 138/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0815 - accuracy: 0.8948 - val_loss: 0.8464 - val_accuracy: 0.7713
Epoch 139/150
197/197 [==============================] - 3s 15ms/step - loss: 0.0823 - accuracy: 0.8944 - val_loss: 0.8497 - val_accuracy: 0.7613
Epoch 140/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0811 - accuracy: 0.8952 - val_loss: 0.9061 - val_accuracy: 0.7450
Epoch 141/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0800 - accuracy: 0.8979 - val_loss: 0.8444 - val_accuracy: 0.7647
Epoch 142/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0778 - accuracy: 0.8997 - val_loss: 0.9039 - val_accuracy: 0.7580
Epoch 143/150
197/197 [==============================] - 3s 16ms/step - loss: 0.0775 - accuracy: 0.8999 - val_loss: 0.8500 - val_accuracy: 0.7697
Epoch 144/150
197/197 [==============================] - 3s 16ms/step - loss: 0.0756 - accuracy: 0.9022 - val_loss: 0.8447 - val_accuracy: 0.7703
Epoch 145/150
197/197 [==============================] - 3s 15ms/step - loss: 0.0788 - accuracy: 0.8983 - val_loss: 0.9667 - val_accuracy: 0.7450
Epoch 146/150
197/197 [==============================] - 3s 16ms/step - loss: 0.0820 - accuracy: 0.8954 - val_loss: 0.8591 - val_accuracy: 0.7683
Epoch 147/150
197/197 [==============================] - 3s 16ms/step - loss: 0.0763 - accuracy: 0.9020 - val_loss: 0.8395 - val_accuracy: 0.7657
Epoch 148/150
197/197 [==============================] - 3s 15ms/step - loss: 0.0764 - accuracy: 0.9010 - val_loss: 0.8419 - val_accuracy: 0.7720
Epoch 149/150
197/197 [==============================] - 3s 14ms/step - loss: 0.0729 - accuracy: 0.9052 - val_loss: 0.9097 - val_accuracy: 0.7580
Epoch 150/150
197/197 [==============================] - 3s 16ms/step - loss: 0.0743 - accuracy: 0.9041 - val_loss: 0.8867 - val_accuracy: 0.7650

In [22]:

plot_network_summaries(history, 'best_model_B1-9.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_B1-9.hdf5', val_images, val_labels)

Optimal point:  64 epochs
94/94 [==============================] - 0s 2ms/step

94/94 [==============================] - 0s 2ms/step - loss: 0.6797 - accuracy: 0.7550
Test accuracy: 0.7549999952316284

The point of overfitting is determined to be reached at about 60 epochs, and have the loss increased afterward.

B10 - Smaller second dense layer¶

In [19]:

from tensorflow.keras import models, layers
from tensorflow.keras import regularizers

checkpoint = ModelCheckpoint(
    'best_model_B1-10.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

network = models.Sequential()
network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(64, activation='relu'))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=150, batch_size=512, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers from tensorflow.keras import regularizers checkpoint = ModelCheckpoint( 'best_model_B1-10.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) network = models.Sequential() network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(64, activation='relu')) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=150, batch_size=512, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/150
394/394 [==============================] - 3s 7ms/step - loss: 0.4422 - accuracy: 0.3519 - val_loss: 1.2981 - val_accuracy: 0.4317
Epoch 2/150
394/394 [==============================] - 3s 7ms/step - loss: 0.3843 - accuracy: 0.4845 - val_loss: 1.1761 - val_accuracy: 0.4867
Epoch 3/150
394/394 [==============================] - 3s 7ms/step - loss: 0.3591 - accuracy: 0.5300 - val_loss: 1.0681 - val_accuracy: 0.5437
Epoch 4/150
394/394 [==============================] - 3s 7ms/step - loss: 0.3400 - accuracy: 0.5630 - val_loss: 1.0493 - val_accuracy: 0.5657
Epoch 5/150
394/394 [==============================] - 3s 6ms/step - loss: 0.3265 - accuracy: 0.5885 - val_loss: 1.0006 - val_accuracy: 0.5970
Epoch 6/150
394/394 [==============================] - 3s 6ms/step - loss: 0.3134 - accuracy: 0.6095 - val_loss: 0.9290 - val_accuracy: 0.6180
Epoch 7/150
394/394 [==============================] - 3s 7ms/step - loss: 0.3038 - accuracy: 0.6260 - val_loss: 0.8559 - val_accuracy: 0.6443
Epoch 8/150
394/394 [==============================] - 3s 6ms/step - loss: 0.2943 - accuracy: 0.6408 - val_loss: 0.9024 - val_accuracy: 0.6347
Epoch 9/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2875 - accuracy: 0.6505 - val_loss: 0.8443 - val_accuracy: 0.6580
Epoch 10/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2827 - accuracy: 0.6570 - val_loss: 0.8304 - val_accuracy: 0.6597
Epoch 11/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2760 - accuracy: 0.6669 - val_loss: 0.7782 - val_accuracy: 0.6873
Epoch 12/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2713 - accuracy: 0.6741 - val_loss: 0.8039 - val_accuracy: 0.6763
Epoch 13/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2658 - accuracy: 0.6808 - val_loss: 0.8113 - val_accuracy: 0.6753
Epoch 14/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2633 - accuracy: 0.6843 - val_loss: 0.7977 - val_accuracy: 0.6717
Epoch 15/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2591 - accuracy: 0.6895 - val_loss: 0.7729 - val_accuracy: 0.6860
Epoch 16/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2549 - accuracy: 0.6947 - val_loss: 0.7691 - val_accuracy: 0.6827
Epoch 17/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2524 - accuracy: 0.6994 - val_loss: 0.7542 - val_accuracy: 0.6993
Epoch 18/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2488 - accuracy: 0.7032 - val_loss: 0.7760 - val_accuracy: 0.6863
Epoch 19/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2463 - accuracy: 0.7070 - val_loss: 0.7527 - val_accuracy: 0.6923
Epoch 20/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2428 - accuracy: 0.7118 - val_loss: 0.7438 - val_accuracy: 0.6930
Epoch 21/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2410 - accuracy: 0.7147 - val_loss: 0.7422 - val_accuracy: 0.6957
Epoch 22/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2371 - accuracy: 0.7188 - val_loss: 0.7322 - val_accuracy: 0.7057
Epoch 23/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2355 - accuracy: 0.7208 - val_loss: 0.7553 - val_accuracy: 0.6900
Epoch 24/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2329 - accuracy: 0.7245 - val_loss: 0.7259 - val_accuracy: 0.7033
Epoch 25/150
394/394 [==============================] - 3s 6ms/step - loss: 0.2298 - accuracy: 0.7282 - val_loss: 0.7282 - val_accuracy: 0.6893
Epoch 26/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2270 - accuracy: 0.7311 - val_loss: 0.7166 - val_accuracy: 0.7090
Epoch 27/150
394/394 [==============================] - 3s 6ms/step - loss: 0.2261 - accuracy: 0.7321 - val_loss: 0.7450 - val_accuracy: 0.6950
Epoch 28/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2232 - accuracy: 0.7369 - val_loss: 0.7290 - val_accuracy: 0.7057
Epoch 29/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2217 - accuracy: 0.7392 - val_loss: 0.7419 - val_accuracy: 0.7013
Epoch 30/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2186 - accuracy: 0.7430 - val_loss: 0.7427 - val_accuracy: 0.7047
Epoch 31/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2174 - accuracy: 0.7434 - val_loss: 0.7389 - val_accuracy: 0.7047
Epoch 32/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2150 - accuracy: 0.7463 - val_loss: 0.6994 - val_accuracy: 0.7173
Epoch 33/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2126 - accuracy: 0.7490 - val_loss: 0.7317 - val_accuracy: 0.7110
Epoch 34/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2103 - accuracy: 0.7518 - val_loss: 0.6935 - val_accuracy: 0.7273
Epoch 35/150
394/394 [==============================] - 3s 6ms/step - loss: 0.2084 - accuracy: 0.7550 - val_loss: 0.7162 - val_accuracy: 0.7090
Epoch 36/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2074 - accuracy: 0.7554 - val_loss: 0.7955 - val_accuracy: 0.6867
Epoch 37/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2054 - accuracy: 0.7573 - val_loss: 0.7301 - val_accuracy: 0.7153
Epoch 38/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2042 - accuracy: 0.7597 - val_loss: 0.6883 - val_accuracy: 0.7253
Epoch 39/150
394/394 [==============================] - 3s 6ms/step - loss: 0.2016 - accuracy: 0.7608 - val_loss: 0.7475 - val_accuracy: 0.7060
Epoch 40/150
394/394 [==============================] - 3s 7ms/step - loss: 0.2010 - accuracy: 0.7637 - val_loss: 0.6781 - val_accuracy: 0.7290
Epoch 41/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1984 - accuracy: 0.7665 - val_loss: 0.7144 - val_accuracy: 0.7157
Epoch 42/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1969 - accuracy: 0.7678 - val_loss: 0.6772 - val_accuracy: 0.7367
Epoch 43/150
394/394 [==============================] - 3s 6ms/step - loss: 0.1958 - accuracy: 0.7697 - val_loss: 0.6968 - val_accuracy: 0.7277
Epoch 44/150
394/394 [==============================] - 3s 6ms/step - loss: 0.1930 - accuracy: 0.7717 - val_loss: 0.7053 - val_accuracy: 0.7233
Epoch 45/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1938 - accuracy: 0.7718 - val_loss: 0.7215 - val_accuracy: 0.7163
Epoch 46/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1908 - accuracy: 0.7749 - val_loss: 0.6772 - val_accuracy: 0.7353
Epoch 47/150
394/394 [==============================] - 3s 6ms/step - loss: 0.1892 - accuracy: 0.7773 - val_loss: 0.6802 - val_accuracy: 0.7290
Epoch 48/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1874 - accuracy: 0.7787 - val_loss: 0.7131 - val_accuracy: 0.7253
Epoch 49/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1863 - accuracy: 0.7801 - val_loss: 0.7076 - val_accuracy: 0.7220
Epoch 50/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1848 - accuracy: 0.7818 - val_loss: 0.7247 - val_accuracy: 0.7240
Epoch 51/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1832 - accuracy: 0.7829 - val_loss: 0.6956 - val_accuracy: 0.7220
Epoch 52/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1820 - accuracy: 0.7854 - val_loss: 0.7166 - val_accuracy: 0.7237
Epoch 53/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1807 - accuracy: 0.7863 - val_loss: 0.6856 - val_accuracy: 0.7343
Epoch 54/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1790 - accuracy: 0.7875 - val_loss: 0.7045 - val_accuracy: 0.7247
Epoch 55/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1782 - accuracy: 0.7883 - val_loss: 0.7097 - val_accuracy: 0.7230
Epoch 56/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1761 - accuracy: 0.7908 - val_loss: 0.6799 - val_accuracy: 0.7377
Epoch 57/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1754 - accuracy: 0.7928 - val_loss: 0.6868 - val_accuracy: 0.7360
Epoch 58/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1745 - accuracy: 0.7935 - val_loss: 0.7141 - val_accuracy: 0.7250
Epoch 59/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1721 - accuracy: 0.7959 - val_loss: 0.6911 - val_accuracy: 0.7320
Epoch 60/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1715 - accuracy: 0.7966 - val_loss: 0.6827 - val_accuracy: 0.7293
Epoch 61/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1703 - accuracy: 0.7989 - val_loss: 0.6851 - val_accuracy: 0.7350
Epoch 62/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1703 - accuracy: 0.7981 - val_loss: 0.6902 - val_accuracy: 0.7297
Epoch 63/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1678 - accuracy: 0.8005 - val_loss: 0.6834 - val_accuracy: 0.7327
Epoch 64/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1658 - accuracy: 0.8031 - val_loss: 0.6698 - val_accuracy: 0.7397
Epoch 65/150
394/394 [==============================] - 3s 6ms/step - loss: 0.1651 - accuracy: 0.8038 - val_loss: 0.6855 - val_accuracy: 0.7380
Epoch 66/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1640 - accuracy: 0.8046 - val_loss: 0.6786 - val_accuracy: 0.7437
Epoch 67/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1627 - accuracy: 0.8058 - val_loss: 0.6934 - val_accuracy: 0.7313
Epoch 68/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1613 - accuracy: 0.8085 - val_loss: 0.6951 - val_accuracy: 0.7360
Epoch 69/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1603 - accuracy: 0.8091 - val_loss: 0.6891 - val_accuracy: 0.7337
Epoch 70/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1594 - accuracy: 0.8095 - val_loss: 0.6972 - val_accuracy: 0.7363
Epoch 71/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1576 - accuracy: 0.8111 - val_loss: 0.6985 - val_accuracy: 0.7400
Epoch 72/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1572 - accuracy: 0.8125 - val_loss: 0.6945 - val_accuracy: 0.7340
Epoch 73/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1580 - accuracy: 0.8113 - val_loss: 0.7016 - val_accuracy: 0.7300
Epoch 74/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1550 - accuracy: 0.8148 - val_loss: 0.6961 - val_accuracy: 0.7303
Epoch 75/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1544 - accuracy: 0.8155 - val_loss: 0.7008 - val_accuracy: 0.7430
Epoch 76/150
394/394 [==============================] - 3s 8ms/step - loss: 0.1520 - accuracy: 0.8174 - val_loss: 0.7165 - val_accuracy: 0.7280
Epoch 77/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1530 - accuracy: 0.8173 - val_loss: 0.6725 - val_accuracy: 0.7467
Epoch 78/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1509 - accuracy: 0.8185 - val_loss: 0.7125 - val_accuracy: 0.7310
Epoch 79/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1499 - accuracy: 0.8194 - val_loss: 0.6970 - val_accuracy: 0.7487
Epoch 80/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1481 - accuracy: 0.8229 - val_loss: 0.7053 - val_accuracy: 0.7370
Epoch 81/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1477 - accuracy: 0.8226 - val_loss: 0.6673 - val_accuracy: 0.7467
Epoch 82/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1481 - accuracy: 0.8213 - val_loss: 0.6620 - val_accuracy: 0.7547
Epoch 83/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1464 - accuracy: 0.8241 - val_loss: 0.7076 - val_accuracy: 0.7380
Epoch 84/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1448 - accuracy: 0.8251 - val_loss: 0.7075 - val_accuracy: 0.7313
Epoch 85/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1448 - accuracy: 0.8251 - val_loss: 0.6703 - val_accuracy: 0.7553
Epoch 86/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1435 - accuracy: 0.8264 - val_loss: 0.6924 - val_accuracy: 0.7457
Epoch 87/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1423 - accuracy: 0.8284 - val_loss: 0.6987 - val_accuracy: 0.7460
Epoch 88/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1420 - accuracy: 0.8291 - val_loss: 0.7011 - val_accuracy: 0.7447
Epoch 89/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1406 - accuracy: 0.8299 - val_loss: 0.7127 - val_accuracy: 0.7347
Epoch 90/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1397 - accuracy: 0.8320 - val_loss: 0.6976 - val_accuracy: 0.7477
Epoch 91/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1387 - accuracy: 0.8317 - val_loss: 0.6937 - val_accuracy: 0.7497
Epoch 92/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1380 - accuracy: 0.8331 - val_loss: 0.6863 - val_accuracy: 0.7523
Epoch 93/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1386 - accuracy: 0.8325 - val_loss: 0.7055 - val_accuracy: 0.7520
Epoch 94/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1363 - accuracy: 0.8348 - val_loss: 0.7056 - val_accuracy: 0.7500
Epoch 95/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1350 - accuracy: 0.8359 - val_loss: 0.7614 - val_accuracy: 0.7360
Epoch 96/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1340 - accuracy: 0.8379 - val_loss: 0.7210 - val_accuracy: 0.7457
Epoch 97/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1344 - accuracy: 0.8381 - val_loss: 0.7344 - val_accuracy: 0.7350
Epoch 98/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1339 - accuracy: 0.8385 - val_loss: 0.7250 - val_accuracy: 0.7363
Epoch 99/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1331 - accuracy: 0.8388 - val_loss: 0.6853 - val_accuracy: 0.7550
Epoch 100/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1311 - accuracy: 0.8404 - val_loss: 0.7321 - val_accuracy: 0.7397
Epoch 101/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1300 - accuracy: 0.8415 - val_loss: 0.7099 - val_accuracy: 0.7453
Epoch 102/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1308 - accuracy: 0.8405 - val_loss: 0.6953 - val_accuracy: 0.7507
Epoch 103/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1294 - accuracy: 0.8423 - val_loss: 0.7298 - val_accuracy: 0.7437
Epoch 104/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1285 - accuracy: 0.8433 - val_loss: 0.7355 - val_accuracy: 0.7450
Epoch 105/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1272 - accuracy: 0.8442 - val_loss: 0.7183 - val_accuracy: 0.7557
Epoch 106/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1273 - accuracy: 0.8447 - val_loss: 0.7459 - val_accuracy: 0.7440
Epoch 107/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1270 - accuracy: 0.8450 - val_loss: 0.7085 - val_accuracy: 0.7520
Epoch 108/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1255 - accuracy: 0.8469 - val_loss: 0.7364 - val_accuracy: 0.7427
Epoch 109/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1242 - accuracy: 0.8485 - val_loss: 0.7194 - val_accuracy: 0.7397
Epoch 110/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1245 - accuracy: 0.8479 - val_loss: 0.7327 - val_accuracy: 0.7520
Epoch 111/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1226 - accuracy: 0.8493 - val_loss: 0.7093 - val_accuracy: 0.7613
Epoch 112/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1222 - accuracy: 0.8511 - val_loss: 0.7374 - val_accuracy: 0.7553
Epoch 113/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1233 - accuracy: 0.8493 - val_loss: 0.7241 - val_accuracy: 0.7500
Epoch 114/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1211 - accuracy: 0.8527 - val_loss: 0.7329 - val_accuracy: 0.7513
Epoch 115/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1219 - accuracy: 0.8506 - val_loss: 0.7347 - val_accuracy: 0.7467
Epoch 116/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1207 - accuracy: 0.8518 - val_loss: 0.7228 - val_accuracy: 0.7560
Epoch 117/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1194 - accuracy: 0.8530 - val_loss: 0.7455 - val_accuracy: 0.7430
Epoch 118/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1203 - accuracy: 0.8527 - val_loss: 0.7687 - val_accuracy: 0.7403
Epoch 119/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1192 - accuracy: 0.8532 - val_loss: 0.7304 - val_accuracy: 0.7550
Epoch 120/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1184 - accuracy: 0.8545 - val_loss: 0.7387 - val_accuracy: 0.7453
Epoch 121/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1165 - accuracy: 0.8569 - val_loss: 0.7147 - val_accuracy: 0.7637
Epoch 122/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1167 - accuracy: 0.8567 - val_loss: 0.7146 - val_accuracy: 0.7590
Epoch 123/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1167 - accuracy: 0.8570 - val_loss: 0.7566 - val_accuracy: 0.7503
Epoch 124/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1154 - accuracy: 0.8577 - val_loss: 0.7208 - val_accuracy: 0.7610
Epoch 125/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1143 - accuracy: 0.8592 - val_loss: 0.7508 - val_accuracy: 0.7543
Epoch 126/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1142 - accuracy: 0.8594 - val_loss: 0.7448 - val_accuracy: 0.7640
Epoch 127/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1142 - accuracy: 0.8589 - val_loss: 0.7629 - val_accuracy: 0.7503
Epoch 128/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1124 - accuracy: 0.8619 - val_loss: 0.7532 - val_accuracy: 0.7500
Epoch 129/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1128 - accuracy: 0.8608 - val_loss: 0.7645 - val_accuracy: 0.7567
Epoch 130/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1129 - accuracy: 0.8608 - val_loss: 0.7653 - val_accuracy: 0.7483
Epoch 131/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1132 - accuracy: 0.8602 - val_loss: 0.7902 - val_accuracy: 0.7473
Epoch 132/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1094 - accuracy: 0.8643 - val_loss: 0.7694 - val_accuracy: 0.7517
Epoch 133/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1079 - accuracy: 0.8657 - val_loss: 0.7835 - val_accuracy: 0.7487
Epoch 134/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1110 - accuracy: 0.8620 - val_loss: 0.7731 - val_accuracy: 0.7510
Epoch 135/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1102 - accuracy: 0.8637 - val_loss: 0.7483 - val_accuracy: 0.7617
Epoch 136/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1079 - accuracy: 0.8657 - val_loss: 0.7617 - val_accuracy: 0.7490
Epoch 137/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1073 - accuracy: 0.8669 - val_loss: 0.8086 - val_accuracy: 0.7557
Epoch 138/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1068 - accuracy: 0.8664 - val_loss: 0.7639 - val_accuracy: 0.7500
Epoch 139/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1085 - accuracy: 0.8659 - val_loss: 0.7886 - val_accuracy: 0.7487
Epoch 140/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1079 - accuracy: 0.8648 - val_loss: 0.8009 - val_accuracy: 0.7467
Epoch 141/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1065 - accuracy: 0.8674 - val_loss: 0.8217 - val_accuracy: 0.7440
Epoch 142/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1053 - accuracy: 0.8690 - val_loss: 0.7806 - val_accuracy: 0.7627
Epoch 143/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1042 - accuracy: 0.8688 - val_loss: 0.7817 - val_accuracy: 0.7603
Epoch 144/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1042 - accuracy: 0.8694 - val_loss: 0.7825 - val_accuracy: 0.7533
Epoch 145/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1046 - accuracy: 0.8691 - val_loss: 0.7715 - val_accuracy: 0.7573
Epoch 146/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1031 - accuracy: 0.8705 - val_loss: 0.8348 - val_accuracy: 0.7483
Epoch 147/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1020 - accuracy: 0.8718 - val_loss: 0.7914 - val_accuracy: 0.7603
Epoch 148/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1033 - accuracy: 0.8706 - val_loss: 0.7714 - val_accuracy: 0.7630
Epoch 149/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1005 - accuracy: 0.8734 - val_loss: 0.7662 - val_accuracy: 0.7617
Epoch 150/150
394/394 [==============================] - 3s 7ms/step - loss: 0.1013 - accuracy: 0.8725 - val_loss: 0.8027 - val_accuracy: 0.7500

In [22]:

plot_network_summaries(history, 'best_model_B1-10.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_B1-10.hdf5', val_images, val_labels)

Optimal point:  82 epochs
94/94 [==============================] - 0s 1ms/step

94/94 [==============================] - 0s 2ms/step - loss: 0.6620 - accuracy: 0.7547
Test accuracy: 0.7546666860580444

The smaller amount of neurons, has a similar accuracy score, while training faster.

B11 - Batch size increase¶

Since the larger batch size increase positively affected the earlier model, the new model is also tested with it.

In [26]:

from tensorflow.keras import models, layers
from tensorflow.keras import regularizers

checkpoint = ModelCheckpoint(
    'best_model_B1-11.hdf5',
    monitor='val_loss',
    verbose=False,
    save_best_only=True,
    mode='min'
)

network = models.Sequential()
network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(64, activation='relu'))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(train_images, train_labels, epochs=150, batch_size=1024, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

from tensorflow.keras import models, layers from tensorflow.keras import regularizers checkpoint = ModelCheckpoint( 'best_model_B1-11.hdf5', monitor='val_loss', verbose=False, save_best_only=True, mode='min' ) network = models.Sequential() network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(64, activation='relu')) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(train_images, train_labels, epochs=150, batch_size=1024, validation_data = (val_images, val_labels), class_weight=class_weight_dict, callbacks=[checkpoint])

Epoch 1/150
197/197 [==============================] - 3s 10ms/step - loss: 0.4496 - accuracy: 0.3371 - val_loss: 1.3282 - val_accuracy: 0.4097
Epoch 2/150
 12/197 [>.............................] - ETA: 1s - loss: 0.4092 - accuracy: 0.4402

c:\Projects\UoL\(DSM150-2023-OCT) Neural networks\CW1\.venv\Lib\site-packages\keras\src\engine\training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

197/197 [==============================] - 2s 10ms/step - loss: 0.3982 - accuracy: 0.4615 - val_loss: 1.2907 - val_accuracy: 0.4553
Epoch 3/150
197/197 [==============================] - 2s 10ms/step - loss: 0.3742 - accuracy: 0.5052 - val_loss: 1.1159 - val_accuracy: 0.5167
Epoch 4/150
197/197 [==============================] - 2s 10ms/step - loss: 0.3561 - accuracy: 0.5335 - val_loss: 1.0750 - val_accuracy: 0.5423
Epoch 5/150
197/197 [==============================] - 2s 10ms/step - loss: 0.3439 - accuracy: 0.5555 - val_loss: 1.0541 - val_accuracy: 0.5577
Epoch 6/150
197/197 [==============================] - 2s 11ms/step - loss: 0.3322 - accuracy: 0.5751 - val_loss: 1.0225 - val_accuracy: 0.5753
Epoch 7/150
197/197 [==============================] - 2s 10ms/step - loss: 0.3201 - accuracy: 0.5949 - val_loss: 0.9586 - val_accuracy: 0.5980
Epoch 8/150
197/197 [==============================] - 2s 10ms/step - loss: 0.3110 - accuracy: 0.6124 - val_loss: 0.9731 - val_accuracy: 0.5923
Epoch 9/150
197/197 [==============================] - 2s 10ms/step - loss: 0.3019 - accuracy: 0.6260 - val_loss: 0.8915 - val_accuracy: 0.6327
Epoch 10/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2951 - accuracy: 0.6376 - val_loss: 0.8670 - val_accuracy: 0.6390
Epoch 11/150
197/197 [==============================] - 2s 11ms/step - loss: 0.2907 - accuracy: 0.6431 - val_loss: 0.8423 - val_accuracy: 0.6533
Epoch 12/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2852 - accuracy: 0.6521 - val_loss: 0.8560 - val_accuracy: 0.6587
Epoch 13/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2789 - accuracy: 0.6614 - val_loss: 0.8252 - val_accuracy: 0.6660
Epoch 14/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2737 - accuracy: 0.6703 - val_loss: 0.8085 - val_accuracy: 0.6673
Epoch 15/150
197/197 [==============================] - 2s 11ms/step - loss: 0.2718 - accuracy: 0.6732 - val_loss: 0.7989 - val_accuracy: 0.6660
Epoch 16/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2660 - accuracy: 0.6803 - val_loss: 0.8244 - val_accuracy: 0.6663
Epoch 17/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2651 - accuracy: 0.6827 - val_loss: 0.8366 - val_accuracy: 0.6533
Epoch 18/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2608 - accuracy: 0.6876 - val_loss: 0.7989 - val_accuracy: 0.6790
Epoch 19/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2562 - accuracy: 0.6941 - val_loss: 0.7716 - val_accuracy: 0.6883
Epoch 20/150
197/197 [==============================] - 2s 11ms/step - loss: 0.2531 - accuracy: 0.6988 - val_loss: 0.7621 - val_accuracy: 0.6907
Epoch 21/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2513 - accuracy: 0.7017 - val_loss: 0.7697 - val_accuracy: 0.6847
Epoch 22/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2480 - accuracy: 0.7061 - val_loss: 0.7419 - val_accuracy: 0.7007
Epoch 23/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2466 - accuracy: 0.7082 - val_loss: 0.7533 - val_accuracy: 0.6910
Epoch 24/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2410 - accuracy: 0.7154 - val_loss: 0.7639 - val_accuracy: 0.6923
Epoch 25/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2400 - accuracy: 0.7150 - val_loss: 0.7917 - val_accuracy: 0.6707
Epoch 26/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2368 - accuracy: 0.7204 - val_loss: 0.7191 - val_accuracy: 0.7033
Epoch 27/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2338 - accuracy: 0.7242 - val_loss: 0.7429 - val_accuracy: 0.6957
Epoch 28/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2333 - accuracy: 0.7255 - val_loss: 0.7263 - val_accuracy: 0.7000
Epoch 29/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2296 - accuracy: 0.7300 - val_loss: 0.7949 - val_accuracy: 0.6867
Epoch 30/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2284 - accuracy: 0.7320 - val_loss: 0.7107 - val_accuracy: 0.7173
Epoch 31/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2254 - accuracy: 0.7353 - val_loss: 0.7116 - val_accuracy: 0.7020
Epoch 32/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2250 - accuracy: 0.7352 - val_loss: 0.7250 - val_accuracy: 0.6993
Epoch 33/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2213 - accuracy: 0.7419 - val_loss: 0.7259 - val_accuracy: 0.7097
Epoch 34/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2199 - accuracy: 0.7427 - val_loss: 0.7007 - val_accuracy: 0.7100
Epoch 35/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2171 - accuracy: 0.7469 - val_loss: 0.7212 - val_accuracy: 0.7073
Epoch 36/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2157 - accuracy: 0.7484 - val_loss: 0.7349 - val_accuracy: 0.7007
Epoch 37/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2150 - accuracy: 0.7494 - val_loss: 0.7028 - val_accuracy: 0.7167
Epoch 38/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2116 - accuracy: 0.7531 - val_loss: 0.7052 - val_accuracy: 0.7070
Epoch 39/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2115 - accuracy: 0.7536 - val_loss: 0.6900 - val_accuracy: 0.7227
Epoch 40/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2092 - accuracy: 0.7557 - val_loss: 0.7020 - val_accuracy: 0.7223
Epoch 41/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2068 - accuracy: 0.7582 - val_loss: 0.7001 - val_accuracy: 0.7183
Epoch 42/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2047 - accuracy: 0.7614 - val_loss: 0.6777 - val_accuracy: 0.7303
Epoch 43/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2054 - accuracy: 0.7618 - val_loss: 0.7362 - val_accuracy: 0.7123
Epoch 44/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2025 - accuracy: 0.7638 - val_loss: 0.7186 - val_accuracy: 0.7183
Epoch 45/150
197/197 [==============================] - 2s 10ms/step - loss: 0.2006 - accuracy: 0.7659 - val_loss: 0.6912 - val_accuracy: 0.7263
Epoch 46/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1988 - accuracy: 0.7688 - val_loss: 0.7067 - val_accuracy: 0.7167
Epoch 47/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1976 - accuracy: 0.7696 - val_loss: 0.6887 - val_accuracy: 0.7287
Epoch 48/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1953 - accuracy: 0.7728 - val_loss: 0.6945 - val_accuracy: 0.7270
Epoch 49/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1964 - accuracy: 0.7723 - val_loss: 0.7079 - val_accuracy: 0.7263
Epoch 50/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1944 - accuracy: 0.7744 - val_loss: 0.6936 - val_accuracy: 0.7207
Epoch 51/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1922 - accuracy: 0.7768 - val_loss: 0.7311 - val_accuracy: 0.7197
Epoch 52/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1922 - accuracy: 0.7772 - val_loss: 0.6819 - val_accuracy: 0.7293
Epoch 53/150
197/197 [==============================] - 2s 11ms/step - loss: 0.1876 - accuracy: 0.7808 - val_loss: 0.7027 - val_accuracy: 0.7270
Epoch 54/150
197/197 [==============================] - 2s 11ms/step - loss: 0.1865 - accuracy: 0.7833 - val_loss: 0.6878 - val_accuracy: 0.7337
Epoch 55/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1862 - accuracy: 0.7837 - val_loss: 0.7134 - val_accuracy: 0.7233
Epoch 56/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1836 - accuracy: 0.7861 - val_loss: 0.7261 - val_accuracy: 0.7170
Epoch 57/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1835 - accuracy: 0.7862 - val_loss: 0.6992 - val_accuracy: 0.7193
Epoch 58/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1828 - accuracy: 0.7864 - val_loss: 0.6772 - val_accuracy: 0.7383
Epoch 59/150
197/197 [==============================] - 2s 11ms/step - loss: 0.1804 - accuracy: 0.7906 - val_loss: 0.6729 - val_accuracy: 0.7450
Epoch 60/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1792 - accuracy: 0.7910 - val_loss: 0.6785 - val_accuracy: 0.7440
Epoch 61/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1783 - accuracy: 0.7923 - val_loss: 0.7166 - val_accuracy: 0.7170
Epoch 62/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1778 - accuracy: 0.7935 - val_loss: 0.6892 - val_accuracy: 0.7353
Epoch 63/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1764 - accuracy: 0.7946 - val_loss: 0.6792 - val_accuracy: 0.7387
Epoch 64/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1745 - accuracy: 0.7973 - val_loss: 0.7038 - val_accuracy: 0.7300
Epoch 65/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1741 - accuracy: 0.7966 - val_loss: 0.6557 - val_accuracy: 0.7433
Epoch 66/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1712 - accuracy: 0.8000 - val_loss: 0.6654 - val_accuracy: 0.7413
Epoch 67/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1699 - accuracy: 0.8016 - val_loss: 0.6655 - val_accuracy: 0.7417
Epoch 68/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1690 - accuracy: 0.8026 - val_loss: 0.6689 - val_accuracy: 0.7433
Epoch 69/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1681 - accuracy: 0.8034 - val_loss: 0.6858 - val_accuracy: 0.7397
Epoch 70/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1669 - accuracy: 0.8056 - val_loss: 0.6784 - val_accuracy: 0.7397
Epoch 71/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1659 - accuracy: 0.8068 - val_loss: 0.6964 - val_accuracy: 0.7347
Epoch 72/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1658 - accuracy: 0.8062 - val_loss: 0.6850 - val_accuracy: 0.7410
Epoch 73/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1628 - accuracy: 0.8088 - val_loss: 0.6846 - val_accuracy: 0.7423
Epoch 74/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1645 - accuracy: 0.8068 - val_loss: 0.6663 - val_accuracy: 0.7443
Epoch 75/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1620 - accuracy: 0.8098 - val_loss: 0.6808 - val_accuracy: 0.7373
Epoch 76/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1591 - accuracy: 0.8137 - val_loss: 0.6775 - val_accuracy: 0.7423
Epoch 77/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1595 - accuracy: 0.8133 - val_loss: 0.6915 - val_accuracy: 0.7410
Epoch 78/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1603 - accuracy: 0.8114 - val_loss: 0.6686 - val_accuracy: 0.7533
Epoch 79/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1569 - accuracy: 0.8154 - val_loss: 0.6592 - val_accuracy: 0.7507
Epoch 80/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1561 - accuracy: 0.8169 - val_loss: 0.6825 - val_accuracy: 0.7433
Epoch 81/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1559 - accuracy: 0.8170 - val_loss: 0.6803 - val_accuracy: 0.7427
Epoch 82/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1536 - accuracy: 0.8197 - val_loss: 0.6753 - val_accuracy: 0.7410
Epoch 83/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1524 - accuracy: 0.8202 - val_loss: 0.6770 - val_accuracy: 0.7503
Epoch 84/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1561 - accuracy: 0.8168 - val_loss: 0.6631 - val_accuracy: 0.7517
Epoch 85/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1509 - accuracy: 0.8226 - val_loss: 0.6754 - val_accuracy: 0.7490
Epoch 86/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1522 - accuracy: 0.8202 - val_loss: 0.6646 - val_accuracy: 0.7510
Epoch 87/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1499 - accuracy: 0.8240 - val_loss: 0.6642 - val_accuracy: 0.7483
Epoch 88/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1500 - accuracy: 0.8231 - val_loss: 0.6976 - val_accuracy: 0.7457
Epoch 89/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1490 - accuracy: 0.8237 - val_loss: 0.7493 - val_accuracy: 0.7257
Epoch 90/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1458 - accuracy: 0.8277 - val_loss: 0.6671 - val_accuracy: 0.7543
Epoch 91/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1460 - accuracy: 0.8278 - val_loss: 0.7184 - val_accuracy: 0.7443
Epoch 92/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1455 - accuracy: 0.8276 - val_loss: 0.6715 - val_accuracy: 0.7573
Epoch 93/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1420 - accuracy: 0.8319 - val_loss: 0.6642 - val_accuracy: 0.7480
Epoch 94/150
197/197 [==============================] - 2s 11ms/step - loss: 0.1435 - accuracy: 0.8303 - val_loss: 0.6829 - val_accuracy: 0.7510
Epoch 95/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1433 - accuracy: 0.8298 - val_loss: 0.6827 - val_accuracy: 0.7470
Epoch 96/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1426 - accuracy: 0.8312 - val_loss: 0.7149 - val_accuracy: 0.7353
Epoch 97/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1418 - accuracy: 0.8326 - val_loss: 0.7129 - val_accuracy: 0.7333
Epoch 98/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1398 - accuracy: 0.8351 - val_loss: 0.6843 - val_accuracy: 0.7463
Epoch 99/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1386 - accuracy: 0.8344 - val_loss: 0.6626 - val_accuracy: 0.7523
Epoch 100/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1378 - accuracy: 0.8361 - val_loss: 0.7031 - val_accuracy: 0.7460
Epoch 101/150
197/197 [==============================] - 2s 11ms/step - loss: 0.1375 - accuracy: 0.8355 - val_loss: 0.6543 - val_accuracy: 0.7640
Epoch 102/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1358 - accuracy: 0.8390 - val_loss: 0.6829 - val_accuracy: 0.7543
Epoch 103/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1361 - accuracy: 0.8373 - val_loss: 0.6567 - val_accuracy: 0.7593
Epoch 104/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1341 - accuracy: 0.8394 - val_loss: 0.6866 - val_accuracy: 0.7597
Epoch 105/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1344 - accuracy: 0.8403 - val_loss: 0.6846 - val_accuracy: 0.7573
Epoch 106/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1335 - accuracy: 0.8415 - val_loss: 0.6843 - val_accuracy: 0.7527
Epoch 107/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1326 - accuracy: 0.8417 - val_loss: 0.6750 - val_accuracy: 0.7580
Epoch 108/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1331 - accuracy: 0.8416 - val_loss: 0.6696 - val_accuracy: 0.7477
Epoch 109/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1317 - accuracy: 0.8422 - val_loss: 0.7005 - val_accuracy: 0.7497
Epoch 110/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1322 - accuracy: 0.8420 - val_loss: 0.6917 - val_accuracy: 0.7500
Epoch 111/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1274 - accuracy: 0.8473 - val_loss: 0.6990 - val_accuracy: 0.7563
Epoch 112/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1298 - accuracy: 0.8446 - val_loss: 0.7147 - val_accuracy: 0.7520
Epoch 113/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1292 - accuracy: 0.8457 - val_loss: 0.6898 - val_accuracy: 0.7550
Epoch 114/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1282 - accuracy: 0.8467 - val_loss: 0.7203 - val_accuracy: 0.7490
Epoch 115/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1265 - accuracy: 0.8486 - val_loss: 0.6960 - val_accuracy: 0.7537
Epoch 116/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1271 - accuracy: 0.8485 - val_loss: 0.7047 - val_accuracy: 0.7600
Epoch 117/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1257 - accuracy: 0.8493 - val_loss: 0.6976 - val_accuracy: 0.7557
Epoch 118/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1253 - accuracy: 0.8494 - val_loss: 0.6846 - val_accuracy: 0.7620
Epoch 119/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1267 - accuracy: 0.8485 - val_loss: 0.7096 - val_accuracy: 0.7517
Epoch 120/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1238 - accuracy: 0.8504 - val_loss: 0.6762 - val_accuracy: 0.7637
Epoch 121/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1217 - accuracy: 0.8523 - val_loss: 0.7350 - val_accuracy: 0.7523
Epoch 122/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1211 - accuracy: 0.8540 - val_loss: 0.7238 - val_accuracy: 0.7433
Epoch 123/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1210 - accuracy: 0.8549 - val_loss: 0.7236 - val_accuracy: 0.7560
Epoch 124/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1205 - accuracy: 0.8546 - val_loss: 0.7334 - val_accuracy: 0.7453
Epoch 125/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1199 - accuracy: 0.8550 - val_loss: 0.7269 - val_accuracy: 0.7477
Epoch 126/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1186 - accuracy: 0.8569 - val_loss: 0.7319 - val_accuracy: 0.7497
Epoch 127/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1202 - accuracy: 0.8557 - val_loss: 0.6947 - val_accuracy: 0.7523
Epoch 128/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1186 - accuracy: 0.8566 - val_loss: 0.7177 - val_accuracy: 0.7577
Epoch 129/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1174 - accuracy: 0.8581 - val_loss: 0.6858 - val_accuracy: 0.7550
Epoch 130/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1158 - accuracy: 0.8600 - val_loss: 0.7068 - val_accuracy: 0.7513
Epoch 131/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1164 - accuracy: 0.8588 - val_loss: 0.7473 - val_accuracy: 0.7460
Epoch 132/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1173 - accuracy: 0.8583 - val_loss: 0.6960 - val_accuracy: 0.7557
Epoch 133/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1141 - accuracy: 0.8608 - val_loss: 0.7119 - val_accuracy: 0.7577
Epoch 134/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1142 - accuracy: 0.8613 - val_loss: 0.7088 - val_accuracy: 0.7520
Epoch 135/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1131 - accuracy: 0.8621 - val_loss: 0.7281 - val_accuracy: 0.7497
Epoch 136/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1116 - accuracy: 0.8644 - val_loss: 0.6955 - val_accuracy: 0.7570
Epoch 137/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1111 - accuracy: 0.8652 - val_loss: 0.7183 - val_accuracy: 0.7600
Epoch 138/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1114 - accuracy: 0.8644 - val_loss: 0.7189 - val_accuracy: 0.7593
Epoch 139/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1132 - accuracy: 0.8622 - val_loss: 0.7276 - val_accuracy: 0.7620
Epoch 140/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1109 - accuracy: 0.8655 - val_loss: 0.7276 - val_accuracy: 0.7533
Epoch 141/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1095 - accuracy: 0.8673 - val_loss: 0.7353 - val_accuracy: 0.7497
Epoch 142/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1097 - accuracy: 0.8663 - val_loss: 0.7370 - val_accuracy: 0.7513
Epoch 143/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1106 - accuracy: 0.8660 - val_loss: 0.7133 - val_accuracy: 0.7667
Epoch 144/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1098 - accuracy: 0.8663 - val_loss: 0.7189 - val_accuracy: 0.7550
Epoch 145/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1087 - accuracy: 0.8681 - val_loss: 0.7265 - val_accuracy: 0.7540
Epoch 146/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1075 - accuracy: 0.8682 - val_loss: 0.7352 - val_accuracy: 0.7547
Epoch 147/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1067 - accuracy: 0.8688 - val_loss: 0.7319 - val_accuracy: 0.7573
Epoch 148/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1048 - accuracy: 0.8713 - val_loss: 0.7449 - val_accuracy: 0.7533
Epoch 149/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1093 - accuracy: 0.8659 - val_loss: 0.7212 - val_accuracy: 0.7577
Epoch 150/150
197/197 [==============================] - 2s 10ms/step - loss: 0.1025 - accuracy: 0.8741 - val_loss: 0.7471 - val_accuracy: 0.7483

In [27]:

plot_network_summaries(history, 'best_model_B1-11.hdf5', val_images, val_labels)

plot_network_summaries(history, 'best_model_B1-11.hdf5', val_images, val_labels)

Optimal point:  101 epochs
94/94 [==============================] - 0s 1ms/step

94/94 [==============================] - 0s 2ms/step - loss: 0.6543 - accuracy: 0.7640
Test accuracy: 0.7639999985694885

This model is the best preforming overall reaching it's optimal point at 101 epochs

---

Retraining on all training data¶

Training on the combined training and validation sets.

Combining Datasets¶

In [25]:

combined_training_images = np.concatenate([train_images, val_images], axis=0)
combined_training_labels = np.concatenate([train_labels, val_labels.flatten()], axis=0)

combined_training_images = np.concatenate([train_images, val_images], axis=0) combined_training_labels = np.concatenate([train_labels, val_labels.flatten()], axis=0)

Training Network¶

After combining the network is trained on the full dataset as combined, for the target mount of epochs.

In [26]:

from tensorflow.keras import models, layers

network = models.Sequential()
network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, )))
network.add(layers.Dense(64, activation='relu'))
network.add(layers.Dense(n_classes, activation='softmax'))

network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = network.fit(combined_training_images, combined_training_labels, epochs=101, batch_size=1024, class_weight=class_weight_dict)

from tensorflow.keras import models, layers network = models.Sequential() network.add(layers.Dense(1024, activation='relu', input_shape = (28*28, ))) network.add(layers.Dense(64, activation='relu')) network.add(layers.Dense(n_classes, activation='softmax')) network.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) history = network.fit(combined_training_images, combined_training_labels, epochs=101, batch_size=1024, class_weight=class_weight_dict)

Epoch 1/101
200/200 [==============================] - 3s 10ms/step - loss: 0.4547 - accuracy: 0.3179
Epoch 2/101
200/200 [==============================] - 2s 10ms/step - loss: 0.4069 - accuracy: 0.4273
Epoch 3/101
200/200 [==============================] - 2s 10ms/step - loss: 0.3790 - accuracy: 0.4833
Epoch 4/101
200/200 [==============================] - 2s 10ms/step - loss: 0.3587 - accuracy: 0.5232
Epoch 5/101
200/200 [==============================] - 2s 10ms/step - loss: 0.3451 - accuracy: 0.5480
Epoch 6/101
200/200 [==============================] - 2s 10ms/step - loss: 0.3305 - accuracy: 0.5724
Epoch 7/101
200/200 [==============================] - 2s 10ms/step - loss: 0.3204 - accuracy: 0.5919
Epoch 8/101
200/200 [==============================] - 2s 10ms/step - loss: 0.3087 - accuracy: 0.6104
Epoch 9/101
200/200 [==============================] - 2s 10ms/step - loss: 0.3013 - accuracy: 0.6228
Epoch 10/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2932 - accuracy: 0.6346
Epoch 11/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2887 - accuracy: 0.6431
Epoch 12/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2834 - accuracy: 0.6518
Epoch 13/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2785 - accuracy: 0.6582
Epoch 14/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2735 - accuracy: 0.6652
Epoch 15/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2699 - accuracy: 0.6706
Epoch 16/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2670 - accuracy: 0.6763
Epoch 17/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2620 - accuracy: 0.6821
Epoch 18/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2593 - accuracy: 0.6863
Epoch 19/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2564 - accuracy: 0.6902
Epoch 20/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2515 - accuracy: 0.6973
Epoch 21/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2502 - accuracy: 0.6993
Epoch 22/101
200/200 [==============================] - 2s 11ms/step - loss: 0.2461 - accuracy: 0.7048
Epoch 23/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2443 - accuracy: 0.7076
Epoch 24/101
200/200 [==============================] - 2s 11ms/step - loss: 0.2407 - accuracy: 0.7121
Epoch 25/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2386 - accuracy: 0.7138
Epoch 26/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2364 - accuracy: 0.7173
Epoch 27/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2349 - accuracy: 0.7190
Epoch 28/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2321 - accuracy: 0.7222
Epoch 29/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2283 - accuracy: 0.7275
Epoch 30/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2258 - accuracy: 0.7308
Epoch 31/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2231 - accuracy: 0.7333
Epoch 32/101
200/200 [==============================] - 2s 10ms/step - loss: 0.2228 - accuracy: 0.7353
Epoch 33/101
200/200 [==============================] - 2s 11ms/step - loss: 0.2202 - accuracy: 0.7373
Epoch 34/101
200/200 [==============================] - 2s 11ms/step - loss: 0.2182 - accuracy: 0.7406
Epoch 35/101
200/200 [==============================] - 2s 11ms/step - loss: 0.2167 - accuracy: 0.7423
Epoch 36/101
200/200 [==============================] - 2s 11ms/step - loss: 0.2150 - accuracy: 0.7443
Epoch 37/101
200/200 [==============================] - 2s 11ms/step - loss: 0.2117 - accuracy: 0.7484
Epoch 38/101
200/200 [==============================] - 2s 11ms/step - loss: 0.2109 - accuracy: 0.7485
Epoch 39/101
200/200 [==============================] - 2s 11ms/step - loss: 0.2075 - accuracy: 0.7528
Epoch 40/101
200/200 [==============================] - 2s 11ms/step - loss: 0.2066 - accuracy: 0.7545
Epoch 41/101
200/200 [==============================] - 2s 11ms/step - loss: 0.2052 - accuracy: 0.7557
Epoch 42/101
200/200 [==============================] - 2s 11ms/step - loss: 0.2023 - accuracy: 0.7594
Epoch 43/101
200/200 [==============================] - 2s 11ms/step - loss: 0.2031 - accuracy: 0.7590
Epoch 44/101
200/200 [==============================] - 2s 11ms/step - loss: 0.2004 - accuracy: 0.7612
Epoch 45/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1987 - accuracy: 0.7632
Epoch 46/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1983 - accuracy: 0.7635
Epoch 47/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1942 - accuracy: 0.7691
Epoch 48/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1948 - accuracy: 0.7680
Epoch 49/101
200/200 [==============================] - 2s 12ms/step - loss: 0.1930 - accuracy: 0.7697
Epoch 50/101
200/200 [==============================] - 2s 12ms/step - loss: 0.1906 - accuracy: 0.7726
Epoch 51/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1874 - accuracy: 0.7763
Epoch 52/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1875 - accuracy: 0.7773
Epoch 53/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1870 - accuracy: 0.7769
Epoch 54/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1858 - accuracy: 0.7788
Epoch 55/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1845 - accuracy: 0.7810
Epoch 56/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1826 - accuracy: 0.7821
Epoch 57/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1799 - accuracy: 0.7850
Epoch 58/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1808 - accuracy: 0.7835
Epoch 59/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1774 - accuracy: 0.7884
Epoch 60/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1767 - accuracy: 0.7881
Epoch 61/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1761 - accuracy: 0.7893
Epoch 62/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1750 - accuracy: 0.7905
Epoch 63/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1744 - accuracy: 0.7902
Epoch 64/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1721 - accuracy: 0.7952
Epoch 65/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1692 - accuracy: 0.7980
Epoch 66/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1698 - accuracy: 0.7958
Epoch 67/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1687 - accuracy: 0.7982
Epoch 68/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1681 - accuracy: 0.7989
Epoch 69/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1650 - accuracy: 0.8015
Epoch 70/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1637 - accuracy: 0.8024
Epoch 71/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1647 - accuracy: 0.8018
Epoch 72/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1626 - accuracy: 0.8044
Epoch 73/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1610 - accuracy: 0.8065
Epoch 74/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1600 - accuracy: 0.8074
Epoch 75/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1604 - accuracy: 0.8075
Epoch 76/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1589 - accuracy: 0.8100
Epoch 77/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1592 - accuracy: 0.8085
Epoch 78/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1562 - accuracy: 0.8113
Epoch 79/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1551 - accuracy: 0.8130
Epoch 80/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1556 - accuracy: 0.8119
Epoch 81/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1543 - accuracy: 0.8140
Epoch 82/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1533 - accuracy: 0.8152
Epoch 83/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1528 - accuracy: 0.8147
Epoch 84/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1514 - accuracy: 0.8158
Epoch 85/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1498 - accuracy: 0.8190
Epoch 86/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1481 - accuracy: 0.8217
Epoch 87/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1484 - accuracy: 0.8203
Epoch 88/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1453 - accuracy: 0.8232
Epoch 89/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1448 - accuracy: 0.8245
Epoch 90/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1464 - accuracy: 0.8229
Epoch 91/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1453 - accuracy: 0.8229
Epoch 92/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1412 - accuracy: 0.8286
Epoch 93/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1427 - accuracy: 0.8266
Epoch 94/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1442 - accuracy: 0.8256
Epoch 95/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1417 - accuracy: 0.8272
Epoch 96/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1381 - accuracy: 0.8317
Epoch 97/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1388 - accuracy: 0.8304
Epoch 98/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1385 - accuracy: 0.8312
Epoch 99/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1372 - accuracy: 0.8335
Epoch 100/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1379 - accuracy: 0.8315
Epoch 101/101
200/200 [==============================] - 2s 11ms/step - loss: 0.1349 - accuracy: 0.8358

Evaluate Final Model¶

In [38]:

from sklearn.metrics import confusion_matrix
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

probabilities = network.predict(test_images.reshape((len(test_images), 28*28)))
predicted_classes = np.argmax(probabilities, axis=1)

# Generate the confusion matrix
labels = [info['label'][str(label)] for label in range(n_classes)]
cm = confusion_matrix(test_labels, predicted_classes.flatten())

# If you want to have a DataFrame representation
cm_df = pd.DataFrame(cm, index=labels, columns=labels)

plt.figure(figsize=(10,7))
sns.heatmap(cm_df, annot=True, fmt='g')
plt.title('Confusion Matrix')
plt.ylabel('Actual Labels')
plt.xlabel('Predicted Labels')
plt.xticks(rotation=45)
plt.show()
    
test_loss, test_acc = network.evaluate(test_images.reshape((len(test_images), 28*28)), test_labels)
print('Test accuracy:', test_acc)

from sklearn.metrics import confusion_matrix import pandas as pd import seaborn as sns import matplotlib.pyplot as plt probabilities = network.predict(test_images.reshape((len(test_images), 28*28))) predicted_classes = np.argmax(probabilities, axis=1) # Generate the confusion matrix labels = [info['label'][str(label)] for label in range(n_classes)] cm = confusion_matrix(test_labels, predicted_classes.flatten()) # If you want to have a DataFrame representation cm_df = pd.DataFrame(cm, index=labels, columns=labels) plt.figure(figsize=(10,7)) sns.heatmap(cm_df, annot=True, fmt='g') plt.title('Confusion Matrix') plt.ylabel('Actual Labels') plt.xlabel('Predicted Labels') plt.xticks(rotation=45) plt.show() test_loss, test_acc = network.evaluate(test_images.reshape((len(test_images), 28*28)), test_labels) print('Test accuracy:', test_acc)

 1/32 [..............................] - ETA: 1s

32/32 [==============================] - 0s 2ms/step

32/32 [==============================] - 0s 2ms/step - loss: 1.2294 - accuracy: 0.6100
Test accuracy: 0.6100000143051147

It unfortunately seems that the accuracy of the resulting network is much lower that the previous networks, which may indicate that the modeling of the network and hyperparameters may have be "overfit" to the validation data.