A very basic example -- approximate functions with Deep Neural Networks and Tensorflow and Keras

• This notebook contains two gentle examples how to perform supervised (regression and classification) machine learning tasks with fully connected deep neural networks.

  • we train a deep NN to learn an analytical, 2-dimensional function and perform regression.
  • we look at a standart data set (Zalando fashion MNIST dataset) to perform classification. This data-set is already avaialble within the Keras API.

• From the technical side

  • we look at different, pre-implemented cost functions (e.g., MSE, MAE, Cross-entropy loss).
  • we look at initialization of the network weights.
  • early stopping.
  • we look at dropout.
  • we look at batch normalization.

• A comprehensive set of examples can be found here and here:

  • Guide to the Functional API.
  • Guide to making new Layers & Models via subclassing.

The basic setup

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
print(tf.__version__)


import numpy as np
import math
from random import uniform
import matplotlib.pyplot as plt

2.4.0

1. A simple regression example

As a first example, we want to approximate 2-d analytical functions on a d-dimensional unit cube [-1,1]^d.

Generate training data

dim_x = 2            #input dimension of the analytical function
dim_y = 1            #output dimension of the analytical function
no_samples = 10000  #number of observations
no_test  = 1000      #test data
filename = 'my_fun.weights.h5'  #where to store data

#Generate training data
#x coord
aPnts = np.empty([no_samples, dim_x])  
for iI in range(no_samples):
    for iJ in range(dim_x):
        aPnts[iI][iJ] = uniform(-1.0, 1.0)
data = aPnts #np.random.random((no_samples, dim_x))

#y value
aTres = np.empty([no_samples,])
for iI in range(no_samples):
    aTres[iI] = math.cos(0.5 * math.pi * aPnts[iI][0]) * math.cos(0.5 * math.pi * aPnts[iI][1])
labels = aTres #np.random.random((no_samples,dim_y ))


#Test data 
aPnts2 = np.empty([no_test, dim_x])  
for iI in range(no_test):
    for iJ in range(dim_x):
        aPnts2[iI][iJ] = uniform(-1.0, 1.0)
data2 = aPnts2 #np.random.random((no_samples, dim_x))

## y value
aTres2 = np.empty([no_test,])
for iI in range(no_test):
    aTres2[iI] = math.cos(0.5 * math.pi * aPnts2[iI][0]) * math.cos(0.5 * math.pi * aPnts2[iI][1])
labels2 = aTres2 #np.random.random((no_samples,dim_y ))

Fully-connected deep neural network -- creating a Sequential model

Tensorflow with the Keras API defines a Sequential model to be the appropriate choice for a plain stack of layers, where each layer has exactly one input tensor and one output tensor.

Schematically, the following Sequential model:

#Define Sequential model with 3 layers
model = tf.keras.Sequential([
#Adds a densely-connected layer with 64 units to the model:
    layers.Dense(64, activation='relu', input_shape=(dim_x,), name='layer1'),
    # Add another:
    layers.Dense(64, activation='relu', name='layer2'),
    # Add an output layer with dim_y output units:
    layers.Dense(dim_y)],name='my_first_model')

• Also note that the Sequential constructor accepts a name argument, just like any layer or model in Keras. This is useful to annotate TensorBoard graphs (later today) with semantically meaningful names.

• Once a model is "built", you can call its summary() method to display its contents:

model.summary()

Model: "my_first_model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
layer1 (Dense)               (None, 64)                192       
_________________________________________________________________
layer2 (Dense)               (None, 64)                4160      
_________________________________________________________________
dense (Dense)                (None, 1)                 65        
=================================================================
Total params: 4,417
Trainable params: 4,417
Non-trainable params: 0
_________________________________________________________________

Its layers are accessible via the layers attribute:

model.layers

[<tensorflow.python.keras.layers.core.Dense at 0x7f6ca5907d00>,
 <tensorflow.python.keras.layers.core.Dense at 0x7f6ca5907ac0>,
 <tensorflow.python.keras.layers.core.Dense at 0x7f6ca5895670>]

What to do once you have a model

Once your model architecture is ready, you will want to:

• Train your model, evaluate it, and run inference. See guide to training & evaluation with the built-in loops.
• Save your model to disk and restore it. See guide to serialization & saving.
• Speed up model training by leveraging multiple GPUs. See guide to multi-GPU and distributed training.

Compile the model

Before the model is ready for training, it needs a few more settings. These are added during the model's compile step:

• Loss function —This measures how accurate the model is during training. You want to minimize this function to "steer" the model in the right direction.
• Optimizer —This is how the model is updated based on the data it sees and its loss function.
• Metrics —Used to monitor the training and testing steps.
• The first example below uses MSE and MAE.
• The second example below for instance uses accuracy, the fraction of the images that are correctly classified.

#Configure a model for mean-squared error regression.
model.compile(optimizer=tf.keras.optimizers.Adam(0.01),
              loss='mse',       # mean squared error
              metrics=['mae'])  # mean absolute error



#fit model
model.fit(data, labels, epochs=3, batch_size=32)


#model.weights
#print("Number of weights after calling the model:", len(model.weights))  

Epoch 1/3
313/313 [==============================] - 2s 3ms/step - loss: 0.0318 - mae: 0.0928
Epoch 2/3
313/313 [==============================] - 1s 3ms/step - loss: 1.1309e-04 - mae: 0.0081
Epoch 3/3
313/313 [==============================] - 1s 3ms/step - loss: 3.4926e-04 - mae: 0.0136

<tensorflow.python.keras.callbacks.History at 0x7f6c9c73e730>

• Alternatively, you can also create a Sequential model incrementally via the add() method:

  model = keras.Sequential()

  model.add(layers.Dense(2, activation="relu"))

  model.add(layers.Dense(3, activation="relu"))

  model.add(layers.Dense(4))

• Also note that the Sequential constructor accepts a name argument, just like any layer or model in Keras. This is useful to annotate TensorBoard graphs (later today) with semantically meaningful names.

Test Accuracy

test_loss, test_acc = model.evaluate(data2,  labels2, verbose=2)
print('Test accuracy:', test_acc)

32/32 - 0s - loss: 1.6588e-04 - mae: 0.0099
Test accuracy: 0.00992078147828579

Predict individual values

predictions = model.predict(data2)
x = data2[0]

x1 = x[0,]
y1 = x[1,]
print('point to test', x[0,], '  ',x[1,])
#x.shape
#print(x.shape, data2.shape)

# Analytical solution:
res = math.cos(0.5 * math.pi * x1) * math.cos(0.5 * math.pi * y1)
print('NN prediction: ' , predictions[0], ', Analytical solution', res, ', Difference' ,abs(predictions[0]-res))

point to test 0.4519528042375167    0.001170324504295106
NN prediction:  [0.75682425] , Analytical solution 0.7584089553294859 , Difference [0.00158471]

Save weights to a TensorFlow Checkpoint file -> Restart etc.

# Save weights to a TensorFlow Checkpoint file
print("store the model")
model.save_weights(filename)

# this requires a model with the same architecture.
print("reload the model")
model.load_weights(filename)

predictions = model.predict(data2)
res = math.cos(0.5 * math.pi * x1) * math.cos(0.5 * math.pi * y1)
print('NN prediction: ' , predictions[0], ', Analytical solution', res, ', Difference' ,abs(predictions[0]-res))

store the model
reload the model
NN prediction:  [0.75682425] , Analytical solution 0.7584089553294859 , Difference [0.00158471]

Network weight initialization and early stopping

• Different pre-implemented initializers

• Early stopping

#Define Sequential model 

#initialize weigths
initializer = tf.keras.initializers.RandomNormal(mean=0., stddev=1.)

#create the model
model2 = tf.keras.Sequential([
#Adds a densely-connected layer with 64 units to the model:
    layers.Dense(64, activation='relu', input_shape=(dim_x,),kernel_initializer=initializer, name='layer1'),
    # Add another:
    layers.Dense(64, activation='relu', kernel_initializer=initializer, name='layer2'),
    # Add an output layer with dim_y output units:
    layers.Dense(dim_y)],name='my_first_model')

• Assuming the goal of a training is to minimize the loss. With this, the metric to be monitored would be 'loss', and mode would be 'min'. A model.fit() training loop will check at end of every epoch whether the loss is no longer decreasing, considering the min_delta and patience if applicable. Once it's found no longer decreasing, model.stop_training is marked True and the training terminates.

• The quantity to be monitored needs to be available in logs dict. To make it so, pass the loss or metrics at model.compile().

• Note: below, we use patience: one typically defines a patience, i.e., the number of epochs to wait before early stop if no progress on the validation set. In practice, the patience is often set somewhere between 10 and 20, but it really depends on your dataset and network.

# This callback will stop the training when there is no improvement in

# the validation loss for three consecutive epochs.
callback = tf.keras.callbacks.EarlyStopping(monitor='loss', patience=3)


#Configure a model for mean-squared error regression.
model2.compile(optimizer=tf.keras.optimizers.Adam(0.01),
              loss='mse',       # mean squared error
              metrics=['mae'])  # mean absolute error
#fit model
history = model2.fit(data, labels, epochs=3, batch_size=32,callbacks=[callback], verbose=0)

#early stopping
len(history.history['loss'])  # Only 4 epochs are run.

3

• This callback will stop the training when there is no improvement in the quantity to be monitored

• the validation loss for delta -- which is the inimum change in the monitored quantity to qualify as an improvement, i.e. an absolute change of less than min_delta, will count as no improvement.

callback = tf.keras.callbacks.EarlyStopping(monitor='loss', min_delta=0.01)


#Configure a model for mean-squared error regression.
model2.compile(optimizer=tf.keras.optimizers.Adam(0.01),
              loss='mse',       # mean squared error
              metrics=['mae'])  # mean absolute error
#fit model
history = model2.fit(data, labels, epochs=3, batch_size=32,callbacks=[callback], verbose=1)

len(history.history['loss'])  # Only 2 epochs are run.

Epoch 1/3
313/313 [==============================] - 1s 2ms/step - loss: 0.0413 - mae: 0.1132
Epoch 2/3
313/313 [==============================] - 1s 2ms/step - loss: 0.0029 - mae: 0.0399
Epoch 3/3
313/313 [==============================] - 1s 2ms/step - loss: 0.0034 - mae: 0.0417

3

Dropout layers

• Dropout

• Dropout can be applied to input neurons called the visible layer.

• In the example below we add a new Dropout layer between the input (or visible layer) and the first hidden layer. The dropout rate is set to 20%.

#Define Sequential model 
model3 = tf.keras.Sequential([
#Adds a densely-connected layer with 64 units to the model:
    layers.Dense(64, activation='relu', input_shape=(dim_x,), name='layer1'),
    # Add another:
    layers.Dense(64, activation='relu', name='layer2'),
    #Dropout
    layers.Dropout(0.2),
    # Add another:
    layers.Dense(64, activation='relu', name='layer3'),
    #Dropout
    layers.Dropout(0.2),    
    # Add an output layer with dim_y output units:
    layers.Dense(dim_y)],name='my_first_model')


#Configure a model for mean-squared error regression.
model3.compile(optimizer=tf.keras.optimizers.Adam(0.01),
              loss='mse',       # mean squared error
              metrics=['mae'])  # mean absolute error
#fit model
model3.fit(data, labels, epochs=3, batch_size=32, verbose=1)         

Epoch 1/3
313/313 [==============================] - 1s 3ms/step - loss: 0.0362 - mae: 0.1221
Epoch 2/3
313/313 [==============================] - 1s 3ms/step - loss: 0.0036 - mae: 0.0466
Epoch 3/3
313/313 [==============================] - 1s 3ms/step - loss: 0.0028 - mae: 0.0412

<tensorflow.python.keras.callbacks.History at 0x7f6c78613dc0>

#Inspect the network 
model3.summary()

Model: "my_first_model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
layer1 (Dense)               (None, 64)                192       
_________________________________________________________________
layer2 (Dense)               (None, 64)                4160      
_________________________________________________________________
dropout (Dropout)            (None, 64)                0         
_________________________________________________________________
layer3 (Dense)               (None, 64)                4160      
_________________________________________________________________
dropout_1 (Dropout)          (None, 64)                0         
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 65        
=================================================================
Total params: 8,577
Trainable params: 8,577
Non-trainable params: 0
_________________________________________________________________

Tips for using Dropout

• The original paper on Dropout provides experimental results on a suite of standard machine learning problems. As a result they provide a number of useful heuristics to consider when using dropout in practice.

• Generally, use a small dropout value of 20%-50% of neurons with 20% providing a good starting point. A probability too low has minimal effect and a value too high results in under-learning by the network.

• Use a larger network. You are likely to get better performance when dropout is used on a larger network, giving the model more of an opportunity to learn independent representations.

• Use dropout on incoming (visible) as well as hidden units. Application of dropout at each layer of the network has shown good results.

A useful functionality

• Note that there's also a corresponding pop() method to remove layers: a Sequential model behaves very much like a list of layers.

model3.pop()
print(len(model.layers))  # 2

3

Batch normalization

• Batch normalization

• Normalize the activations of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1.

#Define Sequential model with 3 layers
model4 = tf.keras.Sequential([
#Adds a densely-connected layer with 64 units to the model:
    layers.Dense(64, activation='relu', input_shape=(dim_x,), name='layer1'),
    # Add another:
    layers.Dense(64, activation='relu', name='layer2'),
    #Batch Normalization
    layers.BatchNormalization(),
    # Add another:
    layers.Dense(64, activation='relu', name='layer3'),
    #Batch Normalization
    layers.BatchNormalization(),    
    # Add an output layer with dim_y output units:
    layers.Dense(dim_y)],name='my_first_model')

#Configure a model for mean-squared error regression.
model4.compile(optimizer=tf.keras.optimizers.Adam(0.01),
              loss='mse',       # mean squared error
              metrics=['mae'])  # mean absolute error
#fit model
model4.fit(data, labels, epochs=3, batch_size=32, verbose=1)  

Epoch 1/3
313/313 [==============================] - 2s 4ms/step - loss: 0.1576 - mae: 0.1904
Epoch 2/3
313/313 [==============================] - 1s 4ms/step - loss: 0.0090 - mae: 0.0744
Epoch 3/3
313/313 [==============================] - 1s 4ms/step - loss: 0.0076 - mae: 0.0691

<tensorflow.python.keras.callbacks.History at 0x7f6c784cc130>

2. Classification example

Getting the data set

fashion_mnist = keras.datasets.fashion_mnist

(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz
32768/29515 [=================================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz
26427392/26421880 [==============================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz
8192/5148 [===============================================] - 0s 0us/step
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz
4423680/4422102 [==============================] - 0s 0us/step

Loading the dataset returns four NumPy arrays:

• The train_images and train_labels arrays are the training set—the data the model uses to learn.
• The model is tested against the test set, the test_images, and test_labels arrays.
• The images are 28x28 NumPy arrays, with pixel values ranging from 0 to 255. The labels are an array of integers, ranging from 0 to 9. These correspond to the class of clothing the image represents:

class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

Explore the data

Let's explore the format of the dataset before training the model. The following shows there are 60,000 images in the training set, with each image represented as 28 x 28 pixels:

#shape
train_images.shape

(60000, 28, 28)

#lables
len(train_labels)
train_labels

array([9, 0, 0, ..., 3, 0, 5], dtype=uint8)

# There are 10,000 images in the test set, and each image is represented as 28 x 28 pixels: 
test_images.shape

(10000, 28, 28)

• The data must be preprocessed before training the network.

• If you inspect the first image in the training set, you will see that the pixel values fall in the range of 0 to 255.

• We need to scale these values to a range of 0 to 1 before feeding them to the neural network model. To do so, divide the values by 255. It's important that the training set and the testing set be preprocessed in the same way.

plt.figure()
plt.imshow(train_images[0])
plt.colorbar()
plt.grid(False)
plt.show()

train_images = train_images / 255.0

test_images = test_images / 255.0

Now, we visually verify that the data is in the correct format and that you're ready to build and train the network, let's display the first 25 images from the training set and display the class name below each image.

plt.figure(figsize=(10,10))
for i in range(16):
    plt.subplot(4,4,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(train_images[i], cmap=plt.cm.binary)
    plt.xlabel(class_names[train_labels[i]])
plt.show()

Fully-connected deep neural network -- creating a Sequential model

#set-up the model
model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10)
])

#compile the model
model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

#train the model
model.fit(train_images, train_labels, epochs=10)

Epoch 1/10
1875/1875 [==============================] - 7s 4ms/step - loss: 0.6342 - accuracy: 0.7768
Epoch 2/10
1875/1875 [==============================] - 7s 4ms/step - loss: 0.3865 - accuracy: 0.8591
Epoch 3/10
1875/1875 [==============================] - 7s 4ms/step - loss: 0.3446 - accuracy: 0.8753
Epoch 4/10
1875/1875 [==============================] - 7s 4ms/step - loss: 0.3133 - accuracy: 0.8850
Epoch 5/10
1875/1875 [==============================] - 7s 4ms/step - loss: 0.2999 - accuracy: 0.8904
Epoch 6/10
1875/1875 [==============================] - 7s 4ms/step - loss: 0.2870 - accuracy: 0.8936
Epoch 7/10
1875/1875 [==============================] - 7s 4ms/step - loss: 0.2683 - accuracy: 0.9035
Epoch 8/10
1875/1875 [==============================] - 7s 4ms/step - loss: 0.2542 - accuracy: 0.9066
Epoch 9/10
1875/1875 [==============================] - 7s 4ms/step - loss: 0.2457 - accuracy: 0.9089
Epoch 10/10
1875/1875 [==============================] - 7s 4ms/step - loss: 0.2385 - accuracy: 0.9106

<tensorflow.python.keras.callbacks.History at 0x7f6c9c074a30>

#Evaluate the accuracy
test_loss, test_acc = model.evaluate(test_images,  test_labels, verbose=2)

print('\nTest accuracy:', test_acc)

313/313 - 0s - loss: 0.3466 - accuracy: 0.8746

Test accuracy: 0.8745999932289124

Make predictions

• With the model trained, you can use it to make predictions about some images.

• The model's linear outputs, logits.

• Attach a softmax layer to convert the logits to probabilities, which are easier to interpret.

probability_model = tf.keras.Sequential([model, 
                                         tf.keras.layers.Softmax()])

predictions = probability_model.predict(test_images)

#Lets have a look at the first prediction
predictions[0]

array([4.4164270e-07, 1.3861331e-12, 1.6880713e-09, 7.4217255e-10,
       1.5809281e-09, 1.3972708e-03, 1.4475570e-07, 4.6838480e-03,
       2.5023925e-07, 9.9391806e-01], dtype=float32)

A prediction is an array of 10 numbers. They represent the model's "confidence" that the image corresponds to each of the 10 different articles of clothing. You can see which label has the highest confidence value:

np.argmax(predictions[0])

9

• The model is most confident that this image is an ankle boot, or class_names[9].

• Examining the test label shows that this classification is correct:

Verify predictions

• With the model trained, you can use it to make predictions about some images.

• Let's look at the 0th image, predictions, and prediction array. Correct prediction labels are blue and incorrect prediction labels are red.

• The number gives the percentage (out of 100) for the predicted label.

def plot_image(i, predictions_array, true_label, img):
  predictions_array, true_label, img = predictions_array, true_label[i], img[i]
  plt.grid(False)
  plt.xticks([])
  plt.yticks([])

  plt.imshow(img, cmap=plt.cm.binary)

  predicted_label = np.argmax(predictions_array)
  if predicted_label == true_label:
    color = 'blue'
  else:
    color = 'red'

  plt.xlabel("{} {:2.0f}% ({})".format(class_names[predicted_label],
                                100*np.max(predictions_array),
                                class_names[true_label]),
                                color=color)

def plot_value_array(i, predictions_array, true_label):
  predictions_array, true_label = predictions_array, true_label[i]
  plt.grid(False)
  plt.xticks(range(10))
  plt.yticks([])
  thisplot = plt.bar(range(10), predictions_array, color="#777777")
  plt.ylim([0, 1])
  predicted_label = np.argmax(predictions_array)

  thisplot[predicted_label].set_color('red')
  thisplot[true_label].set_color('blue')

i = 10
plt.figure(figsize=(6,3))
plt.subplot(1,2,1)
plot_image(i, predictions[i], test_labels, test_images)
plt.subplot(1,2,2)
plot_value_array(i, predictions[i],  test_labels)
plt.show()

# Take an image from the test dataset.
img = test_images[1]
print(img.shape)

(28, 28)

# Add the image to a batch where it's the only member.
img = (np.expand_dims(img,0))

print(img.shape)

(1, 28, 28)

# predict the correct label for this image:
predictions_single = probability_model.predict(img)
print(predictions_single)

[[2.2718454e-05 5.0456752e-13 9.9943191e-01 1.7911441e-13 1.2616580e-04
  7.1405181e-13 4.1917901e-04 1.6020190e-14 8.8627272e-11 2.4638422e-15]]

plot_value_array(1, predictions_single[0], test_labels)
_ = plt.xticks(range(10), class_names, rotation=45)

keras.Model.predict returns a list of lists—one list for each image in the batch of data. Grab the predictions for our (only) image in the batch:

np.argmax(predictions_single[0])

2