08_a-neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# IGNORE THIS CELL WHICH CUSTOMIZES LAYOUT AND STYLING OF THE NOTEBOOK !\n",
    "from numpy.random import seed\n",
    "seed(42)\n",
    "import tensorflow as tf\n",
    "tf.random.set_seed(36)\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib as mpl\n",
    "import seaborn as sns\n",
    "sns.set(style=\"darkgrid\")\n",
    "mpl.rcParams['lines.linewidth'] = 3\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "%config IPCompleter.greedy=True\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore', category=FutureWarning)\n",
    "from IPython.core.display import HTML; HTML(open(\"custom.html\", \"r\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 8: Introduction to Neural Networks\n",
    "\n",
    "\n",
    "\n",
    "<img src=\"./images/3042en.jpg\" title=\"made at imgflip.com\" width=35%/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## History of Neural networks\n",
    "\n",
    "\n",
    "1943 - Threshold Logic\n",
    "\n",
    "1940s - Hebbian Learning\n",
    "\n",
    "1958 - Perceptron\n",
    "\n",
    "1980s - Neocognitron\n",
    "\n",
    "1982 - Hopfield Network\n",
    "\n",
    "1989 - Convolutional neural network (CNN) kernels trained via backpropagation\n",
    "\n",
    "1997 - Long-short term memory (LSTM) model\n",
    "\n",
    "1998 - LeNet-5\n",
    "\n",
    "2014 - Gated Recurrent Units (GRU), Generative Adversarial Networks (GAN)\n",
    "\n",
    "2015 - ResNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why the boom now?\n",
    "* Data\n",
    "* Data\n",
    "* Data\n",
    "* Availability of GPUs\n",
    "* Algorithmic developments which allow for efficient training and making networks networks\n",
    "* Development of high-level libraries/APIs have made the field much more accessible than it was a decade ago"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feed-Forward neural network\n",
    "<center>\n",
    "<figure>\n",
    "<img src=\"./images/neuralnets/neural_net_ex.svg\" width=\"700\"/>\n",
    "<figcaption>A 3 layer densely connected Neural Network (By convention the input layer is not counted).</figcaption>\n",
    "</figure>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building blocks\n",
    "### Perceptron\n",
    "\n",
    "The smallest unit of a neural network is a **perceptron** like node.\n",
    "\n",
    "**What is a Perceptron?**\n",
    "\n",
    "It is a simple function which can have multiple inputs and has a single output.\n",
    "\n",
    "<center>\n",
    "<figure>\n",
    "<img src=\"./images/neuralnets/perceptron_ex.svg\" width=\"400\"/>\n",
    "<figcaption>A simple perceptron with 3 inputs and 1 output.</figcaption>\n",
    "</figure>\n",
    "</center>\n",
    "\n",
    "\n",
    "It works as follows: \n",
    "\n",
    "Step 1: A **weighted sum** of the inputs is calculated\n",
    "\n",
    "\\begin{equation*}\n",
    "weighted\\_sum = w_{1} x_{1} + w_{2} x_{2} + w_{3} x_{3} + ...\n",
    "\\end{equation*}\n",
    "\n",
    "Step 2: A **step** activation function is applied\n",
    "\n",
    "$$\n",
    "f = \\left\\{\n",
    "        \\begin{array}{ll}\n",
    "            0 & \\quad weighted\\_sum < threshold \\\\\n",
    "            1 & \\quad weighted\\_sum \\geq threshold\n",
    "        \\end{array}\n",
    "    \\right.\n",
    "$$\n",
    "\n",
    "You can see that this is also a linear classifier as the ones we introduced in script 02."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plotting the step function\n",
    "x = np.arange(-2,2.1,0.01)\n",
    "y = np.zeros(len(x))\n",
    "threshold = 0.\n",
    "y[x>threshold] = 1.\n",
    "step_plot = sns.lineplot(x, y).set_title('Step function') ;\n",
    "plt.xlabel('weighted_sum') ;\n",
    "plt.ylabel('f(weighted_sum)') ;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def perceptron(X, w, threshold=1):\n",
    "    # This function computes sum(w_i*x_i) and\n",
    "    # applies a perceptron activation\n",
    "    linear_sum = np.dot(np.asarray(X).T, w)\n",
    "    output = np.zeros(len(linear_sum), dtype=np.int8)\n",
    "    output[linear_sum >= threshold] = 1\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boolean AND\n",
    "\n",
    "| x$_1$ | x$_2$ | output |\n",
    "| --- | --- | --- |\n",
    "| 0 | 0 | 0 |\n",
    "| 1 | 0 | 0 |\n",
    "| 0 | 1 | 0 |\n",
    "| 1 | 1 | 1 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculating Boolean AND using a perceptron\n",
    "threshold = 1.5\n",
    "# (w1, w2)\n",
    "w = [1, 1]\n",