neural_nets_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Neural Networks\n",
    "\n",
    "## TO DO: Almost all the figues and schematics will be replaced or improved slowly\n",
    "\n",
    "<center>\n",
    "<figure>\n",
    "<img src=\"./images/neuralnets/neural_net_ex.svg\" width=\"700\"/>\n",
    "<figcaption>A 3 layer Neural Network (By convention the input layer is not counted).</figcaption>\n",
    "</figure>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## History of Neural networks\n",
    "\n",
    "**TODO: Make it more complete and format properly**\n",
    "\n",
    "1943 - Threshold Logic\n",
    "\n",
    "1940s - Hebbian Learning\n",
    "\n",
    "1958 - Perceptron\n",
    "\n",
    "1975 - Backpropagation\n",
    "\n",
    "1980s - Neocognitron\n",
    "\n",
    "1982: Hopfield Network\n",
    "\n",
    "1986: Convolutional Neural Networks\n",
    "\n",
    "1997: Long-short term memory (LSTM) model\n",
    "\n",
    "2014: Gated Recurrent Units, Generative Adversarial Networks(Check)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why the boom now?\n",
    "* Data\n",
    "* Data\n",
    "* Data\n",
    "* Availability of GPUs\n",
    "* Algorithmic developments which allow for efficient training and training for deeper networks\n",
    "* Much easier access than a decade ago"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building blocks\n",
    "### Perceptron\n",
    "\n",
    "The smallest unit of a neural network is a **perceptron** like node.\n",
    "\n",
    "**What is a Perceptron?**\n",
    "\n",
    "It is a simple function which can have multiple inputs and has a single output.\n",
    "\n",
    "<center>\n",
    "<figure>\n",
    "<img src=\"./images/neuralnets/perceptron_ex.svg\" width=\"400\"/>\n",
    "<figcaption>A simple perceptron with 3 inputs and 1 output.</figcaption>\n",
    "</figure>\n",
    "</center>\n",
    "\n",
    "\n",
    "It works as follows: \n",
    "\n",
    "Step 1: A **weighted sum** of the inputs is calculated\n",
    "\n",
    "\\begin{equation*}\n",
    "weighted\\_sum = \\sum_{k=1}^{num\\_inputs} w_{i} x_{i}\n",
    "\\end{equation*}\n",
    "\n",
    "Step 2: A **step** activation function is applied\n",
    "\n",
    "$$\n",
    "f(weighted\\_sum) = \\left\\{\n",
    "        \\begin{array}{ll}\n",
    "            0 & \\quad weighted\\_sum < threshold \\\\\n",
    "            1 & \\quad weighted\\_sum \\geq threshold\n",
    "        \\end{array}\n",
    "    \\right.\n",
    "$$\n",
    "\n",
    "You can see that this is also a linear classifier as we introduced in script 02."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config IPCompleter.greedy=True\n",
    "import matplotlib as mpl\n",
    "mpl.rcParams['lines.linewidth'] = 3\n",
    "#mpl.rcParams['font.size'] = 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "def perceptron(X, w, threshold=1):\n",
    "    # This function computes sum(w_i*x_i) and\n",
    "    # applies a perceptron activation\n",
    "    linear_sum = np.dot(X, w)\n",
    "    output = 0\n",
    "    if linear_sum >= threshold:\n",
    "        output = 1\n",
    "    return output\n",
    "\n",
    "\n",
    "X = [1, 0]\n",
    "w = [1, 1]\n",
    "perceptron(X, w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Boolean AND\n",
    "\n",
    "| x$_1$ | x$_2$ | output |\n",
    "| --- | --- | --- |\n",
    "| 0 | 0 | 0 |\n",
    "| 1 | 0 | 0 |\n",
    "| 0 | 1 | 0 |\n",
    "| 1 | 1 | 1 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Perceptron output for x1, x2 =  [0, 0]  is  0\n",
      "Perceptron output for x1, x2 =  [1, 0]  is  0\n",
      "Perceptron output for x1, x2 =  [0, 1]  is  0\n",
      "Perceptron output for x1, x2 =  [1, 1]  is  1\n"
     ]
    }
   ],
   "source": [
    "# Calculating Boolean AND using a perceptron\n",
    "threshold = 1.5\n",
    "w = [1, 1]\n",
    "X = [[0, 0], [1, 0], [0, 1], [1, 1]]\n",
    "for i in X:\n",
    "    print(\"Perceptron output for x1, x2 = \", i,\n",
    "          \" is \", perceptron(i, w, threshold))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this simple case we can rewrite our equation to $x_2 = ...... $ which describes a line in 2D:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,