08_neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    \n",
       "    @import url('http://fonts.googleapis.com/css?family=Source+Code+Pro');\n",
       "    \n",
       "    @import url('http://fonts.googleapis.com/css?family=Kameron');\n",
       "    @import url('http://fonts.googleapis.com/css?family=Crimson+Text');\n",
       "    \n",
       "    @import url('http://fonts.googleapis.com/css?family=Lato');\n",
       "    @import url('http://fonts.googleapis.com/css?family=Source+Sans+Pro');\n",
       "    \n",
       "    @import url('http://fonts.googleapis.com/css?family=Lora'); \n",
       "\n",
       "    \n",
       "    body {\n",
       "        font-family: 'Lora', Consolas, sans-serif;\n",
       "       \n",
       "        -webkit-print-color-adjust: exact important !;\n",
       "        \n",
       "      \n",
       "       \n",
       "    }\n",
       "    \n",
       "    .alert-block {\n",
       "        width: 95%;\n",
       "        margin: auto;\n",
       "    }\n",
       "    \n",
       "    .rendered_html code\n",
       "    {\n",
       "        color: black;\n",
       "        background: #eaf0ff;\n",
       "        background: #f5f5f5; \n",
       "        padding: 1pt;\n",
       "        font-family:  'Source Code Pro', Consolas, monocco, monospace;\n",
       "    }\n",
       "    \n",
       "    p {\n",
       "      line-height: 140%;\n",
       "    }\n",
       "    \n",
       "    strong code {\n",
       "        background: red;\n",
       "    }\n",
       "    \n",
       "    .rendered_html strong code\n",
       "    {\n",
       "        background: #f5f5f5;\n",
       "    }\n",
       "    \n",
       "    .CodeMirror pre {\n",
       "    font-family: 'Source Code Pro', monocco, Consolas, monocco, monospace;\n",
       "    }\n",
       "    \n",
       "    .cm-s-ipython span.cm-keyword {\n",
       "        font-weight: normal;\n",
       "     }\n",
       "     \n",
       "     strong {\n",
       "         background: #f5f5f5;\n",
       "         margin-top: 4pt;\n",
       "         margin-bottom: 4pt;\n",
       "         padding: 2pt;\n",
       "         border: 0.5px solid #a0a0a0;\n",
       "         font-weight: bold;\n",
       "         color: darkred;\n",
       "     }\n",
       "     \n",
       "    \n",
       "    div #notebook {\n",
       "        # font-size: 10pt; \n",
       "        line-height: 145%;\n",
       "        }\n",
       "        \n",
       "    li {\n",
       "        line-height: 145%;\n",
       "    }\n",
       "\n",
       "    div.output_area pre {\n",
       "        background: #fff9d8 !important;\n",
       "        padding: 5pt;\n",
       "       \n",
       "       -webkit-print-color-adjust: exact; \n",
       "        \n",
       "    }\n",
       " \n",
       "    \n",
       " \n",
       "    h1, h2, h3, h4 {\n",
       "        font-family: Kameron, arial;\n",
       "\n",
       "\n",
       "    }\n",
       "    \n",
       "    div#maintoolbar {display: none !important;}\n",
       "</style>\n",
       "    <script>\n",
       "IPython.OutputArea.prototype._should_scroll = function(lines) {\n",
       "        return false;\n",
       "}\n",
       "    </script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# IGNORE THIS CELL WHICH CUSTOMIZES LAYOUT AND STYLING OF THE NOTEBOOK !\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib as mpl\n",
    "import seaborn as sns\n",
    "sns.set(style=\"darkgrid\")\n",
    "mpl.rcParams['lines.linewidth'] = 3\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "%config IPCompleter.greedy=True\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore', category=FutureWarning)\n",
    "from IPython.core.display import HTML; HTML(open(\"custom.html\", \"r\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 8: Introduction to Neural Networks\n",
    "\n",
    "\n",
    "\n",
    "<img src=\"./images/3042en.jpg\" title=\"made at imgflip.com\" width=35%/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## History of Neural networks\n",
    "\n",
    "\n",
    "1943 - Threshold Logic\n",
    "\n",
    "1940s - Hebbian Learning\n",
    "\n",
    "1958 - Perceptron\n",
    "\n",
    "1980s - Neocognitron\n",
    "\n",
    "1982 - Hopfield Network\n",
    "\n",
    "1989 - Convolutional neural network (CNN) kernels trained via backpropagation\n",
    "\n",
    "1997 - Long-short term memory (LSTM) model\n",
    "\n",
    "1998 - LeNet-5\n",
    "\n",
    "2014 - Gated Recurrent Units (GRU), Generative Adversarial Networks (GAN)\n",
    "\n",
    "2015 - ResNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why the boom now?\n",
    "* Data\n",
    "* Data\n",
    "* Data\n",
    "* Availability of GPUs\n",
    "* Algorithmic developments which allow for efficient training and making networks networks\n",
    "* Development of high-level libraries/APIs have made the field much more accessible than it was a decade ago"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feed-Forward neural network\n",
    "<center>\n",
    "<figure>\n",
    "<img src=\"./images/neuralnets/neural_net_ex.svg\" width=\"700\"/>\n",
    "<figcaption>A 3 layer densely connected Neural Network (By convention the input layer is not counted).</figcaption>\n",
    "</figure>\n",
    "</center>"
   ]
  },