neural_nets_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Neural Networks\n",
    "\n",
    "## TO DO: Almost all the figues and schematics will be replaced or improved slowly\n",
    "\n",
    "<img src=\"./images/neuralnets/Colored_neural_network.svg\"/>\n",
    "source: https://en.wikipedia.org/wiki/Artificial_neural_network\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## History of Neural networks\n",
    "\n",
    "**TODO: Make it more complete and format properly**\n",
    "\n",
    "1943 - Threshold Logic\n",
    "\n",
    "1940s - Hebbian Learning\n",
    "\n",
    "1958 - Perceptron\n",
    "\n",
    "1975 - Backpropagation\n",
    "\n",
    "1980s - Neocognitron\n",
    "\n",
    "1982: Hopfield Network\n",
    "\n",
    "1986: Convolutional Neural Networks\n",
    "\n",
    "1997: Long-short term memory (LSTM) model\n",
    "\n",
    "2014: GRU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building blocks\n",
    "### Perceptron\n",
    "\n",
    "Smallest unit of a neural network is a **perceptron** like node.\n",
    "\n",
    "**What is a Perceptron?**\n",
    "\n",
    "It is a simple function which has multiple inputs and a single output.\n",
    "\n",
    "Step 1: Weighted sum of the inputs is calculated\n",
    "\n",
    "\\begin{equation*}\n",
    "weighted\\_sum = \\sum_{k=1}^{num\\_inputs} w_{i} x_{i}\n",
    "\\end{equation*}\n",
    "\n",
    "Step 2: The following activation function is applied\n",
    "\n",
    "$$\n",
    "f(weighted\\_sum) = \\left\\{\n",
    "        \\begin{array}{ll}\n",
    "            0 & \\quad g < threshold \\\\\n",
    "            1 & \\quad g \\geq threshold\n",
    "        \\end{array}\n",
    "    \\right.\n",
    "$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config IPCompleter.greedy=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "def perceptron(X, w, threshold=1):\n",
    "    # This function computes sum(w_i*x_i) and \n",
    "    # applies a perceptron activation\n",
    "    linear_sum = np.dot(X,w)\n",
    "    output=0\n",
    "    if linear_sum >= threshold:\n",
    "        output = 1\n",
    "        # print(\"The perceptron has peaked\")\n",
    "    return output\n",
    "X = [1,0]\n",
    "w = [1,1]\n",
    "perceptron(X,w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Boolean AND\n",
    "\n",
    "| x$_1$ | x$_2$ | output |\n",
    "| --- | --- | --- |\n",
    "| 0 | 0 | 0 |\n",
    "| 1 | 0 | 0 |\n",
    "| 0 | 1 | 0 |\n",
    "| 1 | 1 | 1 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Perceptron output for x1, x2 = 0, 0 is 0\n",
      "Perceptron output for x1, x2 = 1, 0 is 0\n",
      "Perceptron output for x1, x2 = 0, 1 is 0\n",
      "Perceptron output for x1, x2 = 1, 1 is 1\n"
     ]
    }
   ],
   "source": [
    "# Calculating Boolean AND using a perceptron\n",
    "import matplotlib.pyplot as plt\n",
    "threshold=1.5\n",
    "w=[1,1]\n",
    "X=[[0,0],[1,0],[0,1],[1,1]]\n",
    "for i in X:\n",
    "    print(\"Perceptron output for x1, x2 = \" + str(i)[1:-1] + \" is \" + str(perceptron(i,w,threshold)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3X2cjXX+x/HXZ2ZyL7m/q0hqc8a9s26XiE2sLav8sN3Q0oSEam3KdidL7Eq00W+sfuhGkgqRktykooZiGL9+3Ty2iCQ0KIzh+/tjjtlJg5k5Z851bt7Px+M8zrnO+Z7z/VxdOu+5znWdzzHnHCIiIgleFyAiIpFBgSAiIoACQUREAhQIIiICKBBERCRAgSAiIkAIAsHMLjKzVWa23cy2mdmIfMaYmU0zs8/NbIuZNQ92XhERCa2kELxGNnCPc26TmZUHNprZCudcRp4x3YDLApdWwIzAtYiIRIig9xCcc7udc5sCtw8B24Hapw27DpjrcqwHLjCzmsHOLSIioROKPYRcZlYXaAZsOO2h2sCOPMs7A/ftPu35KUAKQNmyZVtcccUVoSwvIhw9epSvvvqK7OxsfD4fZuZ1SSISQzZu3Pi9c65qUZ4bskAws3LAQmCkc+7g6Q/n85Rf9MxwzqUCqQB+v9+lpaWFqryIcvLkSb799ltq1arF4cOHmTdvHgMHDiQhQcf4RSQ4ZvZVUZ8bkncgMzuPnDB43jn3Sj5DdgIX5Vm+ENgVirmjUUJCArVq1QJg7ty5pKSk0KFDB/73f//X48pEJJ6F4iwjA2YB251zj59h2GLglsDZRq2BTOfc7jOMjStDhgxhzpw5bN++nSZNmjB+/HiOHz/udVkiEodCsYfQDrgZuMrMPglcupvZYDMbHBizDPgS+ByYCQwNwbwxwcy45ZZbyMjIoGfPnowZM4bhw4d7XZaIxKGgjyE459aR/zGCvGMccEewc8Wy6tWrM3/+fPr160fDhg0B2LdvH2XKlKF06dIeVyci8UBHMSNMz549qV+/PgADBw6kadOmrFu3zuOqRCQeKBAi2LBhw8jKyqJ9+/YMGzaMQ4cOeV2SiMQwBUIE69KlC+np6YwYMYLp06eTnJzMpk2bvC5LRGKUAiHClStXjieeeIL33nuPSy+9lDp16nhdkojEKAVClGjTpg2rVq2icuXKZGdn87vf/Y4FCxag38QWkVBRIEShvXv3smfPHv7rv/6LXr16sXu3vtIhIsFTIEShmjVrsn79eiZNmsTy5ctp0KABzzzzjPYWRCQoCoQolZSUxKhRo9iyZQtNmjRh8uTJZGVleV2WiEQxBUKUu+yyy1i1ahUrV66kZMmSHDp0iKeffpoTJ054XZqIRBkFQgxISEigRo0aADz33HMMGTKE3/zmN2RkZJzjmSIi/6FAiDGDBw/mueee47PPPqNZs2Y8+uij+ihJRApEgRBjzIwbb7yRjIwMevXqxYMPPsiIEb/4mWsRkV8I6S+mSeSoVq0a8+bNo1+/fiQnJwPw/fffU7ZsWTXLE5F8aQ8hxl177bVceumlQE6zvMaNG7NmzRqPqxKRSKRAiCMjRozg5MmTdOzYkSFDhnDw4Om/dCoi8UyBEEeuuuoqtmzZwt13301qairJycls3LjR67JEJEIoEOJM2bJlmTx5Mu+//z6/+tWvuOSSS7wuSUQihAIhTrVq1Yq3336bSpUqkZ2dzTXXXMOLL76o9hcicUyBIHz//ffs37+ffv360bNnT7755huvSxIRDygQhBo1avDBBx8wefJkVqxYgc/nY+bMmdpbEIkzCgQBIDExkbvvvpv09HRatGjBE088oW84i8QZBYL8zKWXXsrKlStzm+UdPHiQ6dOnq1meSBxQIMgvmFlus7wXXniBO+64g7Zt27J161aPKxOR4qRAkLO6/fbbmTdvHl9++SXNmzfn4Ycf1kdJIjFKgSBnZWb07duX7du307t3bx555BHuvPNOr8sSkWKg5nZSIFWqVOH555+nX79+NGjQAMj5beeyZctSpkwZj6sTkVDQHoIUSo8ePX7WLK9Ro0asWrXK46pEJBRCEghm9oyZfWdm+R51NLOOZpZpZp8ELg+GYl7x1j333ENCQgJXXXUVKSkpZGZmel2SiAQhVHsIs4FrzjHmXedc08BlbIjmFQ9deeWVbN68mVGjRjFr1ix8Ph9paWlelyUiRRSSQHDOrQX2h+K1JLqUKVOGSZMmsWHDBpKTk6lXr57XJYlIEYXzGEIbM9tsZm+YWXIY55Uw8Pv9vPXWW7nN8rp27crzzz+v9hciUSRcgbAJqOOcawI8CbyW3yAzSzGzNDNL27t3b5hKk1Dbt28fmZmZ3HTTTfz+979nx44dXpckIgUQlkBwzh10zh0O3F4GnGdmVfIZl+qc8zvn/FWrVg1HaVIMqlevznvvvceUKVNYtWoVycnJ/Pd//zcnT570ujQROYuwBIKZ1TAzC9xuGZh3XzjmFm8kJiYycuRI0tPTadmyJdOmTeP48eNelyUiZxGSL6aZ2TygI1DFzHYCDwHnATjnngZuAIaYWTZwBOjr9OFyXKhXrx4rVqxg7969uc3y5syZw5AhQ0hK0vciRSKJRer7st/vdzqFMfY8/fTTDBkyhBYtWjBr1iyaNGnidUkiMcXMNjrn/EV5rr6pLGF1++2389JLL7Fjxw78fj8PPPAAx44d87osEUGBIGFmZvTu3ZuMjAz69evHuHHjGD58uNdliQhqbiceqVy5MnPnzqVfv35cccUVAHz33XeULVuWsmXLelydSHzSHoJ4qlu3blxyySXAf5rlrVy50uOqROKTAkEixqhRo0hKSqJLly4MGjSIH374weuSROKKAkEiRocOHdi8eTOjR49m9uzZ+Hw+PvroI6/LEokbCgSJKKVLl2bChAls2LCBxo0bU79+fQD1RBIJAwWCRKQWLVqwfPlyKlasyPHjx/ntb3/L3LlzFQwixUiBIBFv//79HDlyhP79+9O9e3e+/vprr0sSiUkKBIl41atX591332XatGm8++67JCcn89RTT6lZnkiIKRAkKiQkJHDnnXeydetW2rZty4wZM8jOzva6LJGYokCQqFK3bl2WL1/OqlWrKFGiBAcPHmTq1KkKB5EQUCBI1DEzTv1exosvvsjIkSNp1aoVn3zyiceViUQ3BYJEtZSUFF5++WW++eYb/H4/Y8aM4ejRo16XJRKVFAgS9a6//noyMjK46aabGD9+vJrliRSRmttJTKhUqRKzZ8+mX79+XH755UBOs7wyZcpQrlw5j6sTiQ7aQ5CY0rVr1581y2vYsCFvvfWWx1WJRAcFgsSs0aNHU7p0abp27cqtt97K/v37vS5JJKIpECRmtWvXjo8//pgxY8bw7LPP4vP5+PDDD70uSyRiKRAkppUqVYpx48aRlpaG3+/PPb6gnkgiv6RAkLjQtGlTXn/9dS644AKOHz9Oly5dmD17toJBJA8FgsSdAwcOkJWVxa233krXrl3597//7XVJIhFBgSBxp1q1aqxZs4annnqKDz74gIYNG/Lkk0+qWZ7EPQWCxKWEhASGDh3K1q1bad++PampqeqHJHFPgSBxrU6dOixbtiy3WV5mZiaPP/44x48f97o0kbBTIEjcMzOqVKkCwEsvvcQ999zDr3/9azZt2uRxZSLhpUAQyeO2227j1VdfZc+ePbRs2ZLRo0dz5MgRr8sSCQsFgshpevbsSUZGBgMGDGDixImMGDHC65JEwiIkgWBmz5jZd2a29QyPm5lNM7PPzWyLmTUPxbwSeYYOhaQkMMu5HjrU64qKpmLFivzrX/9ixYoV3H///QDs2bOHQ4cOeVxZ8YqV7SdFE6o9hNnANWd5vBtwWeCSAswI0bwSQYYOhRkz4MSJnOUTJ3KWo/lNpUuXLtStWxfIaZaXnJzMG2+84W1RxSQWt58UTkgCwTm3Fjhb57DrgLkux3rgAjOrGYq5JXKkphbu/mgzZswYypUrR/fu3bnlllvYt2+f1yWFVKxvPzm3cB1DqA3syLO8M3Dfz5hZipmlmVna3r17w1SahMqpvywLen+0adOmDR9//DEPPPAA8+bNi7lmebG+/eTcwhUIls99v2gi45xLdc75nXP+U7+ZK9EjMbFw90ejkiVLMnbsWDZu3EjLli1jqllePGw/ObtwBcJO4KI8yxcCu8I0t4RJSkrh7o9mjRs3ZsmSJbnN8jp16sSsWbOiOhjiaftJ/sIVCIuBWwJnG7UGMp1zu8M0t4TJ9OkwZMh//qJMTMxZnj7d27qK24EDBwAYNGgQXbp04csvv/S4oqKJ1+0n/2Gh+IvGzOYBHYEqwB7gIeA8AOfc02ZmwD/JORPpJ+BW51za2V7T7/e7tLSzDhGJGCdPnmTmzJmMGjWKEydOMG7cOIYPH06iPm+RMDOzjc45f5GeG6m7uAoEiUY7d+5k8ODBfP3116SlpVGiRAmvS5I4E0wg6JvKIiF04YUXsmTJElavXp3bLO8f//gHWVlZXpcmck4KBJEQMzMqVaoEwIIFCxg1ahR+v5+PPvrI48pEzk6BIFKMBg0axKJFi9i3bx+tW7fmL3/5Cz/99JPXZYnkS4EgUsyuvfZaMjIyGDhwIH//+9/VLE8iVpLXBYjEgwoVKpCamkrfvn259NJLAfj2228pXbo0FSpU8Lg6kRzaQxAJo6uuuoo6deoA/2mWt3TpUo+rEsmhQBDxyEMPPUTFihXp0aMHN954I+rfJV5TIIh4pGXLlmzcuJGHH36YBQsW4PP5WL9+vddlSRxTIIh4qESJEjz00ENs2rSJdu3a0aBBAyA2muVJ9FEgiESAhg0b8tprr1GhQgWysrLo2LEjM2fOVDBIWCkQRCJMZmYmiYmJpKSk0LlzZ7744guvS5I4oUAQiTBVq1Zl5cqVpKamsnHjRho1asTjjz/OCf1SjRQzBYJIBDIzbrvtNjIyMujSpQtz5szh5MmTXpclMU6BIBLBateuzaJFi1i9ejXnnXceP/zwAxMnTlSzPCkWCgSRCGdmVKxYEYCFCxcyevRomjdvzoYNGzyuTGKNAkEkigwcOJDXX3+dzMxM2rRpw913382PP/7odVkSIxQIIlHmd7/7Hdu2bWPw4MFMmTJFzfIkZNTcTiQKnX/++UyfPp0+ffpQt25dAHbv3k2ZMmXULE+KTHsIIlHsyiuvzG2WN2jQIHw+H4sXL/a4KolWCgSRGPHII49QuXJlrrvuOvr27ct3333ndUkSZRQIIjHC7/eTlpbGo48+yquvvorP5+ODDz7wuiyJIgoEkRhSokQJ/vrXv/Lxxx/ToUMHfD4foGZ5UjAKBJEY5PP5eOWVV3Kb5bVv354ZM2bo285yVgoEkRiXmZlJ6dKlGTp0KJ06deKzzz7zuiSJUAoEkRhXtWpV3nrrLWbNmsXmzZtp3LgxkyZNIjs72+vSJMIoEETigJnxpz/9iYyMDK655hpeeOEFHVeQX1AgiMSRWrVq8corr/ysWd6ECRM4duyY16VJBAhJIJjZNWb2qZl9bmaj83l8gJntNbNPApdBoZhXRArPzLjgggsAePXVV7n//vtp1qyZTlGV4APBzBKBp4BugA/oZ2a+fIbOd841DVz+Fey8IhK8W2+9lWXLlnH48GHatWvHyJEj1SwvjoViD6El8Llz7kvnXBbwInBdCF5XRMKgW7dubNu2jaFDhzJ16lQ1y4tjoWhuVxvYkWd5J9Aqn3HXm1kH4P+Au5xzO04fYGYpQArAxRdfHILSRKQgypcvzz//+U/69OmT2xtp165dlC5dOve3GCT2hWIPwfK57/TTF5YAdZ1zjYG3gTn5vZBzLtU553fO+atWrRqC0kSkMNq3b5/7x9ipZnmvvvqqx1VJuIQiEHYCF+VZvhDYlXeAc26fc+7UaQwzgRYhmFdEitG4ceOoUaMGvXr1onfv3nz77bdelyTFLBSB8BFwmZldYmYlgL7Az/rvmlnNPIvXAttDMK+IFKPmzZvz4YcfMn78eJYsWaJmeXEg6EBwzmUDw4A3yXmjf8k5t83MxprZtYFhw81sm5ltBoYDA4KdV0SK33nnncd9993HJ598QufOnUlOTgbULC9WWaRuWL/f79LS0rwuQ0ROk5WVRceOHbnxxhsZMmQICQn6fmskMbONzjl/UZ6rLSkihXLw4EHKly/PsGHDuPLKK/n000+9LklCRIEgIoVSpUoVli9fzuzZs9m2bRtNmjThscceU7O8GKBAEJFCMzP69+9PRkYGPXr0YP78+TquEAMUCCJSZDVq1ODll1/ObZZ34MABHn30UY4ePep1aVIECgQRCVqFChUAWLRoEQ8++CBNmzblvffe87gqKSwFgoiEzIABA3jzzTc5duwY7du358477+TQoUNelyUFpEAQkZC6+uqrSU9P58477+Spp55i5MiRXpckBRSK5nYiIj9Trlw5pk6dSp8+fbjoopzONt988w2lS5emUqVKHlcnZ6I9BBEpNm3bts0NhNtuuw2fz8fChQs9rkrORIEgImExfvx4ateuzQ033MD111/P7t27vS5JTqNAEJGwaNq0KRs2bOCxxx5j6dKl+Hw+nYkUYRQIIhI2SUlJ3HvvvWzZsoWrr76aRo0aAXDy5EmPKxNQIIiIBy6//HLmz5/P+eefT1ZWFm3btmXatGmcOHHC69LimgJBRDx16NAhKlasyIgRI2jfvj3bt+vnUryiQBART1WuXJlly5Yxd+5cPv30U5o2bcq4ceM4fvy416XFHQWCiHjOzLj55pvJyMigZ8+evPLKK16XFJcUCCISMapXr878+fNZs2ZNbrO8Rx55hCNHjnhdWlxQIIhIxClfvjwAixcv5uGHH6ZJkyasXbvW46pinwJBRCJW//79efvtt8nOzubKK6/kjjvu4ODBg16XFbMUCCIS0Tp37kx6ejojR45kxowZ3HXXXV6XFLPU3E5EIl7ZsmWZMmUKffr04cILLwRg586dlCpViipVqnhcXezQHoKIRI3WrVvnBsKpZnn6+c7QUSCISFSaOHEiF198MX379uUPf/gDu3bt8rqkqKdAEJGo1LhxY9avX8/f//533nzzTXw+H+vWrfO6rKimQBCRqJWUlMSf//xn0tPT6d69O40bNwbULK+oFAgiEvXq16/PCy+8wPnnn8+xY8do3bo1TzzxhJrlFZICQURiyuHDh6lWrRp33XUX7dq1Y9u2bV6XFDVCEghmdo2ZfWpmn5vZ6HweL2lm8wOPbzCzuqGYVyLP0KGQlARmOddDh3pdkRRGLGy/ypUrs2TJEp5//nk+//xzmjVrxtixY9UsrwCCDgQzSwSeAroBPqCfmflOGzYQOOCcqw9MASYGO69EnqFDYcYMOLWXfuJEznI0vqnEo1jafmbGH//4R7Zv384NN9zAokWLMDOvy4p4Fuz5u2bWBnjYOdc1sHwfgHNuQp4xbwbGfGBmScC3QFV3lsn9fr9LS0sLqjYJr6Sk/7yZ5JWYCNnZ4a9HCieWt9+hQ4coX748+/fvZ+rUqdx7772UKVPG67KKhZltdM75i/LcUHxkVBvYkWd5Z+C+fMc457KBTKDy6S9kZilmlmZmaXv37g1BaRJOZzp+p+N60SGWt9+pZnmvv/46Y8eOpUmTJqxevdrboiJQKAIhv/2w0//yL8gYnHOpzjm/c85ftWrVEJQm4ZSYWLj7JbLEw/a75ZZbeOedd3DO0alTJwYPHkxmZqbXZUWMUATCTuCiPMsXAqd/ZTB3TOAjowrA/hDMLREkJaVw90tkiZft16lTJ7Zs2cI999zDzJkz1Swvj1A0t/sIuMzMLgG+AfoCfzxtzGKgP/ABcAPwztmOH0h0mj495zo1NedjhsTEnDeTU/dLZIun7VemTBn+8Y9/0KdPH2rWrAnkNMsrWbIk8fzpRNAHlQHMrDvwBJAIPOOc+5uZjQXSnHOLzawU8CzQjJw9g77OuS/P9po6qCwi4dStWzfS0tKYNm0affv2jdqzkoI5qBySQCgOCgQRCaetW7cycOBAPvzwQ3r06MGMGTNyO6tGE6/PMhIRiXoNGzbk/fff5/HHH2flypUkJyfz7rvvel1WWCkQREQCEhMTueuuu9i6dSu///3vadKkCRA/zfIUCCIip6lXrx7PPfdcbrO8Vq1aMXny5JhvlqdAEBE5ix9//JFatWrx5z//mTZt2pCenu51ScVGgSAichaVKlXitdde48UXX+Tf//43zZs356GHHiIrK8vr0kJOgSAicg5mRp8+fcjIyKBv374sXbqUhITYe/uMvTUSESkmVapU4dlnn2XNmjUkJSWxf/9+/vrXv/Ljjz96XVpIKBBERAqpbNmyACxdupS//e1vNG7cmHfeecfjqoKnQBARKaKbb76ZNWvWkJiYSOfOnbntttv44YcfvC6ryBQIIiJB6NChA5s3b+Yvf/kLzzzzDHfffbfXJRVZKJrbiYjEtdKlSzNx4kR69+6d2yxvx44dlCxZkmrVqnlcXcFpD0FEJET8fj+1a+f8PlhKSgoNGjTgueeeI1J7xp1OgSAiUgwmT57M5Zdfzs0330yPHj3YsWPHuZ/kMQWCiEgx8Pl8rFu3jqlTp7J69Wp8Pl/EN8tTIIiIFJPExESGDx/O1q1b6dWrF02bNgWI2J5ICgQRkWJ2ySWXMGfOHMqXL8/Ro0f59a9/zaRJk8jOzva6tJ9RIIiIhNFPP/1E3bp1uffee2nVqhWbN2/2uqRcCgQRkTCqVKkSCxcuZMGCBezcuRO/388DDzwQEc3yFAgiImFmZtxwww1kZGRw4403snz58oholud9BSIicapy5crMnj07t1nevn37uP/++zl8+LAn9SgQREQ8VqZMGQDeeOMNJkyYQKNGjVixYkXY61AgiIhEiJtuuom1a9dSokQJrr76av70pz9x4MCBsM2vQBARiSDt27dn8+bN3HfffcydOzeszfLU3E5EJMKUKlWK8ePH07t379zmeF9//TUlSpSgRo0axTav9hBERCJUs2bNcpvl3X777fh8PubMmVNszfIUCCIiUWDKlCn4fD4GDBhAt27d+Oqrr0I+hwJBRCQKXHHFFaxdu5Ynn3ySdevWkZyczJo1a0I6R1CBYGaVzGyFmX0WuK54hnEnzOyTwGVxMHOKiMSrhIQEhg0bxrZt2+jduzctWrQAQtcsL9g9hNHASufcZcDKwHJ+jjjnmgYu1wY5p4hIXKtTpw7/8z//Q7ly5Th69Ch+v58JEyZw/PjxoF432EC4DpgTuD0H6Bnk64mISCEcOXKE+vXrc//999OqVaugXivYQKjunNsNELg+04+HljKzNDNbb2YKDRGREKlYsSILFixg4cKF7N69O6jXOuf3EMzsbSC/E1/HFGKei51zu8ysHvCOmaU7577IZ64UIAXg4osvLsTLi4jEt169etGpUycqVapU5Nc4ZyA457qc6TEz22NmNZ1zu82sJvDdGV5jV+D6SzNbDTQDfhEIzrlUIBXA7/dHx69Si4hEiIoV8z2vp8CC/choMdA/cLs/sOj0AWZW0cxKBm5XAdoBGUHOKyIiIRZsIDwG/NbMPgN+G1jGzPxm9q/AmAZAmpltBlYBjznnFAgiIhEmqF5Gzrl9QOd87k8DBgVuvw80CmYeEREpfvqmsoiIAAoEEREJUCCIiAigQBARkQAFgoiIAAoEEREJUCCIiAigQBARkQAFgoiIAAoEEREJUCCIiAigQBARkQAFgoiIAAoEEREJUCCIiAigQBARkQAFgoiIAAoEEREJUCCIiAigQBARkQAFgoiIAAoEEREJUCCIiAigQBARkQAFgoiIAAoEEREJUCCIiAigQBARkYCgAsHMepvZNjM7aWb+s4y7xsw+NbPPzWx0MHOKiEjxCHYPYSvQC1h7pgFmlgg8BXQDfEA/M/MFOa+IiIRYUjBPds5tBzCzsw1rCXzunPsyMPZF4DogI5i5RUQktIIKhAKqDezIs7wTaJXfQDNLAVICi8fMbGsx1+alKsD3XhdRjLR+0S2W1y+W1w3gV0V94jkDwczeBmrk89AY59yiAsyR3+6Dy2+gcy4VSA3Mm+acO+NxiWin9YtuWr/oFcvrBjnrV9TnnjMQnHNdivriATuBi/IsXwjsCvI1RUQkxMJx2ulHwGVmdomZlQD6AovDMK+IiBRCsKed/sHMdgJtgKVm9mbg/lpmtgzAOZcNDAPeBLYDLznnthXg5VODqS0KaP2im9YvesXyukEQ62fO5ftxvoiIxBl9U1lERAAFgoiIBERMIMR6Gwwzq2RmK8zss8B1xTOMO2FmnwQuEX/w/Vzbw8xKmtn8wOMbzKxu+KssugKs3wAz25tnmw3yos6iMLNnzOy7M33fx3JMC6z7FjNrHu4ag1GA9etoZpl5tt2D4a6xqMzsIjNbZWbbA++bI/IZU/jt55yLiAvQgJwvVKwG/GcYkwh8AdQDSgCbAZ/XtRdw/SYBowO3RwMTzzDusNe1FmKdzrk9gKHA04HbfYH5Xtcd4vUbAPzT61qLuH4dgObA1jM83h14g5zvErUGNnhdc4jXryPwutd1FnHdagLNA7fLA/+Xz7/NQm+/iNlDcM5td859eo5huW0wnHNZwKk2GNHgOmBO4PYcoKeHtYRKQbZH3vV+Gehs5+h1EkGi+d/bOTnn1gL7zzLkOmCuy7EeuMDMaoanuuAVYP2ilnNut3NuU+D2IXLO4Kx92rBCb7+ICYQCyq8Nxun/ESJVdefcbsjZmEC1M4wrZWZpZrbezCI9NAqyPXLHuJxTkDOBymGpLngF/fd2fWCX/GUzuyifx6NVNP//VlBtzGyzmb1hZsleF1MUgY9hmwEbTnuo0NsvHL2McoWzDYYXzrZ+hXiZi51zu8ysHvCOmaU7574ITYUhV5DtEdHb7BwKUvsSYJ5z7piZDSZnb+iqYq8sPKJ52xXEJqCOc+6wmXUHXgMu87imQjGzcsBCYKRz7uDpD+fzlLNuv7AGgovxNhhnWz8z22NmNZ1zuwO7bd+d4TV2Ba6/NLPV5CR/pAZCQbbHqTE7zSwJqED07Mafc/2cc/vyLM4EJoahrnCJ6P/fgpX3DdQ5t8zMpptZFedcVDS+M7PzyAmD551MtY1PAAABQ0lEQVRzr+QzpNDbL9o+MormNhiLgf6B2/2BX+wRmVlFMysZuF0FaEdktwkvyPbIu943AO+4wBGvKHDO9TvtM9lryfksN1YsBm4JnK3SGsg89bFnLDCzGqeOZ5lZS3LeD/ed/VmRIVD3LGC7c+7xMwwr/Pbz+mh5niPifyAn0Y4Be4A3A/fXApadduT8/8j5q3mM13UXYv0qAyuBzwLXlQL3+4F/BW63BdLJOZslHRjodd0FWK9fbA9gLHBt4HYpYAHwOfAhUM/rmkO8fhOAbYFttgq4wuuaC7Fu84DdwPHA/3sDgcHA4MDjRs6PW30R+PeY79l/kXopwPoNy7Pt1gNtva65EOv2G3I+/tkCfBK4dA92+6l1hYiIANH3kZGIiBQTBYKIiAAKBBERCVAgiIgIoEAQEZEABYKIiAAKBBERCfh/0ezu6YQKKxgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f37f81bd400>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plotting the decision boundary\n",
    "plt.xlim(-1,2)\n",
    "plt.ylim(-1,2)\n",
    "for i in X:\n",
    "    plt.plot(i,\"o\",color=\"b\");\n",
    "# Plotting the decision boundary\n",
    "# that is a line given by w_1*x_1+w_2*x_2-threshold=0\n",
    "plt.plot(np.arange(-3,4), 1.5-np.arange(-3,4), \"--\", color=\"black\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise :Can you compute a Boolean \"OR\" using a perceptron?**\n",
    "\n",
    "Hint: copy the code from the \"AND\" example and edit the weights and/or threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Boolean OR\n",
    "\n",
    "| x$_1$ | x$_2$ | output |\n",
    "| --- | --- | --- |\n",
    "| 0 | 0 | 0 |\n",
    "| 1 | 0 | 1 |\n",
    "| 0 | 1 | 1 |\n",
    "| 1 | 1 | 1 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculating Boolean OR using a perceptron\n",
    "# Edit the code below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Perceptron output for x1, x2 = 0, 0 is 0\n",
      "Perceptron output for x1, x2 = 1, 0 is 1\n",
      "Perceptron output for x1, x2 = 0, 1 is 1\n",
      "Perceptron output for x1, x2 = 1, 1 is 1\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl4VOXdxvHvLwm7FIEgsskiKiRGtmETURSVUjVFAgr6ClQUEdTS4gayiSIWLRVFQLZXoEVRXjYFpaAgKoIGZAmJC1iRfRNBREHI8/6RkUZMIMlMcma5P9c1V2Z5Zs59OCF3zjkzT8w5h4iISIzXAUREJDSoEEREBFAhiIiInwpBREQAFYKIiPipEEREBAhCIZhZDTNbZmYZZrbJzP6cwxgzs+fNbLOZbTCzxoEuV0REgisuCK9xAujvnFtrZmWBNWa2xDmXnm1Me+Ai/6U5MN7/VUREQkTAewjOuV3OubX+698DGUC104b9EZjusqwCzjWzKoEuW0REgicYewinmFktoBGw+rSHqgHbst3e7r9v12nP7wX0AihTpkyTevXqBTOeiEjEW7NmzX7nXKWCPDdohWBm5wD/B/Rzzh0+/eEcnvKbOTOccxOBiQA+n8+lpqYGK56ISFQws60FfW5Q3mVkZsXIKoN/Oefm5DBkO1Aj2+3qwM5gLFtERIIjGO8yMmAKkOGcG53LsAVAN/+7jVoAh5xzu3IZKyIiHgjGIaNWwB3ARjNb579vIHABgHNuArAI+AOwGTgK/CkIyxURkSAKuBCccx+Q8zmC7GMc0DfQZYmISOHRJ5VFRARQIYiIiJ8KQUREABWCiIj4qRBERARQIYiIiJ8KQUREABWCiIj4qRBERARQIYiIiJ8KQUREABWCiIj4qRBERARQIYiIiJ8KQUREABWCiIj4qRBERARQIYiIiJ8KQUREABWCiIj4qRBERARQIYiIiJ8KQUREgBAuBOccHTt25KOPPvI6iohIVAjZQjh27Bhr1qyhVatW9OvXjyNHjngdSUQkooVsIZQsWZK0tDT69OnDmDFjSEpKYsmSJV7HEhGJWCFbCABly5Zl7NixrFixguLFizNw4EAyMzO9jiUiEpGCUghmNtXM9ppZWi6PtzGzQ2a2zn8Zkp/Xb926NevXr2fOnDnExMRw4MAB5s+fH4zoIiLiF6w9hJeB359lzPvOuYb+y/D8LqBkyZLUqFEDgOeee44OHTpwyy23sGfPnvynFRGR3whKITjnVgDfBuO18mLIkCGMGDGC+fPnU79+faZPn45zrqgWLyISkYryHEJLM1tvZm+ZWWIgL1SsWDEGDhzI+vXrqV+/Pt27d2fEiBHByikiEpXiimg5a4GazrkjZvYHYB5w0emDzKwX0AvgggsuOOuL1qtXj/fff5/x48fToUMHAA4ePEi5cuWIiQnp8+UiIiHHgnWoxcxqAW865y7Nw9ivAZ9zbn9uY3w+n0tNTc1XBucc11xzDSdOnGDy5Mlccskl+Xq+iEi4M7M1zjlfQZ5bJL9Gm9n5Zmb+6838yz1QGMvq0aMHmzZtokGDBjz99NP8/PPPhbEYEZGIE6y3nb4CfARcYmbbzaynmfU2s97+IZ2ANDNbDzwPdHGFcBbYzOjevTvp6encdNNNDBgwgObNm/PVV18Fe1EiIhEnaIeMgq0gh4xON2fOHEaNGsWSJUsoW7ZskJKJiISukD9k5JVfJscrW7Ysx44d4+abb+bDDz/0OpaISEiK6EKArMNIAF9//TWffvoprVu35oEHHtBkeSIip4n4QvjFJZdcQlpaGvfffz9jx44lMTGRxYsXex1LRCRkRE0hAJxzzjmMGTOG999/n1KlSjFo0CBNlici4hdVhfCLVq1asW7dOubOnXtqsrw5c+Z4HUtExFNRWQiQNVle9erVARgzZgwpKSmkpKSwa9cuj5OJiHgjagshuyFDhvD000+zcOFCEhIS+N///V9NliciUUeFAMTFxfHII4+wYcMGkpKSuPPOO3nyySe9jiUiUqSKanK7sHDxxRezfPlyJk6cSHJyMgDffvst5cqVIzY21uN0IiKFS3sIp4mJiaF3795UrVqVzMxMOnbsyJVXXklGRobX0URECpUK4QzMjJ49e/LZZ5/RsGFDRowYocnyRCRiqRDOwMy44447SE9Pp0OHDgwaNIimTZtqsjwRiUgqhDyoXLkys2bNYu7cuZQuXZrzzjvP60giIkGnQsiHDh068OGHH3LOOefw008/kZyczIoVK7yOJSISFCqEfPplsrxvvvmGtLQ0rrrqKvr27cvhw4c9TiYiEhgVQgFdfPHFbNy4kb/85S+MHz+eSy+9lEWLFnkdS0SkwFQIAShTpgyjR49m5cqVlC1blmHDhmmyPBEJWyqEIGjRogVr1649NVne/v37mT17tqa/EJGwokIIkhIlSlCtWjUAXnjhBTp37kzHjh3ZuXOnx8lERPJGhVAIBg8ezDPPPMPbb79NQkICU6ZM0d6CiIQ8FUIhiIuL48EHH2Tjxo00bNiQu+66i+HDh3sdS0TkjDS5XSGqW7cu7777LpMnT+bGG28EYP/+/ZQvX16T5YlIyNEeQiGLiYmhV69epybLS0lJoVWrVmzatMnraCIiv6JCKEJmRu/evdmyZQuNGjVi+PDhHD9+3OtYIiKACqFImRldu3YlPT2dTp06MXToUJo0acKWLVu8jiYiokLwQqVKlZg5cyYLFizg3HPPpXLlyl5HEhFRIXjppptuYsWKFacmy7vxxhtZvny517FEJEqpEDyWfbK8zz77jKuvvprevXtz6NAhj5OJSLQJSiGY2VQz22tmabk8bmb2vJltNrMNZtY4GMuNJBdffDEbNmygf//+TJo0icTERBYuXOh1rHzr0wfi4sAs62ufPl4nkvzQ9otuwdpDeBn4/Rkebw9c5L/0AsYHabkRpXTp0jz77LN89NFHlC9fnuHDh4fVZHl9+sD48XDyZNbtkyezbuuHSnjQ9hML1pQKZlYLeNM5d2kOj70ELHfOveK//TnQxjm3K7fX8/l8LjU1NSjZwtHx48fZv38/VatWZf/+/SxdupRbb7311CGmUBQX998fJtnFxsKJE0WfR/JH2y8ymNka55yvIM8tqnMI1YBt2W5v99/3K2bWy8xSzSx13759RRQtNBUvXpyqVasCMHbsWLp27UpycjLbt2/3OFnucvphcqb7JbRo+0lRFUJOv9b+ZtfEOTfROedzzvkqVapUBLHCw+DBgxk9ejTvvPMOiYmJTJw4MSQPJeU2G4dm6QgP2n5SVIWwHaiR7XZ1QPNC51FsbCx/+ctfSEtLw+fzcc899/DEE094Hes3evXK3/0SWrT9pKgKYQHQzf9uoxbAoTOdP5Cc1alTh6VLlzJlyhR6+f+X7t+/nxMhcoB33Di4997//kYZG5t1e9w4b3NJ3mj7SVBOKpvZK0AbIB7YAwwFigE45yZY1pnQsWS9E+ko8Cfn3BnPGEf7SeW8yMzMpE2bNvz0009MmTKFpKQkryOJiMc8P6nsnOvqnKvinCvmnKvunJvinJvgnJvgf9w55/o65y50ziWdrQwkb8yMvn378vXXX9O4cWOGDh3KsWPHvI4lImFKn1QOY2bGrbfeSnp6Ol26dGH48OE0adKEzZs3ex1NRMKQCiECxMfHM2PGDN58800qVqxIlSpVvI4kImFIhRBBbrjhBt577z3KlCnDjz/+SPv27XnnnXe8jiUiYUKFEKG2b9/Oli1buPbaa7n77rv57rvvvI4kIiFOhRChLrroItavX8/DDz/M1KlTSUhIYP78+V7HEpEQpkKIYKVKleJvf/sbq1evplKlSowcOTIkP+EsIqFBhRAFfD4fqampzJ07l5iYGPbt28fMmTMJ1sSGIhIZVAhRolixYqfeffTiiy9y++23c+ONN7Jt27azPFNEooUKIQoNHjyYMWPGsHz5chITExk/frwOJYmICiEaxcbG8sADD5CWlkbz5s3p06cPjz/+uNexRMRjcV4HEO/Url2bf//730ybNo127doBsHfvXipUqEBcnL41RKKN9hCinJnRo0cPqlSpQmZmJikpKTRv3pz169d7HU1EipgKQU6JiYmhX79+7NixA5/Px6BBg/jpp5+8jiUiRUSFIL+SkpJCeno6t99+OyNGjKBRo0Z8+eWXXscSkSKgQpDfqFChAi+//DJvv/02VapUOfW3nUUksqkQJFft2rXj3XffPTVZXrt27ViyZInXsUSkkKgQJE927NjB1q1buf7667nzzjs5ePCg15FEJMhUCJIndevWZd26dQwYMIDp06eTkJDA3LlzvY4lIkGkQpA8K1myJE899RQff/wx559/PqNGjdInnEUiiApB8q1x48Z8/PHHv5osb8aMGZosTyTMqRCkQIoVK8b5558PwLhx4+jWrRvt27dn69atHicTkYJSIUjABg8ezAsvvMAHH3xAYmIiY8eO1aEkkTCkQpCAxcTEcN9997Fp0yauuOIK7r//foYNG+Z1LBHJJ81gJkFTs2ZN3nrrLf75z39y3XXXAbBnzx4qVKhAsWLFPE4nImejPQQJKjPjjjvu4Pzzz//VZHmffvqp19FE5CxUCFJoYmJi+Otf/8rOnTtp2rQpAwcO1GR5IiFMhSCFqmPHjmRkZNCtWzdGjhxJw4YN+eKLL7yOJSI5CEohmNnvzexzM9tsZo/m8HgPM9tnZuv8l7uCsVwJD+XLl2fq1KksXryY6tWrU61aNa8jiUgOAi4EM4sFXgTaAwlAVzNLyGHoLOdcQ/9lcqDLlfBz/fXXs3TpUsqUKcPRo0e59tprefvtt72OJSJ+wdhDaAZsds595Zw7DrwK/DEIrysRbNeuXezcuZP27dvTvXt3Dhw44HUkkagXjEKoBmzLdnu7/77TpZjZBjObbWY1cnohM+tlZqlmlrpv374gRJNQdeGFF/Lpp58yaNAgZs6cSUJCArNnz9b0FyIeCkYhWA73nf6/+g2glnPuMmApMC2nF3LOTXTO+ZxzvkqVKgUhmoSyEiVK8MQTT5CamkqNGjUYPXq0CkHEQ8EohO1A9t/4qwM7sw9wzh1wzh3z35wENAnCciVCNGjQgFWrVp2aLG/v3r1MmzZN5SBSxIJRCJ8AF5lZbTMrDnQBFmQfYGZVst1MBjKCsFyJIHFxcVSuXBmACRMm0KNHD66//nr+85//eJxMJHoEXAjOuRPAfcBisn7Qv+ac22Rmw80s2T/sATPbZGbrgQeAHoEuVyLXoEGDGDduHKtWreLSSy/l+eef5+TJk17HEol4Fqq75T6fz6WmpnodQzz0zTff0Lt3b9566y0GDRrEE0884XUkkZBnZmucc76CPFeT20nIuuCCC1i4cCEzZ87k2muvBWD37t1UrFhRk+WJFAJNXSEhzcy4/fbbqVy5Ms45UlJS8Pl8rFmzxutoIhFHhSBhw8x45JFH2L9/P82aNeORRx7hxx9/9DqWSMRQIUhYSU5OZtOmTdx5552MGjWKBg0aaLI8kSBRIUjYOffcc5k0aRJLly6ldu3aVK9e3etIIhFBhSBhq23btixevJjSpUtz9OhRrrnmGhYtWuR1LJGwpUKQiLB792727NnDDTfcwB133MH+/fu9jiQSdlQIEhHq1KnD2rVrGTp0KK+++ioJCQm89tprmv5CJB9UCBIxSpQowbBhw1i7di01a9ZkzJgxKgSRfFAhSMRJSkrio48+OjVZ3p49e5g6darKQeQsVAgSkeLi4jjvvPMAeOmll+jZsydt27Zly5YtHicTCV0qBIl4gwYNYsKECaSmppKUlMTo0aM1WZ5IDlQIEvFiYmK45557SE9Pp23btvTv359hw4Z5HUsk5GhyO4ka1atXZ8GCBbz22mtcffXVQNbfdq5YsSLFixf3OJ2I97SHIFHFzLj11ls577zzTk2W16RJEz755BOvo4l4ToUgUcvMGDBgAAcPHqRFixY8+OCDHD161OtYIp5RIUhUu+mmm9i0aRN33303f//737nsssv4/PPPvY4l4gkVgkS9cuXKMWHCBJYtW0bdunWpUaOG15FEPKFCEPFr06YNb7/99qnJ8tq0acMbb7zhdSyRIqNCEMnB7t27OXDgAMnJydx2223s27fP60gihU6FIJKDOnXqsGbNGh5//HFmz55N/fr1mTlzpqa/kIimQhDJRfHixRkyZAiffvopdevW5cUXX1QhSERTIYicRWJiIh9++CHz5s07NVne5MmTyczM9DqaSFCpEETyIDY2lkqVKgEwadIk7r77btq2bcvmzZs9TiYSPCoEkXx67LHHmDRpEmvXriUpKYlnn32WEydOeB1LJGAqBJF8MjPuuusu0tPTuf7663nooYcYOnSo17FEAqbJ7UQKqFq1asybN4/Zs2dz1VVXAVmT5VWoUIESJUp4nE4k/4Kyh2Bmvzezz81ss5k9msPjJcxslv/x1WZWKxjLldDTpw/ExYFZ1tc+fbxOVLjMjM6dO/9msrzVq1d7Ha1Aom37ya8FXAhmFgu8CLQHEoCuZpZw2rCewEHnXF3gH8DfAl2uhJ4+fWD8ePjlb8+cPJl1O1p+qJgZgwYN4tChQ7Rs2ZK//vWv/PDDD17HyrNo334CFuj7qs2sJTDMOdfOf3sAgHNuZLYxi/1jPjKzOGA3UMmdYeE+n8+lpqYGlE2KVlzcf3+YZBcbC9F0zvXw4cMMGDCAcePGUbt2bRYtWkS9evW8jnVW2n6RwczWOOd8BXluMA4ZVQO2Zbu93X9fjmOccyeAQ0DF01/IzHqZWaqZpWqqgPCT21+ljLa/Vvm73/2OF198kffee4/69etTs2ZNgJD/UJu2nwSjECyH+07/zs/LGJxzE51zPuec75f3fEv4iI3N3/2R7sorr2ThwoWUKlWKH374gauuuor58+d7HStX2n4SjELYDmSfL7g6sDO3Mf5DRuWAb4OwbAkhvXrl7/5osnfvXg4fPkyHDh249dZb2bNnj9eRfkPbT4JRCJ8AF5lZbTMrDnQBFpw2ZgHQ3X+9E/Dumc4fSHgaNw7uvfe/v1HGxmbdHjfO21yhoHbt2nzyySc8+eSTzJs3j4SEBP75z3+G1GEkbT8J+KQygJn9AXgOiAWmOudGmNlwINU5t8DMSgIzgEZk7Rl0cc59dabX1ElliVQZGRn07NmTmJgYVqxYQUyMPh8qwRPISeWgFEJhUCFIJDt58iQHDx4kPj6e3bt3M3/+fO6++26VgwTM63cZiUg+xcbGEh8fD8CUKVPo3bs3bdq04YsvvvA4mUQzFYKIxwYOHMjUqVPZuHEjDRo0YNSoUZosTzyhQhDxmJnxpz/9ifT0dNq3b88jjzyiyfLEE5rcTiREVKlShTlz5jBnzhxat24NwI4dO6hYsSIlS5b0OJ1EA+0hiISYjh07UqlSJZxzdOrUiUaNGrFy5UqvY0kUUCGIhCgzY9iwYRw9epQrrriCP//5zxw5csTrWBLBVAgiIaxdu3akpaXRt29fXnjhBS699FIyMjK8jiURSoUgEuLKli3LCy+8wIoVK0hKSqJWrVpA6E+WJ+FHhSASJq644greeOONU5PltW7dmjlz5ngdSyKICkEkDO3bt48ffviBlJQUOnXqxO7du72OJBFAhSAShmrVqsXHH3/MU089xZtvvklCQgIvv/yyDiNJQFQIImGqWLFiDBgwgHXr1pGQkMDUqVNVCBIQFYJImKtXrx4rVqxg3rx5xMTEsHv3bsaNG0dmZqbX0STMqBBEIkBMTAwVKlQAYOrUqfTt25crr7ySzz77zONkEk5UCCIRZsCAAUybNo309HQaNGjAU089xc8//+x1LAkDKgSRCGNmdOvWjYyMDJKTk3nssccYNmyY17EkDGhyO5EIVblyZV5//XXmzZtHq1atAE2WJ2emPQSRCNehQ4dfTZbXoEEDPvjgA69jSQhSIYhECTNj+PDhHD9+nNatW3Pffffx/fffex1LQogKQSSKXHfddWzcuJEHHniAcePGkZiYSHp6utexJESoEESizDnnnMOYMWP44IMPaNSoEbVr1wY0WZ6oEESi1uWXX878+fNPTZZ3+eWX8/rrr6sYopgKQUTYt28fP//8M7fccgspKSns2rXL60jiARWCiFCrVi1WrVrFqFGjeOuttzQ3UpRSIYgIAHFxcTz00EOsX7+eyy67jGnTpqkQoowKQUR+5eKLL2bZsmXMnTuXmJgYdu3axdixYzl58qTX0aSQqRBE5DeyT5b38ssvc//999O6dWv9PecIF1AhmFkFM1tiZl/6v5bPZdxJM1vnvywIZJkiUrQeffRRZsyYwRdffEHDhg158sknNVlehAp0D+FR4B3n3EXAO/7bOfnROdfQf0kOcJkiUoTMjP/5n/8hPT2dm2++mcGDBzNkyBCvY0khsEBOGpnZ50Ab59wuM6sCLHfOXZLDuCPOuXPy89o+n8+lpqYWOJuIFI4FCxbQsmVLKlWqxLZt24iPj6dUqVJexxI/M1vjnPMV5LmB7iFUds7tAvB/PS+XcSXNLNXMVplZhwCXKSIeSk5O/tVkeZdddhnvvfee17EkCM5aCGa21MzScrj8MR/LucDfWLcBz5nZhbksq5e/OFL37duXj5cXkaJmZowcOZLMzEzatGnDvffey+HDh72OJQEokkNGpz3nZeBN59zsM43TISOR8PDDDz8wZMgQnnvuOapWrcrixYtJSEjwOlbU8vKQ0QKgu/96d2D+6QPMrLyZlfBfjwdaAZpeUSRClClThr///e+sXLmSpk2bUqdOHUCT5YWjQAvhaeA6M/sSuM5/GzPzmdlk/5j6QKqZrQeWAU8751QIIhGmefPmzJkzh5IlS3LkyBFatGjBrFmzVAxhJKBCcM4dcM61dc5d5P/6rf/+VOfcXf7rK51zSc65Bv6vU4IRXERC14EDB3DO0aVLFzp06MDOnTu9jiR5oE8qi0jQ1axZk5UrV/Lss8/y73//m4SEBCZPnqy9hRCnQhCRQhEXF0f//v3ZuHEjDRs2ZMaMGSqEEKdCEJFCVbduXd59913mzZt3arK8MWPGaLK8EKRCEJFCFxMTQ/nyWVOdTZ8+nX79+nH55ZeTlpbmcTLJToUgIkXq4Ycf5pVXXuGrr76icePGPP744xw/ftzrWIIKQUSKmJnRpUsXMjIy6Ny5M8OGDWPo0KFexxIgzusAIhKd4uPj+de//sVtt91G8+bNAfjmm2+Ij4+ndOnSHqeLTtpDEBFP3XDDDcTHx+Oco3PnziQlJbFs2TKvY0UlFYKIhAQz4+mnn8bMuOaaa7jnnns4dOiQ17GiigpBRELG1VdfzYYNG3jwwQeZPHkyCQkJbNq0yetYUUOFICIhpXTp0jzzzDOsWrWKli1bcuGFWbPl60NthU+FICIhqWnTpsyePZuSJUvy/fff07RpU2bOnKliKEQqBBEJeQcPHiQuLo7bb7+dm266iW3btnkdKSKpEEQk5F1wwQV8+OGH/OMf/2DZsmUkJiby0ksvkZmZ6XW0iKJCEJGwEBsbS79+/di4cSPNmjXj1Vdf9TpSxFEhiEhYqVOnDkuWLGHu3LnExMSwc+dORo8ezYkTJ7yOFvZUCCISdsyMc889F4AZM2bQv39/WrZsyYYNGzxOFt5UCCIS1h5++GFmzZrF1q1badKkCUOHDuXYsWNexwpLKgQRCWtmxi233EJGRgZdu3Zl+PDhmiyvgDS5nYhEhIoVKzJ9+nS6du1Ks2bNANi6dSvx8fGUKVPG43ThQXsIIhJR2rdvT8WKFXHOccstt5CUlMTSpUu9jhUWVAgiEpHMjGeffZa4uDiuu+46evbsyXfffed1rJCmQhCRiNW6dWvWr1/Po48+yrRp00hISNCf7TwDFYKIRLRSpUoxcuRIVq9eTevWralbty6gyfJyokIQkajQpEkTZs2adWqyPJ/Px4wZM1QM2agQRCTqfPfdd5QoUYJu3bpxww038M0333gdKSSoEEQk6tSoUYP333+f559/nhUrVpCYmMj48eOjfrI8FYKIRKXY2Fjuv/9+0tLSaNmyJa+//jpm5nUsTwVUCGbW2cw2mVmmmfnOMO73Zva5mW02s0cDWaaISDDVqlWLxYsXM3fuXMyMHTt28Mwzz0TlZHmB7iGkAR2BFbkNMLNY4EWgPZAAdDWzhACXKyISNGZGuXLlAJg5cyYPP/wwzZs3Z926dR4nK1oBFYJzLsM59/lZhjUDNjvnvnLOHQdeBf4YyHJFRArLQw89xOzZs9mxYwc+n4/HHnuMn376yetYRcKC8ZYrM1sOPOicS83hsU7A751zd/lv3wE0d87dl8PYXkAv/81LydoDiVTxwH6vQxQirV94i+T1i+R1A7jEOVe2IE886+R2ZrYUOD+Hhx5zzs3PwzJyOkuTYws55yYCE/3LTXXO5XpeItxp/cKb1i98RfK6Qdb6FfS5Zy0E59y1BX1xv+1AjWy3qwM7A3xNEREJsqJ42+knwEVmVtvMigNdgAVFsFwREcmHQN92erOZbQdaAgvNbLH//qpmtgjAOXcCuA9YDGQArznnNuXh5ScGki0MaP3Cm9YvfEXyukEA6xeUk8oiIhL+9EllEREBVAgiIuIXMoUQ6dNgmFkFM1tiZl/6v5bPZdxJM1vnv4T8yfezbQ8zK2Fms/yPrzazWkWfsuDysH49zGxftm12lxc5C8LMpprZXjPL8fM+luV5/7pvMLPGRZ0xEHlYvzZmdijbthtS1BkLysxqmNkyM8vw/9z8cw5j8r/9nHMhcQHqA5cAywFfLmNigS1AHaA4sB5I8Dp7HtdvFPCo//qjwN9yGXfE66z5WKezbg+gDzDBf70LMMvr3EFevx7AWK+zFnD9rgQaA2m5PP4H4C2yPkvUAljtdeYgr18b4E2vcxZw3aoAjf3XywJf5PC9me/tFzJ7CC7yp8H4IzDNf30a0MHDLMGSl+2Rfb1nA20tfKaUDOfvt7Nyzq0Avj3DkD8C012WVcC5ZlalaNIFLg/rF7acc7ucc2v9178n6x2c1U4blu/tFzKFkEfVgG3Zbm/nt/8Ioaqyc24XZG1M4LxcxpU0s1QzW2VmoV4aedkep8a4rLcgHwIqFknH05kXAAACO0lEQVS6wOX1+y3Fv0s+28xq5PB4uArn/2951dLM1pvZW2aW6HWYgvAfhm0ErD7toXxvv7N+UjmYinIaDC+caf3y8TIXOOd2mlkd4F0z2+ic2xKchEGXl+0R0tvsLPKS/Q3gFefcMTPrTdbe0DWFnqxohPO2y4u1QE3n3BEz+wMwD7jI40z5YmbnAP8H9HPOHT794RyecsbtV6SF4CJ8GowzrZ+Z7TGzKs65Xf7dtr25vMZO/9ev/JMGNiLrOHYoysv2+GXMdjOLA8oRPrvxZ10/59yBbDcnAX8rglxFJaT/vwUq+w9Q59wiMxtnZvHOubCY+M7MipFVBv9yzs3JYUi+t1+4HTIK52kwFgDd/de7A7/ZIzKz8mZWwn89HmgFpBdZwvzLy/bIvt6dgHed/4xXGDjr+p12TDaZrGO5kWIB0M3/bpUWwKFfDntGAjM7/5fzWWbWjKyfhwfO/KzQ4M89Bchwzo3OZVj+t5/XZ8uznRG/maxGOwbsARb7768KLDrtzPkXZP3W/JjXufOxfhWBd4Av/V8r+O/3AZP91y8HNpL1bpaNQE+vc+dhvX6zPYDhQLL/ekngdWAz8DFQx+vMQV6/kcAm/zZbBtTzOnM+1u0VYBfws///Xk+gN9Db/7iR9cettvi/H3N891+oXvKwfvdl23argMu9zpyPdbuCrMM/G4B1/ssfAt1+mrpCRESA8DtkJCIihUSFICIigApBRET8VAgiIgKoEERExE+FICIigApBRET8/h/ojtcNQhRRCAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f37f8036a90>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Solution\n",
    "# Calculating Boolean OR using a perceptron\n",
    "import matplotlib.pyplot as plt\n",
    "threshold=0.6\n",
    "w=[1,1]\n",
    "X=[[0,0],[1,0],[0,1],[1,1]]\n",
    "for i in X:\n",
    "    print(\"Perceptron output for x1, x2 = \" + str(i)[1:-1] + \" is \" + str(perceptron(i,w,threshold)))\n",
    "# Plotting the decision boundary\n",
    "plt.xlim(-1,2)\n",
    "plt.ylim(-1,2)\n",
    "for i in X:\n",
    "    plt.plot(i,\"o\",color=\"b\");\n",
    "# Plotting the decision boundary\n",
    "# that is a line given by w_1*x_1+w_2*x_2-threshold=0\n",
    "plt.plot(np.arange(-3,4), threshold-np.arange(-3,4), \"--\", color=\"black\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Optional exercise: Create a NAND gate with perceptrons**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculating Boolean NAND using a perceptron\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact a single perceptron can compute \"AND\", \"OR\" and \"NOT\" boolean functions.\n",
    "However, it cannot compute some other boolean functions such as \"XOR\"\n",
    "\n",
    "WHAT CAN WE DO?\n",
    "Hint: What is the significance of the NAND gate we created above\n",
    "\n",
    "We said a single perceptron can't compute these functions. We didn't say that about **multiple Perceptrons**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**XOR function**\n",
    "\n",
    "**TO DO: INSERT IMAGE HERE!!!!!!!!!!!!!!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Activation Functions\n",
    "\n",
    "Some of the commonly used activation functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJztvXt8XNWV5/td1sNP+f2SLT9wLIwfgDEygQ7DDTHGRMkY+iY3sXuSNmMSPvQ4c7s707ntfDIfkjiXHpF8MulOm+40DUzcTGKHpAPmprGJMdBN6BBjCUFj0UYGW7Jk+amSZD1LVbXuH+eUXJZKdkmqxzlH6/v5lHVqn32OlryrfrVq7b3XElXFMAzDCBZjcm2AYRiGkX5M3A3DMAKIibthGEYAMXE3DMMIICbuhmEYAcTE3TAMI4CYuBuGYQQQE3fDMIwAYuJuGIYRQPJz9YtnzpypixcvztWvNxKorKw8r6qz0nEvG1fvYOMaTFId15yJ++LFizl8+HCufr2RgIjUpeteNq7ewcY1mKQ6rhaWMQzDCCAm7oZhGAHExN0wDCOAmLgbhmEEEBN3w/AhJ0+e5M4772T58uWsXLmSv/qrvwKgubmZ9evXU1paClAqItOSXS8iW0Sk1n1syaLpRpa4qriLyFMiclZE3h3kvIjID0XkmIi8IyJr0m+mMVy2bt3K7NmzWbVqVdLzbrGWBcnGzwTAu+Tn5/P973+f9957jzfeeIPHHnuMmpoaKioqWLduHbW1tQAXge39rxWR6cA3gY8CtwDfHOxDwPAvqXjuPwbuucL5TwKl7uNB4G9HbpaRLu6//372798/6Pl9+/YBjKPf+JkAeJvi4mLWrHE+h4uKili+fDmNjY3s3buXLVv6PocvAPcluXwDcEBVm1U1BBzgyu9xw4dcdZ27qv6LiCy+Qpd7gX9QxwV8Q0SmikixqjalycZAEYspjS1d1Dd30twRpqMnQntPhEhMiami6njTMYWY+5MRlUKcQ/OJBi60h/mfvz7KzKKx/OFti/vO7t27F+BC//EDPo4rAAAiEheA3SMwxlc0tXax/93TtHT2AtA3Ch4rTdl8uoF/+e2bPP30Rzlz5gzFxcXxU73AwiSXzAdOJjxvcNsuQ0QexPnAZ+HCZLcxMsmxs+08X93IF25dxOzJ44Z8fTo2MQ32Qhkg7qP5xfJuYytP/uY4rxw92ycWqSIyst/d23KGCx09/PUrx1g2p+gycW9sbAQIJ3SPj19KAuDYF7xx7QpHue+x1znT1pP0/EjHJF3Ewl2c/sl2pv2H/8zkyZNTvSyZ9QM+sVT1ceBxgLKyMm99oo0CXqs9xw9fPsbmjw7vPZUOcU/phQKj88USiynfffEoP/rnDygam8/dK+dStngai6ZPYFbRWCaOzWdiYT6F+WMQcURjjIj7AEmDipw4cYJPv/Y/efd/fGrAuUEKpCujfFxfPHKaM209PHV/GR+/djZjxnhEzRPo7e3l05/+NH/8//wXvvrVrwIwZ84cmpqa4t57AXA2yaUNON/M4pQAr2bWWmOoVNaFmDdlHMVTxg/r+nSIewOwIOF5CXAqDfcNBP9j33v8/WvH2XzLAr5evpzJ4wpybdJllJSUABQmNuGM36gWgGNn28kbI/yH0lmeFHZV5YEHHmD58uV9wg6wceNGdu3axfbt2wFmAD9NcvmLwF8kzKHcDXw90zYbQ6OqLsSaRcOf5krHUsjngT90V83cCrRavN3hN7Xn+fvXjvPFWxfxF79/veeEHRwxAGYkGb8XgbtFZJorAne7baOCExc6KJk2noI8b64Wfv3113n66ad5+eWXWb16NatXr+aFF15g+/btHDhwIL4UcjJQASAiZSLyBIA7j/Id4E33sSM+t2J4g6bWLk61dnPzCMT9qp67iOzG8eBmikgDzgqKAgBV/RHwAlAOHAM6gf88bGsCRCymfPv/O8I1MyfyjU8tT0t4ZThs3ryZV199lfPnz1NSUsK3v/1tenudmP9DDz1EeXk5QA/9xk9Vm0UkLgAwygSgvrmTRTMm5tqMQbn99tsHC6lx8OBBAETk/fiYqeph4EvxPqr6FPBU5i01hkNVXQsAaxZmUNxVdfNVziuwbdgWBJTXjp2n9mw7P/j8jYwryMuZHbt3X3lxi/uhU6+qZf3PjWYBaOvqZbGHxd0INpV1IcYVjGHFvJQnyQfgze+cAeB/v1HHzElj+dT183JtijEMOsNRJhTm7kPZGN1U1oe4oWTqiMKCJu4ZoCsc5V/eP8enrp9LYb79F/uRrnCU8SbuRg7o7o1ypLF1RPF2MHHPCP/6wXl6IjHWLZ+Ta1OMYaCqdPaa527khncaWonElJtHEG8HE/eM8OrRc0wszOOjS6bn2hRjGPREYkRjyoTCnBUqM0YxlXUhAG5aOHVE9zFxzwBvnQyxeuFUxuab5+dHusJRAMbncCLcGL1U1Ye4ZuZEZkwaO6L7mLinme7eKP/edJEbS0b2qWvkjs5eR9wtLGNkG1V1Ni+NMCQDJu5pp6apjUhMucHE3bd0hSMATBhrYRkju9Rd6ORCR3jEk6lg4p523j7pbD64ccGUHFtiDJdONywzwcIyRpaJx9tN3D1I7dl2powvYO4wUnQa3qBP3C0sY2SZyvoQRWPzKZ09acT3MnFPM8fPdbBk1sScpRswRk44EgOwPQpG1qmqcxZjpCNZnb1608zx8x1cM9O2rfuZaMzJ2ZLv0aRhRjC52N3L0TMX0xKSARP3tNLRE+F0WzdLTNx9TSQu7h5M9WsEl+qTLaimJ94OJu5p5cSFDgCumTnyeJmRO6IxJyyTZ+JuZJHKuhAisHpBelbambinkePn4+JunrufMc/dyAWVdSGWzSmiKE11H2whbxppDHUBsGD68MpiGd4gHnP3sue+detWfvWrXzF79mzeffddAD7/+c9z9OhRAFpaWgBWJLtWRE4AF4EoEEmW7tnILrGYUl3fwn9cnb4ssua5p5Gm1m4mjc1P2ydvuti/fz/Lli1j6dKlVFRUJOuyQESq3cf7ItISPyEi0YRzz2fP6twRicY9d+++Pe6//372799/WdvPfvYzqqurqa6u5jOf+QxA6Aq3uFNVV5uwe4Pas+1c7ImMOFlYIua5p5Ezbd3MneKt9e3RaJRt27Zx4MABSkpKWLt2LRs3bmTFisucupPxN7mI/FfgpoRzXaq6Ops255o+zz3Pu577HXfcwYkTJ5KeU1WeeeYZgFFTOcvvpHPzUhzvuiY+pKm123Oblw4dOsTSpUtZsmQJhYWFbNq0ib17917pks3Alcs3BRy/x9xfe+015syZA075xGQo8GsRqRSRB7NnmTEYlXUhZkwsZNGMCWm7p4l7GvGi597Y2MiCBQv6npeUlNDY2Ji0r4gsAq4BXk5oHicih0XkDRG5L6PGegS/r5bZvXs3mzdfsTrmx1R1DfBJYJuI3JGsk4g86I794XPnzmXCVMOlqj7EmkXT0rr50cQ9TURjytmLPZ7z3JMVUb7CC2gT8AtVjSa0LXRDNn8A/KWIfCTJ/QIlAn723CORCL/85S/5/Oc/P2gfVT3l/jwLPAvcMki/x1W1TFXLZs2alRF7DbjQ3sPx8x1pDcmAiXvaON/eQzSmnvPcS0pKOHnyZN/zhoYG5s0bdEZ+E/1CMglC8CHwKpfH4+N9AiUCflgtMxgvvfQS1113HSUlJUnPi8hEESmKHwN3A+9m0USjH1X1zvqFdKT5TcTEPU00tXYDUOwxcV+7di21tbUcP36ccDjMnj172Lhx44B+IrIMmAb8NqFtmoiMdY9nAh8DarJkes645Ll79+2xefNmbrvtNo4ePUpJSQlPPvkkAHv27BkQkhGReSLygvt0DvAbEXkbOAT8k6pevuzGyCpV9SHyxwg3lKQ3k6ytlkkTZ9occZ9d5C1xz8/PZ+fOnWzYsIFoNMrWrVtZuXIlDz/8MGVlZYlCvxnYo5fHcZYDfyciMRxHoEJVAy/ufvDcd+9OPuf94x//eECb++2r3D3+ELgxg6YZQ6SyLsTK+VMYl+YU0ybuaaK5IwzAjEmFObZkIOXl5ZSXl1/WtmPHjsueq+q3+l+nqv8KXJ9J27zIpXXu3hV3Ixj0RmO8fbKF//TRRWm/t3e/d/qMuLhPn+g9cTeGRjQWQ4S0pF01jCtRc6qNnkgs7ZOpYOKeNpo7wkwozEv7Vysj+0Rial67kRXim5fWLEp/WU4T9zTR3BE2rz0gRGPq6Xi7ERyq6kPMmzKO4inpz0dl4p4mLnSEmWHiHggcz93eGkbmqapzNi9lAnsFp4mQee6BwTx3IxucauniVGt3RuLtYOKeNpo7wkwzcQ8EkVjMYu5GxqmqT3+ysERSEncRuUdEjorIMRHZnuT8QhF5RUTeEpF3RKQ82X2CzIWOHgvLBATz3I1sUFkXYlzBGJYXT87I/a8q7iKSBzyGk2RoBbBZRPoXAfjvwDOqehPOFva/SbehXqYzHKG7N8b0iWNzbYqRBiJRE3cj81TVhbihZCoFGSrEnspdbwGOqeqHqhoG9gD39uujQPzjZwpwKn0mep++DUzmuQcC89yNTNPdG+XIqbaMhWQgtR2q84GTCc8bgI/26/MtnPzQ/xWYCNyVFut8QqijF4CpE7xVgckYHrbO3cg07zS0EolpWisv9ScVzz3Zq7x/HtnNwI9VtQQnh8XTIjLg3kFLDRunrdsR9ynjTdyDgHnuRqa5tHkpt+LeACxIeF7CwLDLA8AzAKr6W2AcMLP/jYKWGjZOW5cj7pNN3ANB1Na5Gxmmsi7EkpkTM7p8OpVX8JtAqYhcIyKFOBOm/Qsl1wPrAERkOY64B8c1vwpxz93EPRhEzHM3Moiq9lVeyiRXFXdVjQBfAV4E3sNZFXNERHaISDxf7H8DvuzmiN4N3N8vdWygaeuKADB5nCXZDALRWIx8DxfHNvxN3YVOmjvCaS/O0Z+U1EhVXwBe6Nf2cMJxDU4hh1FJW3cvYwQmFpq4B4GoXrEUoWGMiHi8PZMrZcB2qKaFtq5eisYVWIrYgKCqeN1x37p1K7Nnz2bVqlV9bd/61reYP38+q1evZvXq1eAsSx7A1TYlGpmlsj5E0dh8SmdPyujvMXFPA23dESaP967Xvn//fpYtW8bSpUupqKhI1mWGiJwTkWr38aX4CRHZIiK17mNL9qzOHTFVxnjcc7///vvZv39gdbw//dM/pbq6murqaoDW/udT3JRoZJCquhA3LZqWcWfQxD0NXOzupWisNydTo9Eo27ZtY9++fdTU1LB7925qapJWyvuZqq52H08AiMh04Js4+xpuAb4pIpn9LukBojHvi/sdd9zB9OnTh3NpKpsSjQzR1t3L0TMXM7q+PY6Jexpo6/Ku537o0CGWLl3KkiVLKCwsZNOmTezduzfVyzcAB1S1WVVDwAHgnowZ6xFiCh7X9kHZuXMnN9xwA1u3bgVIVjkm2abE+dmwzYDq+hZUM1Ocoz8m7mmgrbuXyeO86bk3NjayYMGlbQolJSU0NjYm6/oZN+nbL0QkfsGoFAJVfy6F/KM/+iM++OADqqurKS4uhsv3p8RJZVOi0zGgmw5zSVV9CBFYvcDE3Re0dfV6do17shWpSVaCtACLVfUG4CVgV7xrslsmuV+gRCCmeD4sk4w5c+aQl5fHmDFj+PKXvwxOKpD+pLIpEQjupsNcUlkXYtmcIoqy4AyauKeBtu6IZz33kpISTp685Hw3NDQwb968/t2iqtrjHv89cHO8OykIQdBEIKbqy7BMU1NT3/Gzzz4L0JWkWyqbEo0MEI0p1fUtGV8CGcebgWIfEYnGaO/xbsx97dq11NbWcvz4cebPn8+ePXv46U9/2r9b4ifTRpzNauBsXPuLhEnUu4GvZ9bi3OMHz33z5s28+uqrnD9/npKSEr797W/z6quvUl1djYiwePFicENqIjIPeEJVy1U1IiLxTYl5wFOqeiRnf8goovbsRS72REzc/UJ7T3x3qjc99/z8fHbu3MmGDRuIRqNs3bqVlStX8vDDD1NWVsbGjRsBZovIESACNAP3A6hqs4h8B8fbA9ihqs25+Duyiari9ZD77t27B7Q98MADlz0XkV4AVT2Fk9AP9/mATYlG5snW5qU4Ju4jpC/1gEdj7gDl5eWUl19eHGvHjh2JTxtVtSzZtar6FPBU5qzzHn5Y5274j8q6EDMmFrJw+oSs/D6LuY+QvqRhllcmMMRiln7ASD9v1bewZtG0rL22TNxHiKX7DR4xH4RlDH9xob2H4+c7shaSARP3ERP33IvMcw8M6oMJVcNfVNW3ANmLt4OJ+4hp74kCMGmsiXtQiKlitTqMdFJZF6IgT7h+ftJcbhnBXsIjpDPsTKhONHEPDM46d/PcjfRRVRdixbwpjCtIlhEiM5i4j5D4UkjL5R4cLCxjpJPeaIy3G1qykiwsERP3EdLRE2GMwLgC+68MCjahaqSTmlNt9ERiWY23g4n7iOnoiTJxbL59jQ8QftihaviH+OalbGSCTMTEfYR09ERsMjVg+DW3jOFNKutDzJ86nuIp47P6e03cR0hHOGKTqQHDYu5GOqmqC7EmyyEZMHEfMe09USYWZm8G3Mg8FnM30sWpli6aWrtZszC7IRkwcR8xHT3muQcNyy1jpIuq+uwmC0vExH2EmLgHD6fMnom7MXIq60KMKxjD8uLJWf/dJu4jpCNsE6pBww8pfw1/UFUX4saSqRTkZV9qTdxHiLMU0mLuQcKWQhrpoCsc5ciptpyEZMDEfcS090Rsd2rA8MOE6tatW5k9ezarVq3qa/va177Gddddxw033MDv//7vg1NpaQAickJE/k1EqkXkcJZMHnW809BCJKYm7n6kNxojHIl5Pua+f/9+li1bxtKlS6moqEjWZY6I1IjIOyJyUEQWxU+ISNQVgWoRGRW1NmMx7+eWuf/++9m/f/9lbevXr+fdd9/lnXfe4dprrwWYe4Vb3Kmqqwcr0mKMnHgmyJuynHYgjon7COh0M0J6Wdyj0Sjbtm1j37591NTUsHv3bmpqavp36wTKVPUG4BfAdxPOdbkisFpVN2bL7lzih3Xud9xxB9OnT7+s7e677yY/33kt3nrrrQCF2bfMiFNZF2LJzIlMn5ibYTBxHwHtbkbISR6OuR86dIilS5eyZMkSCgsL2bRpE3v37u3f7aKqdrrHbwAl2bXSW/ghLHM1nnrqKYDWQU4r8GsRqRSRB7Nn1ehBVamqz83mpTgm7iOgw80IOcHDMffGxkYWLFjQ97ykpITGxsYrXfIAsC/h+TgROSwib4jIfckuEJEH3T6Hz507lxa7c0lMYYyP1f2RRx6Je/CDFTP/mKquAT4JbBORO5J1Ctq4ZpMTFzpp7gjnLN4OJu4jIi7uXl4KqaoD2gaLJ4vIF4Ay4HsJzQvduOwfAH8pIh9J8jseV9UyVS2bNWtWWuzOJX7OLbNr1y5+9atf8ZOf/GTQPqp6yv15FngWuGWQfoEa12wSTxbmeXEXkXtE5KiIHBOR7YP0+Zw7KXdERH6aXjO9SYcPYu4lJSWcPHmy73lDQwPz5s0b0E9E7gK+AWxU1Z54e4IQfAi8CtyUYZNzjh9i7snYv38/jz76KM8//zwTJkxI2kdEJopIUfwYuBt4N4tmjgoq60IUjctn6axJObPhquIuInnAYzhf4VYAm0VkRb8+pcDXcb7urQT+JAO2eo6+Qh0ejrmvXbuW2tpajh8/TjgcZs+ePWzcOGBedDzwdzjCfjbeKCLTRGSsezwT+BgwYDY2aPgh5r5582Zuu+02jh49SklJCU8++SRf+cpXuHjxIuvXr2f16tUACwFEZJ6IvOBeOgf4jYi8DRwC/klV9yf/LcZweas+xE0Lp+U0vJeKy3kLcMz13BCRPcC9XP4m/zLwmKqGoO/rXuDxQ1gmPz+fnTt3smHDBqLRKFu3bmXlypU8/PDDlJWVxYV+AdAF/NwN2dS7K2OWA38nIjEcR6BCVUeJuHtb3Xfv3j2g7YEHHrjsuYjUQ9+3r3L3+EPgxsxbOHpp6+7l6JmLfHJVcU7tSEWV5gMnE543AB/t1+daABF5HWfjxLeSeQPuzPyDAAsXLhyOvZ4iXj/VyxOqAOXl5ZSXl1/WtmPHjsSn7ydb76yq/wpcn1nrvIflljFGQnV9C6q5jbdDajH3ZK/y/rN0+UAp8HFgM/CEiAzIcRm0CZp2N+buZc/dGBrxCWivh2UM71JZF2KMwI0LpuTUjlTEvQHna3ucEuBUkj57VbVXVY8DR3HEPtBY/dTgEXPdFq+HZQzvUlUfYtncyRSNK8ipHamo0ptAqYhcIyKFwCag/zb054A7oW/i7Vrgw3Qa6kXa3XS/9hU+OMTMczdGQDSmVNe35KQ4R3+uKu6qGgG+ArwIvAc8o6pHRGSHiMSXXbwIXBCRGuAV4GuqeiFTRnuFTkv3Gzji4m4f2MZwqD17kYs9kZzH2yG1CVVU9QXghX5tDyccK/BV9zFq6OiJMsFK7AUKtbCMMQK8sHkpjgWLR0CnFccOHBaWMUZCZV2ImZMKWTg9+SaybGLiPgI6wua5Bw2bUDVGQlVdiDULp3kirGfiPgI6w1aoI2hcirnn2BDDd5xv7+HEhc6cZoJMxMR9BHT2RJlgYZlAoTHnp3nuxlB5yy3O4YV4O5i4j4iOcISJFpYJFBZzN4ZLZV2Igjzh+vm53bwUx8R9BHT2RD2fesAYGn3ibupuDJGquhAr501hXIE3HD4T92Giqo7n7uGMkMbQiU+oemFCzPAP4UiMtxtaPBOSARP3YdMTiRFT7ycNM4aG5ZYxhkNNUxs9kZiJexDo8EEud2Po2FJIYzhUuZuX1iw0cfc9nWEnI+R4j8TXjPQQNc/dGAaV9SHmTx3P3Cnjcm1KHybuwyQu7n7Yobp//36WLVvG0qVLqaioSNZFRORnbhnF34nI4oQTX3fbj4rIhmzZnCtiMX/kltm6dSuzZ89m1apVfW3Nzc2sX7+e0tJS1q9fD05thQGIyBYRqXUfW7JkcqCpqgt5Zn17HBP3YdLRV6jD2557NBpl27Zt7Nu3j5qaGnbv3k1NzYBiSjOBkKouBX4APArgllPcBKwE7gH+xi27GFj8klvm/vvvZ//+y+vhVFRUsG7dOmpra1m3bh3A3P7Xich04Js4BXduAb4pIt5SJZ9xqqWLptZubvZAJshETNyHSacPimMDHDp0iKVLl7JkyRIKCwvZtGkTe/fu7d9tKrDLPf4FsE4c1/VeYI+q9rh5+o/hCEJgiS+FzPP4O+OOO+5g+vTpl7Xt3buXLVscR9z9mUy0NwAHVLXZLYt5AOeD2xgml5KFTb9Kz+zi8Zewd/GL597Y2MiCBZdqrZSUlNDY2Ni/WyFuKUU3xXMrMIPkJRbnZ9LeXHNpE5O3PfdknDlzhuJip26n+zOZ5zHqxjTTVNaFGF+Qx3XFRbk25TJM3IdJvH6q13PLxJf2JZJiPFlJrcQiIvKgiBwWkcPnzp0bso1eYhSsc09pTCFY45pJ3qoPcUPJFAo89nXPW9b4iA43LDPB40shS0pKOHnykqPW0NDAvHnz+ncL45ZSFJF8YArQTGolFgNVG9fP69znzJlDU1MTQPxnJEm3lMYUgjWumaIrHOXIqTZPrW+PY+I+TPziua9du5ba2lqOHz9OOBxmz549bNy4sX+3FiC+auKzwMtuAZbngU0iMlZErsGpi3soa8bnAD+vc9+4cSO7djlTJ+7PliTdXgTuFpFp7kTq3W6bMQzeaWghElMT9yDhl3Xu+fn57Ny5kw0bNrB8+XI+97nPsXLlSh5++GGef76vFO55YIaIHMOpprUdQFWPAM8ANcB+YJuqRnPwZ2QNvyQO27x5M7fddhtHjx6lpKSEJ598ku3bt3PgwAFKS0s5cOAAQBOAiJSJyBMAqtoMfAenNvKbwA63zRgGlfXOZOpNHtq8FMfbbqeH6QxHGV+Q54sEU+Xl5ZSXl1/WtmPHjsSnqqr/V7JrVfUR4JHMWect/FJDdffu3UnbDx482HcsIlEAVT0MfCnerqpPAU9l1sLRQVVdiCWzJjJ9YmGuTRmAee7DpKPHkoYFEb+sczdyj6pSVd/iqZQDiZi4D5POsKX7DSJ+CcsYuefEhU6aO8KejLeDifuw6eiJeH6NuzF0/DyhamSXS5uXTNwDRWc46vndqcbQsRqqRqpU1oUoGpfP0lmTcm1KUkzch0lH2Dz3IKI+3qFqZJequhBrFk7z7KIKE/dh0tkT9fwad2PoWFjGSIW27l7eP3vRsyEZMHEfNh3hiOd3pxpDJ57y16POmOERqutbUPVWcY7+mLgPk65w1MIyAWQU5JYx0kBlXYgxAjcumJJrUwbFxH2YdIQjFpYJIH7OLWNkj6r6EMvmTqZoXEGuTRkUE/dhEI0p3b0xW+ceQPpi7qbuxiBEY8pb9S3cvMhbxTn6Y+I+DPqShlnMPXDYJibjarx/5iLtPRFPT6ZCiuIuIve4NTSPicj2K/T7rIioiJSlz0TvEU8aZp578PBLbhkjd8Q3L3l5MhVSEHe3ZuZjwCeBFcBmt7Zm/35FwP8N/C7dRnqNjh7z3IOK5ZYxrkZVfYiZkwpZOH1Crk25Iql47rcAx1T1Q1UNA3twamv25zvAd4HuNNrnScxzDy4WljGuRnzzkte/3aUi7letuSgiNwELVPVXabTNs/R57rYUMnDYJibjSpxv7+HEhU7Px9shNXG/Ys1FERkD/AD4b1e9UUBqMvZ57pZbJnBYbhnjSlR5PFlYIqmI+9VqLhYBq4BXReQEcCvwfLJJ1aDUZLwUlvG2597c3Mz69espLS1l/fr1hEKhAX2qq6sBrhORIyLyjoh8Pn5ORH4sIsdFpNp9rM6i+TnB77lljh49yurVq1m9ejXAChFpE5E/SewjIh8XkdaEcX04N9b6j8r6EAV5wqr53t28FCcVcX8TKBWRa0SkENiEU1sTAFVtVdWZqrpYVRcDbwAb3eovgaTDXQrpdXGvqKhg3bp11NbWsm7dOioqKgb0mTBhAsBxVV0J3AP8pYgkLuD9mqqudh/V2bE8d/g9LLNs2TKqq6vjH9o1QCfwbJKuryWM644k540kvFXXwsrlEhKWAAAXgklEQVR5Uxjn8fKakIK4q2oE+ApOEd33gGdU9YiI7BCRAZWWRwOdPf4ojr137162bHHqXm/ZsoXnnntuQJ9rr70WoAdAVU8BZwH/fq0aIQGbUJ0MfKCqdbk2JAiEIzHebmjxRUgGUqyhqqovAC/0a0v6VU5VPz5ys7xNR1/M3duf3mfOnKG4uBiA4uJizp49e8X+InILUAh8kND8iPu1/SCwXVV7MmSuJwhYbpnpwA8HOXebiLyNE2L9M7cY+mWIyIPAgwALFy7MmJF+oaapjZ5ILFjiblxOZzhC/hihMC/3G3zvuusuTp8+PaD9kUeGVtNaRIqBp4Etqhpzm78OnMYR/MeBPwcGfIUPkggEJbdMOBwGmAL8PMnpKmCRqraLSDnwHFDav5OqPo4z7pSVlWn/86MNr1de6o+J+zDo6HEyQnrBu3vppZcGPTdnzhyampooLi6mqamJ2bNnD9Z1DPBPwH9X1Tfijara5B72iMj/Av4s2cVBEoGYzydU4+zbtw+gU1XP9D+nqm0Jxy+IyN+IyExVPZ9NG/1GVV2I+VPHM2fyuFybkhK5dz19SGc44osSexs3bmTXrl0A7Nq1i3vvHbj3zPXwlgL/oKqXeXmuN484n2L3Ae9m2OScE3O/s/hd3Hfv3g3QnOyciMx1xzQeihsDXMiedf6ksi7EGp947WDiPiw6wlHGe3ylDMD27ds5cOAApaWlHDhwgO3bnbRAhw8f5ktf+hIAzzzzDMAk4P4kSx5/IiL/BvwbMBP4f7P+R2SZIKxz7+zs5MCBAwAt8TYReUhEHnKffhZ41425/xDYpPF4lJGUUy1dnG7r5uaF3s4EmYj33U8P0tETocgHnvuMGTM4ePDggPaysjKeeOIJAL7whS/wxS9+sUpVk+1L+ETmrfQWGoCUvxMmTODChQuISDTepqo/SjjeCezMiXE+5VK8fXqOLUkd89yHQXu3P8IyxtCJBmRC1UgvlXUhxhfkcV1xUa5NSRkT92HQ3hNhkol7IIm6ayHz/ByXMdJOVX2IGxdMocADK+RSxT+Weoj2ngiTxpm4BxG1fO5GP7rCUWpOtflmCWQcE/dh0O6TmLsxdPo8d4vLGC7vNLQQianni3P0x8R9iKgq7d3muQeVqDuhamEZI05lvTOZepOJe7DpicSIxJRJY71b9dwYPn07VO2dYbhU1YVYMmsi0ycW5tqUIWEv4SFysdtJGmaeezCJh2X8vonJSA+qSmVdiJt95rWDifuQaXczQlrMPZjEl0JazN0AOH6+g1Bnr+8mU8HEfci0xz13E/dAYgWyjUT8liwsERP3IXKxpxfANjEFlEthmRwbYniCqvoWJo/L5yOzJuXalCFj4j5E4p57kcXcA4kthTQSqaoLcdPCab5MR2HiPkTiJfYsLBNMVBUR28RkQGtXL++fvejLkAyYuA+ZdlstE2iiqhZvNwCoPtmCqj/j7WDiPmQu9pjnHmSiMdvAZDhU1oUYI3DjAv+k+U3ExH2ItHdHKMgTxubbf10QianaBiYDgLfqQyybO9m3jpy9jIdIPCOkxWSDSSxmYRnDmVh/q76Fmxf502sHE/ch46e8Ms3Nzaxfv57S0lLWr19PKBQarOvNCVWYno83isg1IvI7EakVkZ+JiL/2Xw+DqKrvwzKLFy/m+uuvB1ghIof7nxeHH4rIMRF5R0TWZN9Kb/P+mYu090R8G28HE/chc7EnwsRCf4h7RUUF69ato7a2lnXr1lFRUTFY15iqrnYfGxPaHwV+oKqlQAh4INM255pYTH257K0/r7zyCkBNsgpbwCeBUvfxIPC3WTTNF/RtXlron8pL/TFxHyLt3RHfrHHfu3cvW7ZsAWDLli0899xzKV/rFlD+BPALt2kXTpHsQBPTUbHG/V6cguiqqm8AU+PF0A2HqroQMyeNZcH08bk2ZdiYuA8RP1VhOnPmDMXFznu2uLiYs2fPDtZ1jIgcFpE3RCQu4DOAFlWNuM8bgPnJLhaRB93rD587dy6Nf0H2cZZC5tqKkSEi3H333QDLReTBJF3mAycTnicd2yCN61CprA9x86Kpvp5b84dKeYjWrl4+Mmtirs3o46677uL06dMD2h955JGh3OYdVS0TkSXAyyLyb0Bbkn6a7GJVfRx4HKCsrCxpH78QhAnV119/nXnz5iEitcA2Efl3Vf2XhC7J/sAB4xakcR0K59t7qLvQyR/csjDXpowIE/ch0trVy5Tx3snl/tJLLw16bs6cOTQ1NVFcXExTUxOzZ88erGsvgKp+KCKvAjcB/4jzdT3f9d5LgFNpNd6DxFR9H5aZN29e/DAC/Aq4BUgU9wZgQcLzUTG2qVLl42RhiVhYZgjEYkpbt7fE/Ups3LiRXbt2AbBr1y7uvffeAX3cFTQCICIzgY/hTMQp8ArwWbfrFmBv5q3OLdGYvzNCdnR0cPHixfjTMcDdwLv9uj0P/KG7auZWoFVVm7JopqeprA9RkCesmj8l16aMCBP3IXCxO4IqTJngjxWB27dv58CBA5SWlnLgwAG2b98OwOHDh/nSl74EwHvvvQdObPZtHDGvUNUa9xZ/DnxVRI7hxOCfzPbfkG38vonpzJkz3H777dx4440Ay4F/UtX9IvKQiDzkdnsB+BA4Bvw98F9yY603qaoLsWr+FMYV5OXalBFhYZkh0NrlpPv1i+c+Y8YMDh48OKC9rKyMJ554AoDf+73fg0GWzKnqhzhf6UcNMZ+vc1+yZAlvv/02ACJyRFUfAVDVH8X7uN/KtuXGQm8TjsR4u6GVP7x1Ua5NGTE+9lGyT0tXGPCPuBtDJxqACVVj+Bw51Uo4EmONz+PtkKK4i8g9InLU3dG2Pcn5r4pIjbvb7aCI+P9jLwl+89yNoeOEZUzcRytV9S2A/ydTIQVxF5E84DGcXW0rgM0isqJft7eAMlW9AWfTy3fTbagXiIv71Akm7kElGvN3WMYYGVV1IeZPHc+cyeNybcqIScVzvwU4pqofqmoY2IOzw60PVX1FVTvdp2/gLK0KHC2d5rkHnZiCafvoRFU5XNccCK8dUhP3lHazJfAAsC/ZCb/veLOwTPCJxfy/zt0YHqdauznT1jOqxD2l3WwAIvIFoAz4XrLzqvq4qpapatmsWbNSt9IjtHX1Upg/xvdLpIzBiQZgE5MxPCoDsnkpTipLIVPazSYidwHfAP4PVe1Jj3neorWrl6nmtQeamPp7E5MxfKrqQowvyOO6uUW5NiUtpOK5vwmUurm9C4FNODvc+hCRm4C/Azaq6qDZqfxOS6d/dqcaw8PJLZNrK4xcUFUf4sYFU8jPC8YK8av+FW5eka8ALwLvAc+o6hER2SEi8dzf3wMmAT/vX/AhSDR3hpk20R+7U43hEbWY+6ikMxzhyKm2wIRkIMUdqqr6As6W5cS2hxOO70qzXZ7kQnsP182dnGszjAwSU9vENBp5p6GVaEwDJe7B+P6RJc63h5kxyTz3IGPiPjqJT6betMDEfdQRjsRo7eplxsSxuTbFyCAWlhmdVNWFWDJrYqDCribuKRLqdPLKmOcebGKKpR8YZagqVfUhbl4YHK8dTNxT5ny7s7pzpol7oIkFoMyeMTSOn+8g1NkbqHg7mLinzIV2x3OfOcnCMkHGcsuMPoK2eSmOiXuKXOhwPPcZJu6BJhrzd1bIkydPcuedd7J8+XKAlSLyx/37iMjHRaTVXbZcLSIPD7zT6KGqPsTkcfl8ZNakXJuSVkzcU+T8Rf/F3Jubm1m/fj2lpaWsX78+XlLvMl555RWAFQlv9G4RuQ9ARH4sIscTzq3O8p+QdVTxdVgmPz+f73//+/EKW+/hFMjun8UV4DVVXe0+dmTXSm9RWRdizaJpvv5QT4aJe4qc7+ihMG8MRWP9U7yqoqKCdevWUVtby7p166ioqBjQ58477wSnEtNq4BNAJ/DrhC5fSxCB6qwYnkP8nlumuLiYNWvWxJ/GcAT+Son+RjWtXb3Unm1nTcAmU8HEPWXOtfUwc1Ih4qN47N69e9myZQsAW7Zs4bnnnrvaJZ8F9iWkbx51OEshA/O2KARuAn6X5NxtIvK2iOwTkZVZtsszVJ9sQTV48XYwcU+ZptZuiqeOz7UZQ+LMmTMUFxcDjkd39uxV0/5sAnb3a3vErbD1AxFJOuHg91TOifRGYxT42HOP097eDvAR4E9Uta3f6SpgkareCPw1kPRTP0jjOhiVdSHGCNy4YGquTUk7Ju4p0tTaRfEU71Vnueuuu1i1atWAx969e4d0HxEpBq7HySEU5+vAdcBaYDrw58mu9Xsq50QiUSU/z9/i3tvby2c+8xmAZlX9Zf/zqtqmqu3u8QtAgYjMTNIvMOM6GFV1Ia6bO5lJPgq3pkrw/qIMoKo0tXZz98q5uTZlAC+99NKg5+bMmUNTUxPFxcU0NTUxe/bsK93qc8Czqtobb1DVJvewR0T+F/Bn6bDZy0RiMV9nBVRVHnjgAZYvX86vf/3rM8n6iMhc4IyqqojcguPkXciqoR4gGlPeqg/xf64JZOE489xTobkjTE8k5knP/Ups3LiRXbt2AbBr1y7uvffeK3XfTL+QjOvNI85Ew33Au5mx1DuEI/4Oy7z++us8/fTTvPzyy3BpFVS5iDwkIg+53T4LvCsibwM/BDapatICPEHm6OmLdISjgYy3g3nuKdHU2g3gO3Hfvn07n/vc53jyySdZuHAhP//5zwE4fPgwP/rRj3jiiSfiXQuBYuCf+93iJyIyC6caVzXwEAEnElMKfOy533777cR1WkRqVLWsfx9V3QnszLZtXqOq3lkaHMSVMmDinhINoS4A5k+dkGNLhsaMGTM4ePDggPaysrJEYQcIq+qA5XKq+okMmudJnJi7f8XdSJ2quhAzJ41lwXR/LZRIFXsVp0DdhQ4AFs30l7gbQ6c3FqPA5xOqRmpU1oe4edFUXy1vHgom7ilw4kIHMyYWMnmcldgLMtGYogr5wVnnbgzCuYs91F3oDGy8HUzcU+LE+U4WzTCvPej0RmMAvl8KaVydeLzdxH2Uc+JCB4tnTsy1GUaGicSciUgLywSfqvoQBXnCynlTcm1KxjBxvwqtXb00tXYHLmOcMZBwxPHc/bxaxkiNqroQq+ZPYVxBXq5NyRj2Kr4K/97k7NxeMc8KYwednkgUINBveMP5EH+7oTVwlZf6Y+J+Fd6Li3uxiXvQ6e51PPdxBfa2CDJHTrUSjsQCHW8HE/ercuRUG9MnFjK7yIp0BJ3uXtdzzzfPPcjEKy+tMXEf3bx5opk1C6cFdi2scYm4uI81zz3QVNWHKJk2njmT/bXjfKjYq/gKnL3YzYkLndxyTbA/4Q2HHndC1Tz34KKqTuWlgMfbwcT9ivzrMSdR3kevmZFjS4xscMlzN3EPKqdauznT1hP4eDuYuF+RF4+cZnbRWK6fH9y1sMYl2nsiABSNs5RLQSUebzdxH8W0dffy6tFzrF8xJ3CFc43ktHWZuAedqroQ4wvyuG5uUa5NyTgm7oPwy8oGunqjfH7tglybYmSJi91OnZIiyyEUWCrrQqxeMHVUZP4M/l84DDrDEf72nz/g5kXTuKEkeLUVjeS0dvWSP0aYWGgx9yDSGY5Q09TGmkWj4z2dkriLyD0iclREjonI9iTnx4rIz9zzvxORxek2NJt8+/kazrT18PVPXpdrU0bEz3/+c1auXMmYMWM4fPjwlbpOTja+InKNO5617vgWZt7q3HG6rZs5k8f5ftnr/v37WbZsGcCq0fB+TZV3GlqJxnRUxNshBXEXkTzgMeCTwApgs4is6NftASCkqkuBHwCPptvQbNDdG+Vbzx/hZ4dP8pU7l1K2eHquTRoRq1at4pe//CV33HHHoH2i0SjAQpKP76PAD1S1FAjhjHNgaWj2ZhH0oRCNRtm2bRv79u0DOEKA369DJT6ZetOC0SHuqcwc3QIcU9UPAURkD3AvUJPQ517gW+7xL4CdIiJer8vY3RvlfHsPH57r4M0TzfxjZQOnWrvZ+rFr+Or6a3Nt3ohZvnz5VfscOnQIoKf/+IrIe8AngD9wu+7CGeO/zYStuaa7N8qRU62+L5Z86NAhli5dypIlSwAUCMz7daRU1YX4yKyJTJsY6C+gfaQi7vOBkwnPG4CPDtZHVSMi0grMAM6nYsT7Zy7yR/+7EgVQ5xWpqu5Pp4/iFFLoe55wvu9cX//E5wn93GMUoqp0hqN9NojAxz4yk+9+9kZuL52ZitmBoLGxESCc0BQf3xlAi6pGEtoHlOK7Et29UT7917/pq+mpff9cGpuEpr6x7DvuJzV945esb0J7/Dfo1X5XQltPJEY4EuM/3jhvKH+i52hsbGTBgssWAaT9/QrwxSd/x2m3trBfqLvQyX03+Xt8h0Iq4p4sANn/Ez6VPojIg8CDAAsXLuxrH5efx3VzJ4M4NxIR9+elG8fbnD7Sd04Sn7sdkp5z74HbPkaEaRMK3BqKE7i+ZIovKy3dddddnD59ekD7I488wr333nvV6wdx1pQUxxQGH9cxIiyb4y45k0s/+saB+JglGWe3Mf5MLrs+fiyXjuXSXa7aN+F3Xfob4Pc+MpNbrvF3KO4K45nIiN6vAItnTPTdktFr5xbxxVsX59qMrJHK6DQAia5ACXBqkD4NIpIPTAGa+99IVR8HHgcoKyvrezEtnDGBx/7TmqFZbgDw0ksvjej6kpISgMTvqfHxPQ9MFZF813tPNu7A4ONamD/GxjXLlJSUcPLkycuaSPP7FeA7961Ko9VGJkhltcybQKm7cqIQ2AQ836/P88AW9/izwMtBj98FhbVr1wKM6z++7vi9gjOe4Izv3txYaaTK2rVrqa2t5fjx4+B46PZ+HaVcVdxdr+0rwIvAe8AzqnpERHaIyEa325PADBE5BnwVGLD8ysg+zz77LCUlJfz2t7/lU5/6FBs2bADg1KlTlJeXA5Cfnw9QT7/xdW/x58BX3XGdgTPOhofJz89n586d8bFeib1fRy2Sqw/ssrIyvcraayNLiEilqpal4142rt7BxjWYpDqutkPVMAwjgJi4G4ZhBBATd8MwjABi4m4YhhFATNwNwzACSM5Wy4jIOaCuX/NMhrAF2gP4zV5IbvMiVZ2VjpsHZFzBfzbbuKZGEGxOaVxzJu7JEJHD6Vq6lQ38Zi/kxmb7f8o8Nq6pMZpstrCMYRhGADFxNwzDCCBeE/fHc23AEPGbvZAbm+3/KfPYuKbGqLHZUzF3wzAMIz14zXM3DMMw0kDOxV1Evici/y4i74jIsyIyNeHc190ivkdFZEMu7ezP1YqGewERWSAir4jIeyJyRET+2G2fLiIH3MLXB0Qk7UUlbVwzh43r0BmV46qqOX0AdwP57vGjwKPu8QrgbWAscA3wAZCXa3td2/Jce5bgFLp4G1iRa7uS2FkMrHGPi4D33f/X7wLb3fbt8f9zG1cbVxvX4Ixrzj13Vf21XqrT+QZO5RhwivjuUdUeVT0OHMMp1u0F+oqGq2qYS0WIPYWqNqlqlXt8ESdf+3wcW3e53XYB92Xgd9u4Zggb1yEzKsc15+Lej63APvc4WWHuIRVoziBeti0pIrIYuAn4HTBHVZvAeUEBszP8621cM4SNa0p42bakpGNcs1LhVkReAuYmOfUNVd3r9vkGEAF+Er8sSX+vLO3xsm0DEJFJwD8Cf6KqbYmFoUd4XxvXHGLjmjJetm0A6RrXrIi7qt51pfMisgX4NLBO3cASqRXmzhVetu0yRKQA54XyE1X9pdt8RkSKVbVJRIqBs8O5t41r7rBxHRJetu0y0jquHphEuAeoAWb1a1/J5RM0H+KdCZp8155ruDRBszLXdiWxU4B/AP6yX/v3uHyC5rs2rjauNq45tzOt4+qFP+gYTjys2n38KOHcN3BmuY8Cn8y1rf3sLseZzf4A5+tqzm1KYuPtOF8/30n4/y3HKXZ9EKh1f063cbVxtXHNuY1pHVfboWoYhhFAvLZaxjAMw0gDJu6GYRgBxMTdMAwjgJi4G4ZhBBATd8MwjABi4m4YhhFATNwNwzACiIm7YRhGAPn/AYK9fD5rDUhtAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f37f8297da0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "fig, ax = plt.subplots(1,3)\n",
    "ax=ax.flatten()\n",
    "\n",
    "pts=np.arange(-20,20, 0.1)\n",
    "\n",
    "# Sigmoid\n",
    "ax[0].plot(pts, 1/(1+np.exp(-pts))) ;\n",
    "\n",
    "# tanh\n",
    "ax[1].plot(pts, np.tanh(pts*np.pi)) ;\n",
    "\n",
    "# Rectified linear unit (ReLu)\n",
    "pts_relu=[max(0,i) for i in pts];\n",
    "ax[2].plot(pts, pts_relu) ;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-layer preceptron neural network\n",
    "Universal function theorem\n",
    "\n",
    "#### Gradient based learning\n",
    "\n",
    "#### back propogation\n",
    "\n",
    "#### loss function\n",
    "\n",
    "#### optimization\n",
    "\n",
    "epochs\n",
    "\n",
    "### Google Playground\n",
    "\n",
    "https://playground.tensorflow.org/\n",
    "\n",
    "<img src=\"./images/neuralnets/google_playground.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is **Keras**?\n",
    "\n",
    "* It is a high level API to create and work with neural networks\n",
    "* Supports multiple backends such as TensorFlow and Keras\n",
    "* Very good for creating neural nets very quickly and hides awy a lot of tedious work\n",
    "* Has been incorporated into official TensorFlow (which obviously only works with tensforflow)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_7 (Dense)              (None, 4)                 20        \n",
      "_________________________________________________________________\n",
      "activation_7 (Activation)    (None, 4)                 0         \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 1)                 5         \n",
      "_________________________________________________________________\n",
      "activation_8 (Activation)    (None, 1)                 0         \n",
      "=================================================================\n",
      "Total params: 25\n",
      "Trainable params: 25\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Say hello to keras\n",
    "\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation\n",
    "\n",
    "# Creating a model\n",
    "model = Sequential()\n",
    "\n",
    "# Adding layers to this model\n",
    "# 1st Hidden layer\n",
    "model.add(Dense(units=4, input_dim=4))\n",
    "model.add(Activation(\"relu\"))\n",
    "\n",
    "# The output layer\n",
    "model.add(Dense(units=1))\n",
    "model.add(Activation(\"sigmoid\"))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fittind the model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network results on dataset used in previous notebooks\n",
    "\n",
    "## Network Architecture\n",
    "\n",
    "## CNN examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}