{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    \n",
       "    @import url('http://fonts.googleapis.com/css?family=Source+Code+Pro');\n",
       "    \n",
       "    @import url('http://fonts.googleapis.com/css?family=Kameron');\n",
       "    @import url('http://fonts.googleapis.com/css?family=Crimson+Text');\n",
       "    \n",
       "    @import url('http://fonts.googleapis.com/css?family=Lato');\n",
       "    @import url('http://fonts.googleapis.com/css?family=Source+Sans+Pro');\n",
       "    \n",
       "    @import url('http://fonts.googleapis.com/css?family=Lora'); \n",
       "\n",
       "    \n",
       "    body {\n",
       "        font-family: 'Lora', Consolas, sans-serif;\n",
       "       \n",
       "        -webkit-print-color-adjust: exact important !;\n",
       "        \n",
       "      \n",
       "       \n",
       "    }\n",
       "    \n",
       "    .alert-block {\n",
       "        width: 95%;\n",
       "        margin: auto;\n",
       "    }\n",
       "    \n",
       "    .rendered_html code\n",
       "    {\n",
       "        color: black;\n",
       "        background: #eaf0ff;\n",
       "        background: #f5f5f5; \n",
       "        padding: 1pt;\n",
       "        font-family:  'Source Code Pro', Consolas, monocco, monospace;\n",
       "    }\n",
       "    \n",
       "    p {\n",
       "      line-height: 140%;\n",
       "    }\n",
       "    \n",
       "    strong code {\n",
       "        background: red;\n",
       "    }\n",
       "    \n",
       "    .rendered_html strong code\n",
       "    {\n",
       "        background: #f5f5f5;\n",
       "    }\n",
       "    \n",
       "    .CodeMirror pre {\n",
       "    font-family: 'Source Code Pro', monocco, Consolas, monocco, monospace;\n",
       "    }\n",
       "    \n",
       "    .cm-s-ipython span.cm-keyword {\n",
       "        font-weight: normal;\n",
       "     }\n",
       "     \n",
       "     strong {\n",
       "         background: #f5f5f5;\n",
       "         margin-top: 4pt;\n",
       "         margin-bottom: 4pt;\n",
       "         padding: 2pt;\n",
       "         border: 0.5px solid #a0a0a0;\n",
       "         font-weight: bold;\n",
       "         color: darkred;\n",
       "     }\n",
       "     \n",
       "    \n",
       "    div #notebook {\n",
       "        # font-size: 10pt; \n",
       "        line-height: 145%;\n",
       "        }\n",
       "        \n",
       "    li {\n",
       "        line-height: 145%;\n",
       "    }\n",
       "\n",
       "    div.output_area pre {\n",
       "        background: #fff9d8 !important;\n",
       "        padding: 5pt;\n",
       "       \n",
       "       -webkit-print-color-adjust: exact; \n",
       "        \n",
       "    }\n",
       " \n",
       "    \n",
       " \n",
       "    h1, h2, h3, h4 {\n",
       "        font-family: Kameron, arial;\n",
       "\n",
       "\n",
       "    }\n",
       "    \n",
       "    div#maintoolbar {display: none !important;}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# IGNORE THIS CELL WHICH CUSTOMIZES LAYOUT AND STYLING OF THE NOTEBOOK !\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib as mpl\n",
    "mpl.rcParams['lines.linewidth'] = 3\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "%config IPCompleter.greedy=True\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore', category=FutureWarning)\n",
    "from IPython.core.display import HTML; HTML(open(\"custom.html\", \"r\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Neural Networks\n",
    "\n",
    "\n",
    "## History of Neural networks\n",
    "\n",
    "<div class=\"alert alert-block alert-danger\"><p>\n",
    "    <strong>TODO</strong>: Make it more complete and format properly\n",
    "</p></div>\n",
    "\n",
    "1943 - Threshold Logic\n",
    "\n",
    "1940s - Hebbian Learning\n",
    "\n",
    "1958 - Perceptron\n",
    "\n",
    "1975 - Backpropagation\n",
    "\n",
    "1980s - Neocognitron\n",
    "\n",
    "1982 - Hopfield Network\n",
    "\n",
    "1986 - Convolutional Neural Networks\n",
    "\n",
    "1997 - Long-short term memory (LSTM) model\n",
    "\n",
    "2014 - Gated Recurrent Units, Generative Adversarial Networks(Check)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feed-Forward neural network\n",
    "<center>\n",
    "<figure>\n",
    "<img src=\"./images/neuralnets/neural_net_ex.svg\" width=\"700\"/>\n",
    "<figcaption>A 3 layer densely connected Neural Network (By convention the input layer is not counted).</figcaption>\n",
    "</figure>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why the boom now?\n",
    "* Data\n",
    "* Data\n",
    "* Data\n",
    "* Availability of GPUs\n",
    "* Algorithmic developments which allow for efficient training and making networks networks\n",
    "* Development of high-level libraries/APIs have made the field much more accessible than it was a decade ago"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building blocks\n",
    "### Perceptron\n",
    "\n",
    "The smallest unit of a neural network is a **perceptron** like node.\n",
    "\n",
    "**What is a Perceptron?**\n",
    "\n",
    "It is a simple function which can have multiple inputs and has a single output.\n",
    "\n",
    "<center>\n",
    "<figure>\n",
    "<img src=\"./images/neuralnets/perceptron_ex.svg\" width=\"400\"/>\n",
    "<figcaption>A simple perceptron with 3 inputs and 1 output.</figcaption>\n",
    "</figure>\n",
    "</center>\n",
    "\n",
    "\n",
    "It works as follows: \n",
    "\n",
    "Step 1: A **weighted sum** of the inputs is calculated\n",
    "\n",
    "\\begin{equation*}\n",
    "weighted\\_sum = w_{1} x_{1} + w_{2} x_{2} + w_{3} x_{3} + ...\n",
    "\\end{equation*}\n",
    "\n",
    "Step 2: A **step** activation function is applied\n",
    "\n",
    "$$\n",
    "f(weighted\\_sum) = \\left\\{\n",
    "        \\begin{array}{ll}\n",
    "            0 & \\quad weighted\\_sum < threshold \\\\\n",
    "            1 & \\quad weighted\\_sum \\geq threshold\n",
    "        \\end{array}\n",
    "    \\right.\n",
    "$$\n",
    "\n",
    "You can see that this is also a linear classifier as the ones we introduced in script 02."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hidecode"
    ]
   },
   "outputs": [],
   "source": [
    "# Plotting the step function\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "x = np.arange(-2,2.1,0.01)\n",
    "y = np.zeros(len(x))\n",
    "threshold = 0.\n",
    "y[x>threshold] = 1.\n",
    "step_plot = sns.lineplot(x, y).set_title('Step function') ;\n",
    "plt.xlabel('weighted_sum') ;\n",
    "plt.ylabel('f(weighted_sum)') ;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "def perceptron(X, w, threshold=1):\n",
    "    # This function computes sum(w_i*x_i) and\n",
    "    # applies a perceptron activation\n",
    "    linear_sum = np.dot(np.asarray(X).T, w)\n",
    "    output = np.zeros(len(linear_sum), dtype=np.int8)\n",
    "    output[linear_sum >= threshold] = 1\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Boolean AND\n",
    "\n",
    "| x$_1$ | x$_2$ | output |\n",
    "| --- | --- | --- |\n",
    "| 0 | 0 | 0 |\n",
    "| 1 | 0 | 0 |\n",
    "| 0 | 1 | 0 |\n",
    "| 1 | 1 | 1 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculating Boolean AND using a perceptron\n",
    "threshold = 1.5\n",
    "# (w1, w2)\n",
    "w = [1, 1]\n",
    "# (x1, x2) pairs\n",
    "x1 = [0, 1, 0, 1]\n",
    "x2 = [0, 0, 1, 1]\n",
    "# Calling the perceptron function\n",
    "output = perceptron([x1, x2], w, threshold)\n",
    "for i in range(len(output)):\n",
    "    print(\"Perceptron output for x1, x2 = \", x1[i], \",\", x2[i],\n",
    "          \" is \", output[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this simple case we can rewrite our equation to $x_2 = ...... $ which describes a line in 2D:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def perceptron_DB(x1, x2, w, threshold):\n",
    "    # Plotting the decision boundary of the perceptron\n",
    "    sns.scatterplot(x1, x2)\n",
    "    plt.xlim(-1,2)\n",
    "    plt.ylim(-1,2)\n",
    "    # The decision boundary is a line given by\n",
    "    # w_1*x_1+w_2*x_2-threshold=0\n",
    "    x1 = np.arange(-3, 4)\n",
    "    x2 = (threshold - x1*w[0])/w[1]\n",
    "    sns.lineplot(x1, x2, **{\"color\": \"black\"})\n",
    "    plt.xlabel(\"x$_1$\", fontsize=16)\n",
    "    plt.ylabel(\"x$_2$\", fontsize=16)\n",
    "    # Coloring the regions\n",
    "    pts_tmp = np.arange(-2, 2.1, 0.02)\n",
    "    points = np.array(np.meshgrid(pts_tmp, pts_tmp)).T.reshape(-1, 2)\n",
    "    outputs = perceptron(points.T, w, threshold)\n",
    "    plt.plot(points[:, 0][outputs == 0], points[:, 1][outputs == 0],\n",
    "             \"o\",\n",
    "             color=\"steelblue\",\n",
    "             markersize=1,\n",
    "             alpha=0.04,\n",
    "             )\n",
    "    plt.plot(points[:, 0][outputs == 1], points[:, 1][outputs == 1],\n",
    "             \"o\",\n",
    "             color=\"chocolate\",\n",
    "             markersize=1,\n",
    "             alpha=0.04,\n",
    "             )\n",
    "    plt.title(\"Blue color = 0 and Chocolate = 1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plotting the perceptron decision boundary\n",
    "perceptron_DB(x1, x2, w, threshold)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1 : Compute a Boolean \"OR\" using a perceptron?**\n",
    "\n",
    "Hint: copy the code from the \"AND\" example and edit the weights and/or threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Boolean OR\n",
    "\n",
    "| x$_1$ | x$_2$ | output |\n",
    "| --- | --- | --- |\n",
    "| 0 | 0 | 0 |\n",
    "| 1 | 0 | 1 |\n",
    "| 0 | 1 | 1 |\n",
    "| 1 | 1 | 1 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculating Boolean OR using a perceptron\n",
    "# Edit the code below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution\n",
    "# Calculating Boolean OR using a perceptron\n",
    "threshold=0.6\n",
    "# (w1, w2)\n",
    "w=[1,1]\n",
    "# (x1, x2) pairs\n",
    "x1 = [0, 1, 0, 1]\n",
    "x2 = [0, 0, 1, 1]\n",
    "output = perceptron([x1, x2], w, threshold)\n",
    "for i in range(len(output)):\n",
    "    print(\"Perceptron output for x1, x2 = \", x1[i], \",\", x2[i],\n",
    "          \" is \", output[i])\n",
    "perceptron_DB(x1, x2, w, threshold)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 2 : Create a NAND gate using a perceptron**\n",
    "\n",
    "#### Boolean NAND\n",
    "\n",
    "| x$_1$ | x$_2$ | output |\n",
    "| --- | --- | --- |\n",
    "| 0 | 0 | 1 |\n",
    "| 1 | 0 | 1 |\n",
    "| 0 | 1 | 1 |\n",
    "| 1 | 1 | 0 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculating Boolean NAND using a perceptron\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution\n",
    "# Calculating Boolean NAND using a perceptron\n",
    "import matplotlib.pyplot as plt\n",
    "threshold=-1.5\n",
    "# (w1, w2)\n",
    "w=[-1,-1]\n",
    "# (x1, x2) pairs\n",
    "x1 = [0, 1, 0, 1]\n",
    "x2 = [0, 0, 1, 1]\n",
    "output = perceptron([x1, x2], w, threshold)\n",
    "for i in range(len(output)):\n",
    "    print(\"Perceptron output for x1, x2 = \", x1[i], \",\", x2[i],\n",
    "          \" is \", output[i])\n",
    "perceptron_DB(x1, x2, w, threshold)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, a single perceptron can compute \"AND\", \"OR\" and \"NOT\" boolean functions.\n",
    "\n",
    "However, it cannot compute some other boolean functions such as \"XOR\".\n",
    "\n",
    "**WHAT CAN WE DO?**\n",
    "\n",
    "\n",
    "Hint: Think about what is the significance of the NAND gate we have created above?\n",
    "\n",
    "Answer: We said a single perceptron can't compute a \"XOR\" function. We didn't say that about **multiple Perceptrons** put together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**XOR function using multiple perceptrons**\n",
    "\n",
    "<center>\n",
    "<figure>\n",
    "<img src=\"./images/neuralnets/perceptron_XOR.svg\" width=\"400\"/>\n",
    "<figcaption>Multiple perceptrons connected together to output a XOR function.</figcaption>\n",
    "</figure>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-layer perceptrons\n",
    "\n",
    "The normal densely connected neural network is sometimes also called \"Multi-layer\" perceptron."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Google Playground\n",
    "\n",
    "A great tool from Google to develop a feeling about neural networks.\n",
    "\n",
    "https://playground.tensorflow.org/\n",
    "\n",
    "<img src=\"./images/neuralnets/google_playground.png\"/>\n",
    "\n",
    "Some concepts to look at:\n",
    "\n",
    "* Effect of activation functions\n",
    "* Effect of network size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning\n",
    "\n",
    "Now we know that we can compute complex functions by combining a number of perceptrons.\n",
    "\n",
    "In the perceptron examples we had set the model parameters (weights and thresholds) by hand.\n",
    "\n",
    "This is something we definitely **DO NOT** want to do or even can do for big networks.\n",
    "\n",
    "We want some algorithm to set the weights for us!\n",
    "\n",
    "This is achieved by choosing an appropriate loss function for the problem at hand and solving an optimization problem.\n",
    "We will explain below what this means.\n",
    "\n",
    "\n",
    "### Loss function\n",
    "\n",
    "To learn using an algorithm we need to define a quantity/function which allows us to measure how close or far are the predictions of our network/setup from reality or the supplied labels. This is done by choosing a so-called \"Loss function\" (as in the case for other machine learning algorithms).\n",
    "\n",
    "Once we have this function, we need an algorithm to update the weights of the network such that this loss function decreases. \n",
    "As one can already imagine the choice of an appropriate loss function is critical to the success of the model. \n",
    "\n",
    "Fortunately, for classification and regression (which cover a large variety of problems) these loss functions are well known. \n",
    "\n",
    "Generally **crossentropy** and **mean squared error** loss functions are used for classification and regression problems, respectively.\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <i class=\"fa fa-info-circle\"></i>&nbsp; <strong>mean squared error</strong> is defined as \n",
    "\n",
    "\n",
    "$$\n",
    "\\frac{1}{n} \\left((y_1 - \\hat{y}_1)^2 + (y_2 - \\hat{y}_2)^2 + ... + (y_n - \\hat{y}_n)^2 \\right)\n",
    "$$\n",
    "\n",
    "\n",
    "</div>\n",
    "\n",
    "### Gradient based learning\n",
    "\n",
    "As mentioned above, once we have chosen a loss function, we want to solve an **optimization problem** which minimizes this loss by updating the weights of the network. This is how the learning takes in a NN, and the \"knowledge\" is stored in the weights.\n",
    "\n",
    "The most popular optimization methods used in Neural Network training are **Gradient-descent (GD)** type methods, such as gradient-descent itself, RMSprop and Adam. \n",
    "\n",
    "**Gradient-descent** uses partial derivatives of the loss function with respect to the network weights and a learning rate to updates the weights such that the loss function decreases and after some iterations reaches its (Global) minimum value.\n",
    "\n",
    "First, the loss function and its derivative are computed at the output node, and this signal is propagated backwards, using the chain rule, in the network to compute the partial derivatives. Hence, this method is called **Backpropagation**.\n",
    "\n",
    "One way to perform a single GD pass is to compute the partial derivatives using **all the samples** in our data, computing average derivatives and using them to update the weights. This is called **Batch gradient descent**. However, in deep learning we mostly work with massive datasets and using batch gradient descent can make the training very slow!\n",
    "\n",
    "The other extreme is to randomly shuffle the dataset and advance a pass of GD with the gradients computed using only **one sample** at a time. This is called **Stochastic gradient descent**.\n",
    "\n",
    "<center>\n",
    "<figure>\n",
    "<img src=\"stochastic-vs-batch-gradient-descent.png\" width=\"600\"/>\n",
    "<figcaption>Source: <a href=\"https://wikidocs.net/3413\">https://wikidocs.net/3413</a></figcaption>\n",
    "</figure>\n",
    "</center>\n",
    "\n",
    "\n",
    "In practice, an approach in-between these two is used. The entire dataset is divided into **m batches** and these are used one by one to compute the derivatives and apply GD. This technique is called **Mini-batch gradient descent**. \n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "<p><i class=\"fa fa-warning\"></i>&nbsp;\n",
    "One pass through the entire training dataset is called 1 epoch of training.\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "\n",
    "plt.figure(figsize=(10, 4)) ;\n",
    "\n",
    "pts=np.arange(-20,20, 0.1) ;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Activation Functions\n",
    "\n",
    "In order to train the network we need to move away from Perceptron's **step** activation function because it does not allow training using the gradient-descent and back-propagation algorithms among other drawbacks.\n",
    "\n",
    "Non-Linear functions such as:\n",
    "\n",
    "* Sigmoid\n",
    "\n",
    "\\begin{equation*}\n",
    "f(z) = \\frac{1}{1+e^{-z}}\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAusAAAH0CAYAAACEkWPuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcHXWd7//35/Sa7k539oWEJCRkY5WEPYIExoiOCzpuV0VkdO44MgMuc2e8w8wozqj3N9cFQZzx6gUGHcXtoVwdxCiERSIDhB2yNnR20p3O1vt2vr8/qjp0uqt6rVNV55zX8/HoR3V/v3WqP10Up9/59re+Zc45AQAAAEifTNIFAAAAAAhGWAcAAABSirAOAAAApBRhHQAAAEgpwjoAAACQUoR1AAAAIKUI6wAAAEBKEdYBAACAlCKsAwAAAClFWAcAAABSirAOAAAApBRhHQAAAEgpwjoAAACQUoR1AAAAIKUI6wAAAEBKEdYBAACAlCpNuoA4mdkrkmolNSRcCgAAAArbIknHnHOnTOQgRRXWJdVOmjRp2sqVK6clXQgAAAAK1+bNm9XR0THh4xRbWG9YuXLltE2bNiVdBwAAAArY6tWr9dRTTzVM9DjMWQcAAABSirAOAAAApBRhHQAAAEgpwjoAAACQUoR1AAAAIKUI6wAAAEBKEdYBAACAlCKsAwAAAClFWAcAAABSirAOAAAApBRhHQAAAEipSMK6mb3bzG41s0fM7JiZOTP7/jiPNd/MbjezfWbWZWYNZnazmU2NolYAAAAgX5RGdJy/l3S2pFZJeyStGM9BzGyJpI2SZkm6R9IWSedLukHSlWa2xjnXHEnFAAAAQMpFNQ3mU5KWSaqV9BcTOM635AX1651zVznnPuucu1zS1yUtl/TFCVcKAAAA5IlIRtadcxv6PzezcR3DzBZLWiepQdJtg7o/J+m/S7razD7jnGsbX6UAABQO55x6s07dvVnvoy+rrp6suvv61OW39WWd9+GcslmpN5tV1jn1ZaW+rPM/f2372udS1jk57xvJeZvj37f/a+d//VpNkpM73je4bWDtr73e69eA75GEBL91oj94oj93zFYtmKq1K2YlXcaYRDUNJgqX+9v1zrnswA7nXIuZPSovzF8o6f64iwMAIErZrNPRjh4dau/W4bZuHWrr1uH2bh1q69GR9m61dvWqratXrV19auvqVVt37/G2jm4/jPdlEw23QL65ds0iwvoELPe320L6t8sL68s0Qlg3s00hXeOaSw8AwFg559TU2qUdB1q161C79h3p0N4jndp3pEP7jnZo/5FOdfdlRz4QgKKWprBe52+PhvT3t0+JoRYAAEatL+u0vbFFT+08ouf3HtWOxhZtb2zVkfaepEsDkOfSFNZH0j8ZfsQ/+DnnVgcewBtxXxVlUQCA4tPbl9XTu4/okW1N2rTrsJ7dfVStXb1JlwWgAKUprPePnNeF9NcO2g8AgNi0dfXqd5sP6L4XXtXvdxxUSyfhHEDupSmsb/W3y0L6l/rbsDntAABEKpt1emh7k3765B7dv+WAOnvSN8c8Y1JFaYnKSzPeR0lGFf2fl2ZUmjGVZEwZ87b9n5dmTJmMqcRv9z7XCW1mJjPvT9ve1vta6m/zvujve+1zf2t2/M/iGvD6/jYb3NZ/vFjOXLBxLmoXzfdO8CdP8ueO09kn599s6jSF9f7lH9eZWWbgijBmNlnSGkkdkh5LojgAQPE42t6jHz25S99/bJd2HWrP2fepKi/RjJoKTa0u17SqMn9brqnV5ZpcWarq8lJVV5SqpqJU1RUlqqkoVVVFqarKSlRR5gXz0pKoHpkCII1iD+tmViZpiaQe51x9f7tzrt7M1stb8eU6SbcOeNlNkqolfZs11gEAuXK0vUff/f3LuuPRhkjmoJeVmBbPqNGSWdU6eVqV5k2ZpJPqJumkKZM0b8ok1U4qHffzSQAUh0jCupldJekq/8s5/vYiM7vT//ygc+6v/c/nSdosaaekRYMO9QlJGyXdYmZX+PtdIGmtvOkvN0ZRLwAAA3X29On/PPyyvvPwy2oZZ0ifVFais+bXadXCqTprXp2WzZmsBdOqVMbIN4AJiGpk/XWSrhnUttj/kLxg/tcagT+6fq6kL0i6UtJbJO2XdIukm5xzhyKqFwAASdKDWxv1uf/3onY2j226S3lpRhcunq5Ll87QhYuna8WcyUxJARC5SMK6c+7zkj4/yn0bNMy9I8653ZKujaIuAADCHGrr1j/84gX95/P7R/2aSWUlumLlLL31rJP0hmUzNam8JIcVAkC6bjAFACAWj73crBvufloHjnWNav8z5tXqwxcu0h+fNVfVFfzqBBAf3nEAAEWjL+t06wPbdcv925Ud8RF70hUrZukTa0/VqgVTuBEUQCII6wCAotDZ06dP3v2M7nvx1RH3vXTZTH3mjcvyck1mAIWFsA4AKHhH2rv1sX9/Uk/uPDzsfvOmTNLn3naa3njabEbSAaQCYR0AUND2HunQNbc/rh2NrcPu97HXn6JPr1umqnJ+NQJID96RAAAFq/FYpz7wnceGXZZxenW5vvres3XZ8lkxVgYAo0NYBwAUpCPt3frw7Y8PG9TPXThVt31wlWbXVsZYGQCMHmEdAFBw2rp6de2dT2jLqy2h+7zlzDn62ntfp8oy1koHkF6EdQBAQclmnf7yB0/p6V1HQvf5yMWL9A9vPU0lGW4iBZBuhHUAQEH5xv3btWFrU2j/X1y2RH/zpuWs9gIgL2SSLgAAgKhs2NKob9y/PbT/QxcuIKgDyCuEdQBAQdh9qF2f/NEzof1vP/skfeHtZxDUAeQVwjoAIO/19GX1if94Skc7egL7zz9lmr763rOVYY46gDxDWAcA5L1vP1Sv5/ceDeybNblC3/zAOSor4VcegPzDOxcAIK9tP9CiW+7fEdhXmjHd9sFVmjWZddQB5CfCOgAgb/Vlnf7mZ8+puy8b2P93b1mp8xZNi7kqAIgOYR0AkLfuePSV0PXUL1k6Q9euWRRvQQAQMcI6ACAv7T/aoa+s3xrYV1Veoi+980xWfgGQ9wjrAIC89NX129TZEzz95W+vXKGTp1XFXBEARI+wDgDIOy/tO6afPbUnsO+8RVN19YULY64IAHKDsA4AyDv/674tcm5oe2nG9OV3ncV66gAKBmEdAJBXHtnepIe3NQX2feCCBTp1Vk3MFQFA7hDWAQB5oy/r9KV7twT21VSU6vorlsZcEQDkFmEdAJA3fvPiq9q8/1hg38ffsFgzaipirggAcouwDgDIC845/dtD9YF9c2or9dHXL465IgDIPcI6ACAvbKxv1nN7jgb2ffqNyzSpvCTmigAg9wjrAIC8EDaqPm/KJL1z1byYqwGAeBDWAQCp98Leo3pk+8HAvj+75BSVlfDrDEBh4t0NAJB6/xoyqj61qkzvPe/kmKsBgPgQ1gEAqbazuU2/fn5/YN81Fy9SVXlpzBUBQHwI6wCAVPvB47uUDXha6aSyEl1z0aLY6wGAOBHWAQCp1d2b1U+f3BPY977zTtbU6vKYKwKAeBHWAQCp9duXDqi5rTuw70/XnBJzNQAQP8I6ACC17n5iV2D760+doQXTq2KuBgDiR1gHAKTSrub20OUa/9v5C2KuBgCSQVgHAKTSj54MHlWfXl2uN542O+ZqACAZhHUAQOr09GX145AbS9+9er7KS/n1BaA48G4HAEidDVsa1dTSFdj3Ph6CBKCIENYBAKlzz7P7AtsvOGWaFs+sibkaAEgOYR0AkCrt3b16YHNjYN/7z2dUHUBxIawDAFLl/s2N6ujpG9JeWZbRutPmJFARACSHsA4ASJVfPRc8BeaKFbNVXVEaczUAkCzCOgAgNVo6e7Rha1Ng39vOnhtzNQCQPMI6ACA1frf5gLp7s0Paq8tLdNnyWQlUBADJIqwDAFLjl8/uD2x/42mzVVlWEnM1AJA8wjoAIBWOtvfoke3BU2DeetZJMVcDAOlAWAcApMJvXnpVPX1uSHttZakuWTYjgYoAIHmEdQBAKvzupQOB7W86fY4qSpkCA6A4EdYBAInr7OnT73ccDOx7y1msAgOgeBHWAQCJ+69XDqm9e+iDkKrKS3TR4ukJVAQA6UBYBwAkbsOWxsD2NafOYBUYAEWNsA4ASJRzTvdvCZ6vfsUK1lYHUNwI6wCARO1obNXuQx2BfWsJ6wCKHGEdAJCoB0KmwJwxr1azaytjrgYA0oWwDgBI1P0hYf3yFbNjrgQA0oewDgBIzNH2Hm3aeTiw73KmwAAAYR0AkJyHtjepLzv0qaUzasp11ry6BCoCgHQhrAMAEvPQ1qbA9rXLZymTsZirAYD0IawDABLhnNMf6oOfWsoqMADgIawDABLR0NyufUc7h7SbSRcv4amlACAR1gEACdkYMqp+xkl1mlJVHnM1AJBOhHUAQCI21jcHtjOqDgCvIawDAGKXzTo9FhLWLyKsA8BxhHUAQOy2HmhRc1v3kPbSjOm8RdMSqAgA0imysG5m883sdjPbZ2ZdZtZgZjeb2dQxHuf1ZnaP//pOM9tlZvea2ZVR1QoASFbYFJhzFkxRdUVpzNUAQHpFEtbNbImkTZKulfS4pK9LelnSDZL+YGaj+pummf2FpEckXeFvvy7pIUlvkPRrM7sxinoBAMkKW7LxoiUzYq4EANItquGLb0maJel659yt/Y1m9jVJn5L0RUkfH+4AZlYm6cuSOiWtds5tHdD3JUlPS7rRzL7inOuKqG4AQMx6+7L6r5cPBfZxcykAnGjCI+tmtljSOkkNkm4b1P05SW2Srjaz6hEONU1SnaRtA4O6JDnnNkvaJmmSpJqJ1gwASM7ze4+qpat3SHtFaUbnLJiSQEUAkF5RTIO53N+ud85lB3Y451okPSqpStKFIxynUVKTpGVmtnRgh5ktk7RU0jPOueCJjgCAvBA2X/28RdNUUVoSczUAkG5RhPXl/nZbSP92f7tsuIM455yk6/yaNpnZv5vZl83sLnnz4V+U9J4I6gUAJOjxV0KmwJzKFBgAGCyKOet1/vZoSH9/+4h/23TO/cTM9kn6oaQPD+g6IOkOeTetjsjMNoV0rRjN6wEAuZHNOj2163Bg34WLCesAMFgc66ybv3Uj7mj2IUm/k7cSzEp502dWSrpf0jcl3Z2jGgEAMdjW2KKWzuD56mecVBfwCgAoblGMrPePnIe9y9YO2i+QPy/9dknPSbp6wPz3LWZ2tbzpNu8xs8uccw8Odyzn3OqQ77FJ0qrhXgsAyJ1NO4NH1c+eP0XlpTynDwAGi+KdsX/llrA56f03i4bNae+3TlKZpIcCblTNSnrY/zIwiAMA0m9TQ3BYX71oTM/PA4CiEUVY3+Bv15nZCcczs8mS1kjqkPTYCMep8LczQ/r724c+nxoAkBeeDBlZP3chYR0Agkw4rDvn6iWtl7RI3mouA90kqVrSXc65tv5GM1thZoNv9nzE377bzM4a2GFmr5P0bnnz3h+YaM0AgPg1tnRq16H2wL5VCwjrABAkqieYfkLSRkm3mNkVkjZLukDSWnnTX24ctP9mf9t/86mcc4+b2R2SrpX0hJn9XNJOef8IuEpSuaSbnXMvRlQzACBGT4WMqi+ZWa2p1eUxVwMA+SGSsO6cqzezcyV9QdKVkt4iab+kWyTd5JwLXlR3qI/Km5v+EUlvkjRZ0jFJv5f0Heccq8EAQJ56MmS++rkLp8VcCQDkj6hG1uWc2y1vVHw0+1pIu5N0p/8BACggm0LWV+fmUgAIxzpZAICc6+zp0wt7g1fwXc3NpQAQirAOAMi55/YcVU/f0GfjTasu1+IZ1QlUBAD5gbAOAMi5sIchrVowVWaBMyMBACKsAwBiEBbWmQIDAMMjrAMAcu65PUcC2wnrADA8wjoAIKdePdqpxpauIe0Zk86cV5dARQCQPwjrAICcChtVXzZ7siaVl8RcDQDkF8I6ACCnntsTvGTjWfMZVQeAkRDWAQA59WzIyPpZ86fEXAkA5B/COgAgZ5xzej7kYUhnE9YBYESEdQBAzuw+1KEj7T1D2stLMlo+Z3ICFQFAfiGsAwByJmwKzMq5k1Veyq8gABgJ75QAgJwJWwmG+eoAMDqEdQBAzrASDABMDGEdAJATfVmnF0JuLmVkHQBGh7AOAMiJl5ta1dbdN6S9qrxEp86qSaAiAMg/hHUAQE48GzIF5oyT6lSSsZirAYD8RFgHAOTE8yE3l57JfHUAGDXCOgAgJ54Lna9OWAeA0SKsAwAi15d12rK/JbCPm0sBYPQI6wCAyDU0t6mjZ+jNpdXlJVo4rSqBigAgPxHWAQCRe2nfscD2lXNrleHmUgAYNcI6ACByL+0PDuunnVQbcyUAkN8I6wCAyIWNrJ82l7AOAGNBWAcARI6RdQCIBmEdABCpxpZONbV0DWkvyZiWzZ6cQEUAkL8I6wCASIVNgVkys1qVZSUxVwMA+Y2wDgCIVOgUGOarA8CYEdYBAJEKG1k//SSeXAoAY0VYBwBEiptLASA6hHUAQGTau3v1ysG2wL6VTIMBgDEjrAMAIrPl1RY5N7R9bl2lplWXx18QAOQ5wjoAIDI8DAkAokVYBwBEhvnqABAtwjoAIDKMrANAtAjrAIBIZLNO2w60BPZxcykAjA9hHQAQiT2HO9Te3TekfVJZiRZMq0qgIgDIf4R1AEAktrwaPAVm2ewaZTIWczUAUBgI6wCASIRNgVk+Z3LMlQBA4SCsAwAiseXVsLDOfHUAGC/COgAgEqEj67MZWQeA8SKsAwAmrLs3q5eb2gL7mAYDAONHWAcATFh9U6t6s25I+7Tqcs2oKU+gIgAoDIR1AMCEDTcFxoyVYABgvAjrAIAJC7+5lCkwADARhHUAwIRtI6wDQE4Q1gEAE8bIOgDkBmEdADAhLZ092nukI7BvGcs2AsCEENYBABOy7UBrYPv8qZNUU1EaczUAUFgI6wCACdkaNgWGUXUAmDDCOgBgQkKXbWS+OgBMGGEdADAhW149FthOWAeAiSOsAwAmZEdj8Jx1wjoATBxhHQAwbkfau3WwtXtIe0nGdMqM6gQqAoDCQlgHAIxbfVPwqPqCaVWqKC2JuRoAKDyEdQDAuIVNgVkysybmSgCgMBHWAQDjVt/UFti+ZBZTYAAgCoR1AMC41TOyDgA5RVgHAIzbjpA566fOIqwDQBQI6wCAcens6dPuQ+2BfYysA0A0COsAgHFpaG5T1g1tnzm5QnWTyuIvCAAKEGEdADAu9Y0hN5fO5OZSAIgKYR0AMC5ha6wzBQYAokNYBwCMS9ga69xcCgDRiSysm9l8M7vdzPaZWZeZNZjZzWY2dRzHOtPM7jKz3f6xGs3sITP7cFT1AgAmhpF1AMi90igOYmZLJG2UNEvSPZK2SDpf0g2SrjSzNc655lEe6yOSviupXdKvJDVImiLpDElvkXRXFDUDAMYvm3WhYZ2RdQCITiRhXdK35AX1651zt/Y3mtnXJH1K0hclfXykg5jZhfKC+guSrnTOvTqon+UFACAF9h3tUGdPdkh7VXmJ5tRWJlARABSmCU+DMbPFktbJGwG/bVD35yS1SbrazEazPMC/SCqR9KHBQV2SnHM9E6sWABCF+qbglWAWz6xWJmMxVwMAhSuKkfXL/e1659wJwyzOuRYze1RemL9Q0v1hBzGz+ZIukfSkpBfNbK2k1ZKcpGckbRh8fABAMkJvLmW+OgBEKoqwvtzfbgvp3y4vrC/TMGFd0nkD9n9A0mWD+p83s3c553aMs04AQES4uRQA4hFFWK/zt0dD+vvbp4xwnFn+9r2SDkp6l7xwP1PedJqrJf2nmZ3pnOse7kBmtimka8UINQAARoFlGwEgHnGss94/eTHgodQnKBmw/Zhz7ufOuWPOuXpJ18ibHrNM0p/kpkwAwGi9HDayTlgHgEhFMbLeP3JeF9JfO2i/MIf9bZekewd2OOecmd0j6Vx5S0L+cLgDOedWB7X7I+6rRqgDADCMI+3dOtg69A+cGZMWTq9KoCIAKFxRjKxv9bfLQvqX+tuwOe2Dj9MSciNpf5ifNIbaAAARC5uvvnB6tSpKSwL7AADjE0VY3+Bv15nZCcczs8mS1kjqkPTYCMd5Tt5c9RlmNjug/wx/2zD+UgEAE1XfGLxs45KZo1mhFwAwFhMO6/6c8vWSFkm6blD3TZKqJd3lnDv+7m5mK8zshJs9nXO9kr7tf/kvA4O/mZ0p6SOSeiX9dKI1AwDGbwfz1QEgNlE9wfQTkjZKusXMrpC0WdIFktbKm/5y46D9N/vbwU/O+JKkKyR9WNKZZvagvNVg/kRSpaTPsHQjACSrPmQlGJZtBIDoRbIajD+6fq6kO+WF9M9IWiLpFkkXOeeaR3mcdnlh/SZJVfJG6t8u7x8Cb3HOfS2KegEA48ca6wAQn6hG1uWc2y3p2lHuG/osaj+wf97/AACkSGdPn3Ydag/s4+mlABC9ONZZBwAUiJ3N7coGPDVjRk2F6qrK4i8IAAocYR0AMGrhTy5lJRgAyAXCOgBg1JivDgDxIqwDAEaNsA4A8SKsAwBGLXwaDGEdAHKBsA4AGJVs1unlppCnlxLWASAnCOsAgFHZd7RDHT19Q9qryks0t7YygYoAoPAR1gEAo1IfMqq+eGa1MpnQx2cAACaAsA4AGJX6kPnq3FwKALlDWAcAjMqOkJVgeHIpAOQOYR0AMCqhI+vcXAoAOUNYBwCMStga6yzbCAC5Q1gHAIzoSHu3DrZ2D2nPmLRwelUCFQFAcSCsAwBGFLYSzMLp1aooLYm5GgAoHoR1AMCIwleCqY65EgAoLoR1AMCIwuars2wjAOQWYR0AMKIdrAQDAIkgrAMARsTIOgAkg7AOABhWV2+fdh1qD+zjgUgAkFuEdQDAsBoOtivrhrbPqKlQXVVZ/AUBQBEhrAMAhhU+BYaVYAAg1wjrAIBhhd1cypNLASD3COsAgGFxcykAJIewDgAYVlhYZ2QdAHKPsA4ACJXNOtU3tgX2scY6AOQeYR0AEGr/sU519PQNaZ9UVqK5tZUJVAQAxYWwDgAIFf7k0mplMhZzNQBQfAjrAIBQ9WFhnZtLASAWhHUAQKjQm0sJ6wAQC8I6ACBU+DQYwjoAxIGwDgAIVd8UshIMI+sAEAvCOgAg0NH2Hh1s7RrSnjFp0YyqBCoCgOJDWAcABNoRMl99wbQqVZSWxFwNABQnwjoAIBBPLgWA5BHWAQCBWLYRAJJHWAcABAobWSesA0B8COsAgEAs2wgAySOsAwCG6Ort065D7YF9PBAJAOJDWAcADLGzuV1ZN7R9Rk2F6qrK4i8IAIoUYR0AMEToFJiZ1TFXAgDFjbAOABgidCUY5qsDQKwI6wCAIcIeiMR8dQCIF2EdADBE6LKNjKwDQKwI6wCAE2SzTvWNbYF9PL0UAOJFWAcAnGD/sU519PQNaZ9UVqK5tZUJVAQAxYuwDgA4QdjNpYtnViuTsZirAYDiRlgHAJwgbNlGpsAAQPwI6wCAE4TeXMpKMAAQO8I6AOAEYWGdkXUAiB9hHQBwgh0hK8Ewsg4A8SOsAwCOO9reo4OtXUPaMyYtmlGVQEUAUNwI6wCA48KeXLpgWpUqSktirgYAQFgHABzHzaUAkC6EdQDAcdxcCgDpQlgHABwX9kAkRtYBIBmEdQDAcfVNISvBzKqOuRIAgERYBwD4unr7tLOZZRsBIE0I6wAASdLO5nZl3dD2GTXlmlJVHn9BAADCOgDAw3x1AEgfwjoAQJK0IyyssxIMACSGsA4AkBT+QCRG1gEgOYR1AICk8JF11lgHgOQQ1gEAymZd6AORlhLWASAxhHUAgPYe6VBnT3ZIe3V5iebWVSZQEQBAijCsm9l8M7vdzPaZWZeZNZjZzWY2dQLHvNTM+szMmdk/R1UrAOBE2xtbAttPnVUjM4u5GgBAv9IoDmJmSyRtlDRL0j2Stkg6X9INkq40szXOueYxHnOypH+X1C6Jv8ECQA5tPxA2X31yzJUAAAaKamT9W/KC+vXOuaucc591zl0u6euSlkv64jiO+Q1JdZK+HFGNAIAQYTeXLp3NWAkAJGnCYd3MFktaJ6lB0m2Duj8nqU3S1WZWPYZjvkPStZKul7RvojUCAIa3PWwlGJZtBIBERTGyfrm/Xe+cO+HuJOdci6RHJVVJunA0BzOzWZK+I+kXzrnvR1AfAGAYzrnQp5cysg4AyYoirC/3t9tC+rf722WjPN7/kVfXxydSFABgdA4c61JLV++Q9vLSjOZPrUqgIgBAvyhuMK3zt0dD+vvbp4x0IDP7U0nvkPQ+59yB8RZkZptCulaM95gAUKjCVoJZMrNGJRlWggGAJMWxznr/O70bdiezRZJulvQT59yPc1wTAMAXenMpD0MCgMRFMbLeP3JeF9JfO2i/MLdL6pD0iYkW5JxbHdTuj7ivmujxAaCQhN5cSlgHgMRFMbK+1d+GzUlf6m/D5rT3WyVv+ccm/yFIzsycpDv8/hv9tl9MrFwAwECMrANAekUxsr7B364zs8zAFWH8BxutkTdi/tgIx7lL3qoxgy2VdKmkZyRtkvT0hCsGABzHGusAkF4TDuvOuXozWy9vrfXrJN06oPsmSdWSvu2ca+tvNLMV/mu3DDjO9UHHN7OPyAvr/+mc+/uJ1gsAeE1za5cOtXUPaS/NmBZOH/XjMQAAORLFyLrkzTPfKOkWM7tC0mZJF0haK2/6y42D9t/sb1lmAAASFDaqvmhGtcpK4liDAAAwnEjeiZ1z9ZLOlXSnvJD+GUlLJN0i6SLnXHMU3wcAEC2eXAoA6RbVyLqcc7slXTvKfUc9ou6cu1PePwIAABFjvjoApBt/4wSAIhYW1lm2EQDSgbAOAEUs7OmlhHUASAfCOgAUqWOdPTpwrGtIu5m0hDnrAJAKhHUAKFJhU2AWTKtSZVlJzNUAAIIQ1gGgSIXOV2dUHQBSg7AOAEUqNKyzEgwApAZhHQCKVOiyjbMmx1wJACAMYR0AihQrwQBA+hHWAaAIdXT3ac/hjsA+wjoApAdhHQCKUH1Tq5wb2j63rlI1FZE93BoAMEGEdQAoQlteZQoMAOQDwjoAFKFtB4LD+oo53FwKAGlCWAeAIhQ2sr6BaH7GAAAdGElEQVR8Tm3MlQAAhkNYB4AitPXVY4Hty2czsg4AaUJYB4Aic6S9WweOdQ1pN5OW8kAkAEgVwjoAFJmtIVNgFk2vVmVZSczVAACGQ1gHgCITdnMpU2AAIH0I6wBQZMJuLl3GSjAAkDqEdQAoMmHTYFi2EQDSh7AOAEXEOaetYdNgCOsAkDqEdQAoIvuPdqqls3dIe3lpRgunVSVQEQBgOIR1ACgiYaPqS2fVqLSEXwkAkDa8MwNAEQmbr85KMACQToR1ACgioWGd+eoAkEqEdQAoImFhnWUbASCdCOsAUCR6+7La0dQa2MeyjQCQToR1ACgSDc1t6u7NDmmfXFmqObWVCVQEABgJYR0AisRL+8NvLjWzmKsBAIwGYR0AisRL+44Ftp92Um3MlQAARouwDgBF4qX9IWF9LmEdANKKsA4ARYKRdQDIP4R1ACgCjS2dOtjaNaS9JGNaxgORACC1COsAUATCRtVPnVmjyrKSmKsBAIwWYR0AikDofHWmwABAqhHWAaAIhM5X5+ZSAEg1wjoAFAFG1gEgPxHWAaDAtXf36pWDbYF9KxlZB4BUI6wDQIHb8mqLnBvaPreuUtOqy+MvCAAwaoR1AChwL4bMVz+dKTAAkHqEdQAocNxcCgD5i7AOAAWOm0sBIH8R1gGggPX2ZbUlLKzPrYu5GgDAWBHWAaCANTS3qas3O6R9ckWp5k+dlEBFAICxIKwDQAF7fu/RwPaVc2uVyVjM1QAAxoqwDgAF7NndwWH99HnMVweAfEBYB4ACFjayfvb8KTFXAgAYD8I6ABSo3r6sXtwXHNbPnM/NpQCQDwjrAFCgth1oVWdP8M2lp0yvTqAiAMBYEdYBoEA9t+dIYPuZ8+u4uRQA8gRhHQAK1HMh89XPYr46AOQNwjoAFKiwkfWzmK8OAHmDsA4ABaizp09b9rcE9hHWASB/ENYBoABt3n9MvVk3pH16dbnmTeHJpQCQLwjrAFCAwtZXP3N+ncy4uRQA8gVhHQAKUNiTS7m5FADyC2EdAApQ2M2lZzNfHQDyCmEdAApMW1evdjS1Bvbx5FIAyC+EdQAoMC/sPSo39N5Sza2r1KzJlfEXBAAYN8I6ABSYTbsOB7azZCMA5B/COgAUmE0NwWH9nAVTY64EADBRhHUAKCDOudCR9XMXEtYBIN8Q1gGggNQ3telIe8+Q9vKSjM6YxzQYAMg3hHUAKCCbdh4KbD9jXq0qy0pirgYAMFGEdQAoIJt2hkyBWTQt5koAAFGILKyb2Xwzu93M9plZl5k1mNnNZjaqSZJmVm1mHzSzH5jZFjNrM7MWM3vSzD5jZuVR1QoAherJkLC+iptLASAvlUZxEDNbImmjpFmS7pG0RdL5km6QdKWZrXHONY9wmEskfV/SIUkbJP1C0jRJb5P0FUnvMrMrnHOdUdQMAIXmUFu3Xm5qC+xbzc2lAJCXIgnrkr4lL6hf75y7tb/RzL4m6VOSvijp4yMc41VJH5L0E+dc94BjTJb0oKSLJV0n6asR1QwABSVsCsyi6VWaObki5moAAFGY8DQYM1ssaZ2kBkm3Der+nKQ2SVebWfVwx3HOPeOc+4+BQd1vb9FrAf2yidYLAIUqLKyvYlQdAPJWFHPWL/e3651z2YEdftB+VFKVpAsn8D361yHrncAxAKCgha0Ec+5Cbi4FgHwVRVhf7m+3hfRv97fLJvA9/tTf3jeBYwBAwerq7dOze44G9p27iJF1AMhXUcxZ73/KRvBvidfap4zn4Gb2l5KulPSMpNtH+ZpNIV0rxlMDAKTdC3uPqbs3O6S9trJUp86sSaAiAEAU4lhn3fytG/MLzd4l6WZ5N5/+iXNu6GP5AAB6/JXgKTCrFk5VJmOBfQCA9ItiZL1/5DzsOda1g/YbFTO7StLdkholrXXOvTza1zrnVoccc5OkVWOpAwDywcb6g4Ht5/EwJADIa1GMrG/1t2Fz0pf627A57UOY2Xsk/UTSAUlvcM5tHeElAFC0unr79ERD8Mj6RUumx1wNACBKUYT1Df52nZmdcDx/jfQ1kjokPTaag5nZByT9UNI+eUF9+wgvAYCi9syuI+rsGTpfvaaiVGfNC/ujJwAgH0w4rDvn6iWtl7RI3kOLBrpJUrWku5xzxx+rZ2YrzGzIzZ5mdo2k70naJenSsUx9AYBitbE++AHRF5wyTaUlcdyaBADIlaieYPoJSRsl3WJmV0jaLOkCSWvlTX+5cdD+m/3t8buezGytvNVeMvJG6681G3JT1BHn3M0R1QwABeEPIWGdKTAAkP8iCevOuXozO1fSF+Qts/gWSfsl3SLpJudc8GTKEy3UayP9fxqyz055q8MAACS1d/fq6d3BTy69eMmMmKsBAEQtqpF1Oed2S7p2lPsOGTJ3zt0p6c6o6gGAYvBEw2H19A1dGXdadblWzJmcQEUAgCgxmREA8ljYko0XLZ7O+uoAUAAI6wCQx5ivDgCFjbAOAHnqaHuPXtgb/Ly5iwnrAFAQCOsAkKcee6VZ2aHT1TWntlKnzKiOvyAAQOQI6wCQpx7c2hjYfvGp0xWw9C0AIA8R1gEgDznn9MCW4LD++lNZshEACgVhHQDy0Iv7junAsa4h7WbSZctnJVARACAXCOsAkIfCRtVXLZiqadXlMVcDAMgVwjoA5KH7Q8L65SsYVQeAQkJYB4A809TSpef2HAnsI6wDQGEhrANAnnlwa6NcwJKNJ9VVasWcyfEXBADIGcI6AOSZDSFLNq5dMYslGwGgwBDWASCPdPdm9fC2g4F9V6xkCgwAFBrCOgDkkScaDqm1q3dIe2VZRhcvYX11ACg0hHUAyCO/fmF/YPuaJTNUWVYSczUAgFwjrANAnujty+rXz78a2LeWVWAAoCAR1gEgTzz28iE1t3UPac+Y9KbT5yRQEQAg1wjrAJAnfvnsvsD2i5ZM18zJFTFXAwCIA2EdAPJAd29W970YPAXmbWedFHM1AIC4ENYBIA88uuOgjnb0DGkvzZiuPIMpMABQqAjrAJAHfvlc8BSY1y+doSlV5TFXAwCIC2EdAFKus6dPv33xQGDfW5kCAwAFjbAOACn30LYmtQQ8CKm8JKN1p89OoCIAQFwI6wCQcj/btCew/dJlM1VbWRZzNQCAOBHWASDFmlq69MCWxsC+t509N+ZqAABxI6wDQIr94um96s26Ie2TK0q17jRWgQGAQkdYB4CUcs7px0/uDux72+tO0qTykpgrAgDEjbAOACn1RMNhbW9sDex777knx1wNACAJhHUASKnvPbYzsH3Z7BqdPb8u5moAAEkgrANACjW1dOm+F/YH9n3g/AUys5grAgAkgbAOACn0w8d3qadv6I2lk8pK9K7V8xOoCACQBMI6AKRMZ0+f7vpDQ2DfVeecxNrqAFBECOsAkDL3PLNXB1u7A/s+dOHCmKsBACSJsA4AKZLNOn33kVcC+y5cPE2nn8SNpQBQTAjrAJAi6186ELpc459dsjjmagAASSOsA0BKOOf0zQ3bA/sWz6zW2uWzYq4IAJA0wjoApMSDW5v0wt5jgX1/fuliZTIs1wgAxYawDgApkM06fWX91sC+eVMm6Z3nsFwjABQjwjoApMC9L+zXi/uCR9U//obFKi/l7RoAihHv/gCQsO7erL66fltg35zaSr3n3JNjrggAkBaEdQBI2F1/aNArB9sC+274o6WqLCuJtyAAQGoQ1gEgQc2tXfrG/cErwJwyo1rvWc1cdQAoZoR1AEjQl3+9RS2dvYF9/+NNy1Vawts0ABQzfgsAQEL+UN+sn27aE9h3/inT9OYz5sRcEQAgbQjrAJCAju4+/d3Pnw/sM5P+8a2nyYx11QGg2BHWASAB/999W0JvKv3gBQt0xry6mCsCAKQRYR0AYvbQtibdubEhsG/W5Ar9zZUr4i0IAJBahHUAiNGew+264e6nQ/tvevvpqq0si7EiAECaEdYBICadPX36xH88pSPtPYH9bzv7JL35zLkxVwUASDPCOgDE5KZfvqjn9hwN7Js1uUL/9I7TY64IAJB2hHUAiMH//f0r+uHjuwP7SjKmb7z/HE2pKo+5KgBA2hHWASDHfvzkbv3Tr14K7f/bK5froiXTY6wIAJAvCOsAkEP3vbBfn/3Zc6H9bz5jjv7sksUxVgQAyCeEdQDIkXuf36/rf/iMsi64f/HMav3v95zNw48AAKFKky4AAArRHY++oi/86iW5kKA+o6ZCt19znmoqeBsGAITjtwQARKgv6/Qv923Rtx9+OXSf2spSfe+j52vRjOoYKwMA5CPCOgBE5MCxTn3y7mf0h5ebQ/eZVFaiO649Xyvn1sZYGQAgXxHWASACG7Y26q9//Kya27pD95lUVqLvXnOuVi+cGmNlAIB8RlgHgAloPNapf/7Pzfp/z+4bdr8ZNeW6/SPn6az5U2KqDABQCAjrADAOnT19+v5jO/WN321XS1fvsPsuml6lf//T87VwOnPUAQBjQ1gHgDHo6u3Tj57Yrds27NCBY10j7n/R4un65gfO0fSaihiqAwAUGsI6AIzC3iMd+uF/7dLdT+zSwdbween9MiZ98o+W6bq1p6okwzrqAIDxIawDQIiWzh799qUD+uWz+/TQtqbQhxsNNreuUje/73W6YPH03BYIACh4hHUA8DnntLO5XQ9ta9KDWxv1aH2zunuzo359ScZ07cWL9Mk3LuNhRwCASPDbBEDR6uzp0wt7j+rpXUf01K7DemrX4VHNQw9y3qKp+sI7zmD9dABApCIL62Y2X9IXJF0pabqk/ZJ+Iekm59zhMRxnmqR/lHSVpLmSmiXdJ+kfnXN7oqoXQHHIZp2aWru053CHdh9q1/bGFm0/0Krtja3a2dw26qktYc4+eYo+9UdL9YZlM2XG3HQAQLQiCetmtkTSRkmzJN0jaYuk8yXdIOlKM1vjnAt/pN9rx5nuH2eZpAck3S1phaRrJf2xmV3knAt/hjeAgteXdWrt6vU+OnvV0tmjQ23dam7rVnNrlw62dutga5cOtnZp/9FO7T/Sqe6+0U9lGa01p07XR19/itYun0VIBwDkTFQj69+SF9Svd87d2t9oZl+T9ClJX5T08VEc50vygvrXnXOfHnCc6yV9w/8+V0ZUc8519fZpZ3N7YJ8LGc1zCu4I2z/qY8XxPcK6XMiLhis3/PtEWVdYe/jBxnyssFdEWe8wBwt/Tdj+Ts55wbk365R1Tn3+trfPqc85ZQf1Hf8Y0Nfdm1VXb1ZdvX3q6smquy+rrh7/6wF9Hd19x8N5W3df6M+Ra3WTyvTOc+bpQxcu1KmzahKrAwBQPGy4wDGqA5gtllQvqUHSEudcdkDfZHnTYUzSLOdc2zDHqZbUJCkraa5zrmVAX8b/Hov87zGu0XUz27Rq1apVmzZtGs/Lx2z7gRa98esPx/K9AORGTUWp3njabL3t7Ll6/akzVV6aSbokAEAeWL16tZ566qmnnHOrJ3KcKEbWL/e36wcGdUlyzrWY2aOS1km6UNL9wxznIkmT/OO0DOxwzmXNbL2k/y5prSSmwgDIiYxJK+fW6pKlM3XZ8platWAqAR0AkJgowvpyf7stpH+7vLC+TMOH9dEcR/5xhmVmYUPnK0Z6LYDiMmtyhc6aX6dzFkzVOQum6Oz5U1TNsosAgJSI4jdSnb89GtLf3z4lpuMAwAlqKko1b8okzZ86SQumV2nZ7MlaNrtGp86crLqqsqTLAwAgVBzDR/3LJExwgbTRHydsbpA/4r5qgnUASFh1eYlqKks1ubJMNRWlqp1Uphk15ZpRU6EZNeWaXl2h6f7XJ0+tUu2kUlZsAQDkpSjCev+Id11If+2g/XJ9nNQoL81o6TArRoRlB1Nwx3iyRlhAGe5QoXVFWG9oV5T1hu4ffrTw14TtP/YfcqzfY7jvE9V/q5G+f5CMmUoy/of/eSZjKs2Y3yeVZDLe1gb0Ddi/ojSjitISVZRlVF6SUUWZ//WA9v7PvXBequryUpVkCN4AgOIQRVjf6m/D5pIv9bdhc9GjPk5qLJxerd9++g1JlwEAAIA8FcUSBxv87Tp/icXj/KUb10jqkPTYCMd5zN9vjf+6gcfJyLtJdeD3AwAAAArahMO6c65e0np5a6BfN6j7JknVku4auMa6ma0wsxNWZnHOtUr6nr//5wcd5y/94/+GJ5gCAACgWER1g+knJG2UdIuZXSFps6QL5K2Jvk3SjYP23+xvB088/TtJl0n6tJm9TtLjklZKeoekRg39xwAAAABQsCJ50oc/un6upDvlhfTPSFoi6RZJFznnmkd5nGZ5D0e6RdKp/nEukHSHpNX+9wEAAACKQmRLNzrndku6dpT7hi7l4Jw7JOkG/wMAAAAoWjxDGwAAAEgpwjoAAACQUoR1AAAAIKUI6wAAAEBKEdYBAACAlCKsAwAAAClFWAcAAABSirAOAAAApBRhHQAAAEgpwjoAAACQUuacS7qG2JhZ86RJk6atXLky6VIAAABQwDZv3qyOjo5DzrnpEzlOsYX1VyTVSmpI4Nuv8LdbEvje+YjzNTacr7HjnI0N52tsOF9jxzkbG87X2CRxvhZJOuacO2UiBymqsJ4kM9skSc651UnXkg84X2PD+Ro7ztnYcL7GhvM1dpyzseF8jU0+ny/mrAMAAAApRVgHAAAAUoqwDgAAAKQUYR0AAABIKcI6AAAAkFKsBgMAAACkFCPrAAAAQEoR1gEAAICUIqwDAAAAKUVYBwAAAFKKsA4AAACkFGEdAAAASCnCOgAAAJBShPUcMrOlZva3ZvaAme02s24zO2Bm95jZ2hFee42ZPW5mrWZ21MweNLO3xlV7EsyszMxuMLM7zOwZ/3w5M/vYMK/5iL9P2MfH4/wZ4jSe8zXgtUV3fQ3HzBaNcB3dnXSNSTGz+WZ2u5ntM7MuM2sws5vNbGrStaWNf27CrqFXk64vKWb2bjO71cweMbNj/vn4/givudjM7jWzQ2bWbmbPmdknzawkrrqTMpbzxXuXZGbTzexjZvZzM9thZh3+77Xfm9lHzSww6+bTNVaadAEF7p8kvU/SS5LulXRI0nJJb5f0djO7wTl3y+AXmdlXJH1G0h5J35FULun9kn5pZn/lnPtmTPXHrVrSzf7nByS9KunkUb72HknPBLQ/GUFdaTWu81XE19doPCvpFwHtL8RdSBqY2RJJGyXNkvf/2BZJ50u6QdKVZrbGOdecYIlpdFSv/X85UGvchaTI30s6W9452CNpxXA7m9k7JP1MUqekH8n73fk2SV+XtEbSe3JZbAqM6Xz5ivm96z2S/lXSfkkbJO2SNFvSuyR9V9Kbzew9bsBTQPPuGnPO8ZGjD0kfkXROQPsbJHVL6pI0d1DfxZKcpB2Spg5oXySpWd6FtSjpny1H56tc0pv7z4mkz/vn4mMjnGMn6SNJ158n56tor68RzuUi/7zcmXQtafqQ9Bv/vPzVoPav+e3/lnSNafqQ1CCpIek60vYhaa2kpZJM0mX+tfP9kH1rJTX6vx/PHdBeKe8fjk7S+5P+mVJ0vor+vUvS5fKCdmZQ+xx5wd1J+pMB7Xl3jTENJoecc3c6554OaH9I0oPywtbFg7r7p2180Tl3eMBrGiTdJqlC0rW5qDdpzrlu59yvnXP7k64lH4zzfBXt9YWxMbPFktbJC6C3Der+nKQ2SVebWXXMpSHPOOc2OOe2Oz8RjeDdkmZKuts5d/wvo865TnkjzpL0FzkoMzXGeL6KnnPuAefcL51z2UHtr0r6N//LywZ05d01RlhPTo+/7R3Ufrm/vS/gNb8etA9e8zp/rtlnzexqM5ufdEEpxfU1vJPM7M/N7O/87VlJF5Sg/utgfcAvwRZJj0qqknRh3IWlXIWZfci/hm4ws7VpnAObYsO9Rz0sqV3SxWZWEV9JeYH3rmBBWSvvrjHmrCfAzBZKukLeBfHwgPZqSfMktYaMlm73t8tyXmT+uWHQ131m9l1Jn/T/tVz0uL5G5Y3+x3Fm9qCka5xzuxKpKDnL/e22kP7t8kbel0m6P5aK8sMcSd8b1PaKmV3r/1UVwwu97pxzvWb2iqTTJS2WtDnOwlKO965BzKxU0of9LwcG87y7xhhZj5n/L7X/kDfd4PMDpyJIqvO3R0Ne3t8+JUfl5aNXJP2VvP/5qiWdJOm98v50/+eSbk+ssvTh+grXLu+G8NWSpvofb5B3s9Jlku4vwukeXC9jd4e8gZg58t6PzpT0bXnzin9tZmcnV1re4LobG967wv0vSWdIutc595sB7Xl3jRHWRzDCUlxBH6HLUfl/Cv2evDuNfyTpK+MsK7Xz2KI8X6PhnHvIOfdN59w251y7c26/c+4n8m7QOSzpv6X5F2Tc52uUUnt9DWci59I51+ic+0fn3FPOuSP+x8PyRo7/S9KpkkZcErPImL/Ny+slF5xzN/nzZw/470cvOOc+Lu+G3EnybgLHxHDdDcB7VzAzu17eqmdbJF091pf729RcY0yDGVm9vBUyRmtfUKMf1L8vbzmgH0v6UMDNI/3/mqtTsJH+NZgGkZyviXLO7TazeyV9UNKl8pa1SqM4z1chXF/Difxc+n8S/a6kC+RdR98YZ235aKTrpXbQfgj3b/KCw6VJF5IHuO4iUMzvXWZ2nbyf9yVJVzjnDg3aJe+uMcL6CJxzV0z0GP68qR/IC+o/kPRh51xfwPdqM7O9kuaZ2dyAecVL/W3YHNLERXG+ItTkb1P7J8A4z1chXF/DyeG5TP11lCNb/W3YPQx5fb3ErNHfFts1NB5bJZ0r77rbNLDD/116irybBV+Ov7S8U3TvXWb2SXlrpb8gL6g3BuyWd9cY02ByzMzKJf1UXlC/S9LVQUF9gAf87ZUBfW8etA+Gd4G/Tc3/cCnA9TV2/audFNt1tMHfrhv8BEAzmyxvOl+HpMfiLiwPXeRvi+0aGo/h3qMulbcC0UbnXFd8JeWtonrvMrO/lRfUn5G0NiSoS3l4jRHWc8i/mfTnkt4h6f9KunbwEmgB+tcEvdEGPM7bzBZJuk7eIv53RF5snjKzSwLazMz+p7xfkAcVvDxTseL6CmBmF/j/sB7cfrmkT/lfxnG/QGo45+olrZd3c+R1g7pvkjdad5dzri3m0lLJzE43s2kB7Qsl9T8VuKiuoXH6qbz37feb2bn9jWZWKemf/S//NYnC0oj3Lo+Z/YO8G0o3yRtRPzjM7nl3jRlr7ueOmd0h7wmbByV9S8E3KzzonHtw0Ou+KunT8h4z/FN5D096n6Tp8p4kWLCPgzezz+q1Ryu/Tt4jlzfqtWUFf++c++6A/Z28P8M/IWmvvDloa+TdAd4u6Z3OufXxVB+/sZ4v/zVFe32F8Zc4O13ew8r2+M1n6bX1eP/BOffPQ19Z2MxsibzraZake+QtY3aBvBu4t0m62DnXnFyF6WFmn5f0WXl/kXhFUoukJZL+WN6TEe+V937UnVSNSTGzqyRd5X85R9Kb5I32PuK3HXTO/fWg/X8q7x6Uu+U9Cv7t8lb9+qmk9xbyA4PGcr5475LM7BpJd0rqk3SrgueaNzjn7hzwmvy6xqJ4DCofoY/AfVBeQB/u4/Mhr71GXgBtk/em/5Cktyb9M6XgnN05aP//7Z+bffL+p2uXd/f3NyUtTvrnSdv5Kvbra5jz+FFJv5K35GervL8w7JK3atMlSdeX8Lk5Wd5fW/ZL6pa0U97NW9OSri1NH/KWy/uh//5zRN7DWJok/VbeWs+WdI0JnpvPj/A+1RDwmjXy/oFzWN50q+fljRSXJP3zpOl88d41qvPl5A2M5u01xsg6AAAAkFLMWQcAAABSirAOAAAApBRhHQAAAEgpwjoAAACQUoR1AAAAIKUI6wAAAEBKEdYBAACAlCKsAwAAAClFWAcAAABSirAOAAAApBRhHQAAAEgpwjoAAACQUoR1AAAAIKUI6wAAAEBKEdYBAACAlCKsAwAAAClFWAcAAABS6v8HCB5mrKV8Q2IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 250,
       "width": 373
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.lineplot(pts, 1/(1+np.exp(-pts))) ;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* tanh\n",
    "\n",
    "\\begin{equation*}\n",
    "f(z) = \\frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAwkAAAH0CAYAAAB2CGFiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XmYZVV96P3vr+duoLshoiCoCNpAjGgAhcANMuQlaJzCkPjeSBSjN04Rp3vjjRrBV42+N1FeHBMnohjRNDcY34ghVwYVggntVYm2gEAr0KgM0kAPVT387h97l1RVn11Vp/rU3qu6vp/nOc/us/aw1tnP7nXqd9YUmYkkSZIkjZjXdQEkSZIklcUgQZIkSdIYBgmSJEmSxjBIkCRJkjSGQYIkSZKkMQwSJEmSJI1hkCBJkiRpDIMESZIkSWMYJEiSJEkawyBBkiRJ0hgGCZIkSZLGMEiQJEmSNIZBgiRJkqQxDBIkSZIkjWGQIEmSJGkMgwRJkiRJYyzougBzQUTcDiwH1nVcFEmSJO3eDgIezMwn7spFDBLasXzp0qX7HH744ft0XRBJkiTtvtauXcvmzZt3+ToGCe1Yd/jhh++zZs2arsshSZKk3dhRRx3Ft7/97XW7eh3HJEiSJEkawyBBkiRJ0hgGCZIkSZLGMEiQJEmSNIZBgiRJkqQxDBIkSZIkjWGQIEmSJGkMgwRJkiRJYxgkSJIkSRrDIEGSJEnSGAYJkiRJksYoIkiIiDMj4oMR8Y2IeDAiMiIunua1DoyIT0XE+ogYioh1EXFBROw9wTm/GhFfjIifR8SWiLgpIs6PiKXT/1SSJEnS7LSg6wLU3gY8DXgYuBM4bDoXiYhDgOuARwNfAn4IPBM4FzgtIo7PzPvGnXMMcCWwEFgN3AGcDPw5cEpEnJKZQ9MpjyRJkjQbFdGSALwBWAUsB161C9f5CFWA8LrMfGFmviUzTwY+ABwKvHv0wRExH/g0sAw4MzP/c2b+KXAMcClwfF02SZIkac4ooiUhM68a+XdETOsaEXEwcCqwDvjwuN3vAP4LcHZEvCkzN9bpzwIOB76emf84qjw7IuK/AWcAr4yI92VmTqtgkqTObB7ezl0PbOaeh4a45+EhHty8lW3bd7BtR7J9R7JtR7Jte7J9x46B5+2XhqQRrznpSSxZOL/rYvSliCBhQE6ut1dk5pjaPjMfiohrqYKIY4GvjTvnq+Mvlpm3RcTNVC0cBwO3zkipJUkDtf6BzXz+337ClT/8OWvvfpAd/rUuqWMv/82DDRI6dGi9vblh/y1UQcIqHgkSpnLOqvo1aZAQEWsadk1rjIUkaeq2bN3Oe76ylouv/7GBgSTtot0pSFhRbzc07B9JX7mL50iSCrP+gc284jM38P31D3ZdFEnaLexOQcJkRgY79PP7Ul/nZOZRPS9StTAc2Ue+kqQp2ji0jRd/8lvcds/GyQ+WJE1JKbMbDcLIr/4rGvYvH3fcdM+RJBXkLy5fa4AgSQO2O7Uk3FRvVzXsf3K9HT3+YDrnSJIKce2P7uXi638y4TGP2nMxj9tnKfvuuZh99ljEogXzmD8vWDi/2i6YF8yfFwTTm11vItOcsE/Sbmbxgtn3u/zuFCSMTKN6akTMGz3DUUTsRbXmwWbg+lHnXAm8FTgN+IvRF6unVF0F/Bi4bQbLLUmapgv+V/NvOKc9ZT/edOoqnvToPac9vbYkzVWzLqyJiIURcVi9uvIvZeatwBXAQcBrxp12PrAH8JlRayQAXAOsBU6IiOePymMe8L767cdcI0GSyrPu3o38+7pf9Nz3O0/dn4+++Eie/Ji9DBAkaRqKaEmIiBcCL6zf7ldvfyMiLqr/fW9mvrn+9wFUf9j/mCogGO3VwHXAhRFxSn3cMcBJVF2G3jr64MzcHhHnULUorI6I1cBPgFOAo4FrqVZrliQV5tJv39kzfd+9FvOe059qcCBJu6CIIAF4OvCScWkH1y+oAoI3M4nMvDUijgbeSdWF6DnA3cCFwPmZeX+Pc74VEc+gam04Fdirzu+dwHszc2han0iSNGN27Ej+57fv6rnv944+kBVLF7ZcIknavRQRJGTmecB5Uzx2HTSPLsvMO4Bz+sz/B8BZ/ZwjSerOmp/8grse2Nxz3+lHHthyaSRp9zPrxiRIkvTv63ZqGAbg1x+/kkP23bPl0kjS7scgQZI063zvjt7L15z2lP16pkuS+mOQIEmadW68q3eQ8LTHrWy5JJK0ezJIkCTNKvc+PNRzPEIE/NoBKzookSTtfgwSJEmzyo139m5FOGTfPdlzcRHzcUjSrGeQIEmaVb7XECQcYSuCJA2MQYIkaVa58a4HeqY/9UCDBEkaFIMESdKs8oP1D/ZMP+JABy1L0qAYJEiSZo2t23fw0we39Nx3+P57tVwaSdp9GSRIkmaNnz24hR25c/reyxaybJGDliVpUAwSJEmzxt0berciPHbl0pZLIkm7N4MESdKssb7H+ggA+68wSJCkQTJIkCTNGusf6N2ScMDKJS2XRJJ2bwYJkqRZo7Elwe5GkjRQBgmSpFnj7g29gwTHJEjSYBkkSJJmjbsauhs9doXdjSRpkAwSJEmzRlNLgt2NJGmwDBIkSbPCpuFtPLBp607p8wIes9fiDkokSbsvgwRJ0qzQNLPRY5YvYcF8v84kaZCsVSVJs4KDliWpPQYJkqRZoXkhNQctS9KgGSRIkmaF+zYO90zfb7lBgiQNmkGCJGlW2NBj0DLA3nssarkkkrT7M0iQJM0KGzb3DhKWL13YckkkafdnkCBJmhWagoQVBgmSNHAGCZKkWaHXGgkAKw0SJGngDBIkSbOCLQmS1B6DBEnSrGCQIEntMUiQJM0KTUHCymUGCZI0aAYJkqTibd2+g4eHtvXct9cSgwRJGjSDBElS8R5smv50yQLmz4uWSyNJuz+DBElS8RrHI9jVSJJmhEGCJKl4DzhoWZJaZZAgSSpe46DlpYtaLokkzQ0GCZKk4jWNSbAlQZJmRlFBQkQcGBGfioj1ETEUEesi4oKI2HuK558YETmF1+PGnTfRsdfPzKeVJE1V02rLyw0SJGlGLOi6ACMi4hDgOuDRwJeAHwLPBM4FTouI4zPzvkkusw44v2HfU4HTge9n5h099v8YuKhH+p2TFl6SNKNcI0GS2lVMkAB8hCpAeF1mfnAkMSLeD7wBeDfwyokukJnrgPN67YuIz9f//JuG09dlZs9zJUndampJsLuRJM2MIrobRcTBwKlULQEfHrf7HcBG4OyI2GOa1/8V4HeBzcBnp19SSVIXmgcuGyRI0kwopSXh5Hp7RWbuGL0jMx+KiGupgohjga9N4/ovBRYDn8nMXzQcszIiXgbsB2wA1mSm4xEkqQCN6yQYJEjSjCglSDi03t7csP8WqiBhFdMLEl5eb/96gmOeBnxydEJEfBc4OzNvnEomEbGmYddhUzlfktTbhs3DPdMNEiRpZhTR3QhYUW83NOwfSV/Z74Uj4llUf6R/PzOvazjs/cDxwL7AXsAzgNVUgcOVEXFAv/lKkgbHFZclqV2ltCRMJuptTuPc/1JvG1sRMvNN45JuAM6KiNXAGcCbqQZPTygzj+qVXrcwHDml0kqSduLAZUlqVyktCSMtBSsa9i8fd9yURMQ+VH/kT3fA8sfq7QnTOFeSNCAPbjFIkKQ2lRIk3FRvVzXsf3K9bRqz0OQlVAOWv5iZD0yjXPfU22nNqiRJ2nXbdyRbtu7ouW+PRbOlQVySZpdSgoSr6u2pETGmTBGxF9V4gc1Av7MNvaLeNq2NMJlj6+1t0zxfkrSLNm/d3jN9ycJ5zJsXPfdJknZNEUFCZt4KXAEcBLxm3O7zqX7J/0xmbhxJjIjDIqJx1qCI+E3gcOA/JhiwTEQc2Wv9hYg4gmoBN4CLp/hRJEkDtml4W8/0ZbYiSNKMKamGfTVwHXBhRJwCrAWOAU6i6mb01nHHr623TT8jjQxYnqwV4XXA6RFxJXAHMEQ1G9JpwHzg48Dnm0+XJM2kzcO9WxKWLpzfckkkae4oJkjIzFsj4mjgnVR/oD8HuBu4EDg/M++f6rUiYm/gTKY2YPkyqoHRR1At6rYEuA+4HPh4Zv5jnx9FkjRAmxqChGWLDBIkaaYUEyQAZOYdwDlTPLaxI2q9qvLSKV7nMqpAQZJUIIMESWpfEWMSJElq0tjdyCBBkmaMQYIkqWgOXJak9hkkSJKK1jQFqi0JkjRzDBIkSUVrHJPg7EaSNGMMEiRJRXPgsiS1zyBBklS0zQ1jEpY6JkGSZoxBgiSpaLYkSFL7DBIkSUUzSJCk9hkkSJKK5joJktQ+gwRJUtE2NUyBakuCJM0cgwRJUtEaBy4vdOCyJM0UgwRJUtEckyBJ7TNIkCQVzSBBktpnkCBJKpoDlyWpfQYJkqSibdrae0zCMhdTk6QZY5AgSSpaU0uC3Y0kaeYYJEiSitY0JsHuRpI0cwwSJEnFykw2N62TsNAgQZJmikGCJKlYW7buIHPn9EXz57Fgvl9hkjRTrGElScXa1LSQml2NJGlGGSRIkorlGgmS1A2DBElSsZrGI9iSIEkzyyBBklQsWxIkqRsGCZKkYjWNSVi20IXUJGkmGSRIkorVtJCa3Y0kaWYZJEiSimV3I0nqhkGCJKlYtiRIUjcMEiRJxWock2CQIEkzyiBBklSsTQ1ToC5b5MBlSZpJBgmSpGJt2bqjZ/qShbYkSNJMMkiQJBVraFvvloQlC/36kqSZZC0rSSrW8LbeLQmL5vv1JUkzyVpWklSsoYYgYbHdjSRpRhkkSJKKNdQwJmHxAr++JGkmFVXLRsSBEfGpiFgfEUMRsS4iLoiIvfu4xtURkRO8ljSc96sR8cWI+HlEbImImyLi/IhYOrhPKEnqR9OYBIMESZpZxcwhFxGHANcBjwa+BPwQeCZwLnBaRByfmff1ccnzG9J3mnQ7Io4BrgQWAquBO4CTgT8HTomIUzJzqI+8JUkD0NjdaIHdjSRpJhUTJAAfoQoQXpeZHxxJjIj3A28A3g28cqoXy8zzpnJcRMwHPg0sA16Qmf9Yp88DvgicUef/3qnmLUkajOYxCbYkSNJMKqKWjYiDgVOBdcCHx+1+B7ARODsi9piB7J8FHA58fSRAAMjMHcB/q9++MiJiBvKWJE1guKm7kbMbSdKMKqWWPbneXlH/cf5LmfkQcC3VL/3HTvWCEfH7EfGWiHhjRDw7IhZPkvdXx+/IzNuAm4EnAAdPNW9J0mDYkiBJ3Silu9Gh9fbmhv23ULU0rAK+NsVrXjLu/c8j4jWZuXoaea+qX7dOlGFErGnYddhE50mSemue3cgxCZI0k0r5KWZFvd3QsH8kfeUUrvUl4HnAgcBSqj/Q/6I+9wsR8ewZzFuSNEDObiRJ3SilJWEyI+MBcrIDM/MD45JuAv4sItYDHwTeA1w+Q3kf1fMCVQvDkX3kKUnC2Y0kqSul/BQz8mv9iob9y8cdNx2foJr+9OkRsVfLeUuSpmG4IUhYZEuCJM2oUmrZm+rtqob9T663TeMGJpWZW4CH6rejZ0ma8bwlSdPT3JJQyteXJO2eSqllr6q3p9brE/xS/av/8cBm4PrpZhARhwJ7UwUK947adWW9Pa3HOQdTBQ8/Bm6bbt6SpOlpHJPg7EaSNKOKqGUz81bgCuAg4DXjdp9P9cv/ZzJz40hiRBwWEWNmDYqIgyPigPHXj4hHUS2YBnBJZo5edfkaYC1wQkQ8f9Q584D31W8/lpmTjkmQJA1OZja2JCxynQRJmlElDVx+NXAdcGFEnEL1h/sxwElUXX3eOu74tfV29CJnJwCfiIhrqKYrvR94PPAcqjEHN/DIAmkAZOb2iDiHqkVhdUSsBn4CnAIcTbVGw/jB0JKkGbZ1e9Lr55kF84IFBgmSNKOKCRIy89aIOBp4J1XXn+cAdwMXAudn5v1TuMwa4GLgKODpVIOOHwJuBL4I/HVmDvfI+1sR8QyqVotTgb2ouhi9E3hvZg7t4seTJPWpqauRg5YlaeYVEyQAZOYdwDlTPDZ6pN0IvHSaef8AOGs650qSBq9pZiMHLUvSzLOmlSQVyTUSJKk7BgmSpCI1BgnObCRJM86aVpJUpMbpT+1uJEkzzppWklSkoa12N5KkrhgkSJKK1LhGgi0JkjTjrGklSUVydiNJ6o41rSSpSI5JkKTuWNNKkorkFKiS1B2DBElSkRpbEpwCVZJmnDWtJKlITbMbLZrvV5ckzTRrWklSkYa3u5iaJHXFmlaSVCTXSZCk7hgkSJKK5OxGktQda1pJUpGc3UiSumOQIEkqkisuS1J3rGklSUUa2mp3I0nqijWtJKlIzm4kSd2xppUkFcnZjSSpOwYJkqQiNQ9c9qtLkmaaNa0kqUhOgSpJ3bGmlSQVydmNJKk71rSSpCI5JkGSumOQIEkq0pCzG0lSZ6xpJUlFcp0ESeqONa0kqUjDjbMb2d1IkmaaQYIkqUhOgSpJ3bGmlSQVySlQJak71rSSpCI1tyTY3UiSZppBgiSpSI1BgrMbSdKMs6aVJBUnMxsHLi+a71eXJM00a1pJUnGGG9ZIWDAvmDcvWi6NJM09BgmSpOI0tiI4aFmSWmFtK0kqztbt2TPdIEGS2mFtK0kqTlNLwkLHI0hSK4qqbSPiwIj4VESsj4ihiFgXERdExN5TPH+PiPiDiPi7iPhhRGyMiIci4oaIeFNELGo4Lyd4XT/YTylJmszWhjEJDlqWpHYs6LoAIyLiEOA64NHAl4AfAs8EzgVOi4jjM/O+SS7zm8DFwP3AVcBlwD7A84C/BE6PiFMyc0uPc38MXNQj/c7+P40kaVc0TX9qdyNJakcxQQLwEaoA4XWZ+cGRxIh4P/AG4N3AKye5xk+BFwN/n5nDo66xF3A1cBzwGuCvepy7LjPP24XyS5IGxJYESepWEbVtRBwMnAqsAz48bvc7gI3A2RGxx0TXyczvZObnRgcIdfpDPBIYnDiIMkuSZk7jmIQFTn8qSW0opSXh5Hp7RWaO+WbIzIci4lqqIOJY4GvTzGNrvd3WsH9lRLwM2A/YAKzJTMcjSFIHbEmQpG6VEiQcWm9vbth/C1WQsIrpBwkvq7dfbdj/NOCToxMi4rvA2Zl54zTzlCRNg7MbSVK3SgkSVtTbDQ37R9JXTufiEfFa4DTgO8CnehzyfuBSqiBlC3AY8KfAmcCVEfH0zLxrCvmsadh12HTKLUlzVdOKyw5clqR2zJbadqQTau/VdSY6MeJ04AKqQc1nZObW8cdk5psy87rMvDczH87MGzLzLKrA4VHAm3eh7JKkPjWuuGxLgiS1opSWhJGWghUN+5ePO25KIuKFwCXAz4GTMvO2Psv1MeAM4ISpHJyZRzWUYw1wZJ95S9Kc5YrLktStUmrbm+rtqob9T663TWMWdhIRZwF/D/wMeFZm3jTJKb3cU28nnFVJkjRYw9u390x3TIIktaOU2vaqentqRIwpU73GwfHAZmBKsw1FxH8GPg+spwoQbplmuY6tt/22QEiSdsHWbbYkSFKXiqhtM/NW4ArgIKrFzkY7n+qX/M9k5saRxIg4LCJ2GhAcES8BPgv8BDhhsi5GEXFkr/UXIuIIqgXcoFrFWZLUkqGGgcu2JEhSO0oZkwDwauA64MKIOAVYCxwDnETVzeit445fW29/ubJORJxENXvRPKrWiXMidlp454HMvGDU+9cBp0fElcAdwBDVbESnAfOBj1O1SkiSWrK1YeDyYlsSJKkVxQQJmXlrRBwNvJPqD/TnAHcDFwLnZ+b9U7jME3ikdeRlDcf8mGq2oxGXUQ2MPoJqUbclwH3A5cDHM/Mf+/wokqRd1DQF6sL5rrgsSW0oJkgAyMw7gHOmeOxO3xSZeRFwUZ95XkYVKEiSCtHUkuCYBElqh7WtJKk4zS0Jfm1JUhusbSVJxXHFZUnqlrWtJKk4rrgsSd2ytpUkFWerLQmS1ClrW0lScZpaEhyTIEntsLaVJBVn6/aGFZcNEiSpFda2kqTiNLYk2N1IklphbStJKk7j7Ea2JEhSK6xtJUnFaZzdaIErLktSGwwSJEnFaZzdaP78lksiSXOTQYIkqTjNsxvZkiBJbTBIkCQVx3USJKlb1raSpOIMuU6CJHXK2laSVJymloTFtiRIUiusbSVJxWmaAtWWBElqh7WtJKk4W7c1rLhsS4IktcLaVpJUHFsSJKlb1raSpOJsbVxMza8tSWqDta0kqThDjYup+bUlSW2wtpUkFSUzXSdBkjpmbStJKsq2HUn2GLc8L2D+PFdclqQ2GCRIkopiK4Ikdc8aV5JUlGFXW5akzlnjSpKK0jT9qastS1J7rHElSUWxJUGSumeNK0kqytbtrrYsSV2zxpUkFcWWBEnqnjWuJKkojbMbGSRIUmuscSVJRRlqakmwu5EktcYaV5JUlOaWBBdSk6S2GCRIkorSNCbBgcuS1B5rXElSUZpaEhy4LEntscaVJBWlsSXBIEGSWmONK0kqStOKyw5clqT2WONKkorS1JKw2JYESWpNMTVuRBwYEZ+KiPURMRQR6yLigojYu8/r7FOft66+zvr6ugfOdN6SpF3XtOKyYxIkqT0Lui4AQEQcAlwHPBr4EvBD4JnAucBpEXF8Zt43hev8Sn2dVcCVwCXAYcA5wO9ExG9k5m0zkbckaTCGt23vme7sRpLUnlJq3I9Q/ZH+usx8YWa+JTNPBj4AHAq8e4rXeQ9VgPCBzDylvs4Lqf7gf3Sdz0zlLUkaAFsSJKl7nde4EXEwcCqwDvjwuN3vADYCZ0fEHpNcZw/g7Pr4d4zb/aH6+r9d5zfQvCVJg9M0cNmWBElqTwk17sn19orMHPPNkJkPAdcCy4BjJ7nObwBLgWvr80ZfZwdwRf32pBnIW5I0IM1ToLrisiS1pYQg4dB6e3PD/lvq7aoZuM6g8gYgItb0elGNi5AkTYEtCZLUvRJq3BX1dkPD/pH0lTNwnUHlLUkakK0NLQmOSZCk9hQxu9EkRtqXe49km9nr9HVOZh7V8yJVa8KRfeQrSXPWVlsSJKlzJdS4I7/Wr2jYv3zccYO8zqDyliQNSOOKy7YkSFJrSqhxb6q3Tf3+n1xvm8YN7Mp1BpW3JGlAhrf1bry1JUGS2lNCjXtVvT01IsaUJyL2Ao4HNgPXT3Kd6+vjjq/PG32deVRTnY7Ob5B5S5IGpHHgsi0JktSazmvczLyVanrSg4DXjNt9PrAH8JnM3DiSGBGHRcSYGYMy82Hgs/Xx5427zmvr6//z6BWXp5O3JGlmNQ1ctiVBktpTysDlVwPXARdGxCnAWuAYqjUNbgbeOu74tfV2/KTZfwacCLwxIp4O/BtwOPAC4OfsHAhMJ29J0gxyTIIkda+IGrf+Rf9o4CKqP9DfBBwCXAj8RmbeN8Xr3Ee1qNqFwJPq6xwDfBo4qs5nRvKWJA2GsxtJUvdKaUkgM+8AzpnisY3Lbmbm/cC59WvgeUuSZtZQ4zoJrrgsSW3xZxlJUlGaWhIW25IgSa2xxpUkFWXYFZclqXPWuJKkojgmQZK6Z40rSSqKLQmS1D1rXElSUbZub1hx2SBBklpjjStJKkrT7EZ2N5Kk9ljjSpKK0jgmwZYESWqNNa4kqSiNYxJsSZCk1ljjSpKKYkuCJHXPGleSVIwdO5JtO3oPXHbFZUlqj0GCJKkYwxO0IkQYJEhSWwwSJEnFaAoSbEWQpHYZJEiSirHV6U8lqQjWupKkYjS3JPh1JUltstaVJBVj67aG1ZZtSZCkVlnrSpKKMbx9e890pz+VpHZZ60qSijFsS4IkFcFaV5JUDMckSFIZrHUlScVoXG3ZlgRJapW1riSpGMMNU6C6ToIktcsgQZJUjMYVlxfMb7kkkjS3GSRIkorR1JKwyJYESWqVQYIkqRiOSZCkMljrSpKK0Twmwa8rSWqTta4kqRiNLQkGCZLUKmtdSVIxGlsS7G4kSa2y1pUkFWN4e8OKy7YkSFKrrHUlScVonN3IlgRJapW1riSpGI5JkKQyWOtKkorh7EaSVAZrXUlSMVwnQZLKYK0rSSrGUGNLgisuS1KbDBIkScVoaklYbEuCJLXKWleSVAzHJEhSGYqpdSPiuIj4SkTcHxGbIuJ7EfH6iJjfxzUOiIg/iYjLI2JdRAxFxH0R8S8RcXrDOSdGRE7weu/gPqUkaSKOSZCkMizougAAEfEC4FJgC/AF4H7gecAHgOOBs6Z4qT8B/hS4HbgK+CnwBOB04Lci4gOZ+caGc68Bru6R/s0p5i1J2kXDDUGCLQmS1K7Og4SIWA58HNgOnJiZN9TpbweuBM6MiBdl5iVTuNy/1de4ZlwehwPXA2+IiM9l5poe516dmeftwkeRJO2i4W0NKy7bkiBJrSqh1j0T2Be4ZCRAAMjMLcDb6revmsqFMvN/jg8Q6vS1VC0UACfuUmklSTOmqSXBxdQkqV2dtyQAJ9fbr/bY93VgE3BcRCzOzKFdyGdrvd3WsP9JEfFaYDlVN6VvZOYtu5CfJKlPWxsGLtuSIEntKiFIOLTe3jx+R2Zui4jbgacABwNrp5NB3aXpDCCBKxoO+4P6Nfq8S4FXZOYvppOvJKk/jkmQpDKUECSsqLcbGvaPpK+czsUjIoBPAI8BPlJ3PRrtHuAtwD8B64AlwNHAe6gCi/0i4oTM7P3NNTavXmMdAA6bTtklaa5xdiNJKsNAat16utGJphEd/7q4n8vX296j2Sb3V1SzI30D2Glmo8z8fma+LzP/IzMfzsx7M/OrVGMXbqeaXel508xbktSH5nUSXHFZkto0qJaEW6mmL52q9aP+PdJSsKLXgVRjBEYfN2UR8T+AN1CNbfidfsY0ZOaDEfF3wFuBE4AvTeGcoxrKsQY4cqp5S9Jc1dTdyBWXJaldAwkSMvOUXTj9JqruPauAMd11ImIB8ESqwca39XPRiPgA8Hqq9RKem5mbplG2e+rtHtM4V5LUJ1dclqQylFDrXllvT+ux7wRgGXDdVFsBovJhqgDhX6haEKYTIAAcW2/7ClAkSdPjmARJKkMJte5q4F7gRRFx9EhiRCwB3lW//ejoEyJiWUQcFhGPH5cewN8ArwYuB56fmZsnyjwijo+Ine5DRLwY+H1gGPgbrlpUAAAc/0lEQVRi359KktQ3WxIkqQydz25U9/1/BVWwcHVEXALcDzyfanrU1TyyENqIZ1J1I7qGsYuj/TnwcmAz8B3gLVXcMMZ3MvOyUe8/B8yLiOuAO6lmN3pGncc24I8zc92ufUpJ0lRs3e6Ky5JUgs6DBIDMvCwinkU1SPgMqj/Uf0Q1G9GFmTnVmY2eWG+XAv+94Zi/BUYHCR8FfotqFqNHUc2mdBdwEXBBZn536p9EkrQrmloSXHFZktpVRJAAkJnXAs+Z4rFX88jUqKPTXwq8tM983we8r59zJEmDl5kupiZJhbDWlSQVoamr0fx5wfx5rpMgSW0ySJAkFaFxZiNbESSpdda8kqQiuNqyJJXDIEGSVIShhiBh8cL5LZdEkmSQIEkqQlNLwmKnP5Wk1lnzSpKKMLRte890gwRJap81rySpCI3djRbY3UiS2maQIEkqQlNLgqstS1L7rHklSUUY2uqYBEkqhTWvJKkIzm4kSeUwSJAkFaF5TIJfVZLUNmteSVIRnN1IksphzStJKoKzG0lSOQwSJElFaAoSnN1IktpnzStJKsLQVrsbSVIprHklSUUY3t40u5FfVZLUNmteSVIRmtdJcEyCJLXNIEGSVASnQJWkcljzSpKK4BSoklQOa15JUhFsSZCkcljzSpKKMOw6CZJUDIMESVIRGlsSnN1IklpnzStJKoLrJEhSOax5JUlFaB6TYHcjSWqbQYIkqQhNsxstsiVBklpnzStJKoKzG0lSOax5JUlFcHYjSSqHQYIkqQjObiRJ5bDmlSQVwRWXJakc1rySpCIMbbW7kSSVwiBBklSEpu5Gzm4kSe2z5pUkFaF54LJfVZLUNmteSVLnMtMxCZJUEGteSVLntu1IduTO6fPnBQvm+1UlSW2z5pUkdc6F1CSpLMXUvhFxXER8JSLuj4hNEfG9iHh9RPQ1rUVE5ASv6yc477kRcXVEbIiIhyPiWxHxkl3/ZJKkyQxttauRJJVkQdcFAIiIFwCXAluALwD3A88DPgAcD5zV5yV/DFzUI/3OhvxfC3wQuA+4GBgGzgQuioinZuab+8xfktSH4e3ObCRJJek8SIiI5cDHge3AiZl5Q53+duBK4MyIeFFmXtLHZddl5nlTzP8g4C+pApOjM3Ndnf5O4N+BN0XEpZn5r33kL0nqg2skSFJZSviJ5kxgX+CSkQABIDO3AG+r375qBvN/GbAY+NBIgFDn/wvgPfXbV85g/pI05zkmQZLK0nlLAnByvf1qj31fBzYBx0XE4swcmuI1V0bEy4D9gA3AmsxsGo8wUf6XjztGkjQDGqc/XWiQIEldKCFIOLTe3jx+R2Zui4jbgacABwNrp3jNpwGfHJ0QEd8Fzs7MG/vI/+6I2AgcGBHLMnPTRJlGxJqGXYdNrdiSNDc1tyTY3UiSulDCTzQr6u2Ghv0j6SuneL33Uw123hfYC3gGsJoqcLgyIg6YZv4rGvZLknZR85iEEr6mJGnuGUhLQkSsA57Qxymfy8wXT/Xy9bbHMjs7y8w3jUu6ATgrIlYDZwBvBt4wxbz7yj8zj+p5gaqF4cg+8pSkOWV4e+/uRs5uJEndGFR3o1uppi+dqvWj/j3ZL/XLxx03XR+jChJOGJe+AXhUnf99E+T/4C7mL0lqYEuCJJVlIEFCZp6yC6ffBBwNrALG9OmPiAXAE4FtwG27kAfAPfV2jx75P6rOf8w0pxGxf338nZONR5AkTZ9jEiSpLCX8RHNlvT2tx74TgGXAdX3MbNTk2Ho7PtiYKP9njztGkjQDGmc3siVBkjpRQu27GrgXeFFEHD2SGBFLgHfVbz86+oSIWBYRh0XE48elHxkR41sKiIgjgHfXby8et/vTwBDw2nphtZFz9gb+rH77sT4/kySpD40tCU6BKkmd6HwK1Mx8MCJeQRUsXB0Rl1Ctfvx8qulJVwNfGHfaM4GrgGuAE0elvw44PSKuBO6g+uP/MKpWgvlUKzt/flz+t0fEfwUuBG6IiC8Aw1SLvB0I/JWrLUvSzBpuCBIWzbe7kSR1ofMgASAzL4uIZwFvpRpcvAT4EfBG4MLMnNLMRsBlVAONj6BaAG0J1WDky4GPZ+Y/NuT/wXqGpjcDf0jVwvID4G2Z+bfT/VySpKmxJUGSylJEkACQmdcCz5nisVfzyNSko9MvowoUppP/l4EvT+dcSdKuGdrqmARJKom1rySpc85uJEllMUiQJHWuOUjwa0qSumDtK0nq3Jam7kaOSZCkTlj7SpI6t2m4d5CwbJHdjSSpCwYJkqTONQUJSxcWM7+GJM0pBgmSpM5t3rqtZ7otCZLUDYMESVLn7G4kSWUxSJAkdW5zU3cjgwRJ6oRBgiSpc80tCY5JkKQuGCRIkjpndyNJKotBgiSpc5uHew9ctruRJHXDIEGS1KnMZFPDYmrLFhokSFIXDBIkSZ0a2raDzJ3TF82fx4L5fk1JUhesfSVJnWpcSM2uRpLUGYMESVKnNjWMR3DQsiR1xyBBktQp10iQpPIYJEiSOuX0p5JUHoMESVKnGoOEhS6kJkldMUiQJHVq81bXSJCk0hgkSJI6ZXcjSSqPQYIkqVNOgSpJ5TFIkCR1qml2I1sSJKk7BgmSpE41dzdy4LIkdcUgQZLUqc0Ni6ktXWhLgiR1xSBBktQpBy5LUnkMEiRJndq01SBBkkpjkCBJ6lTTwOWljkmQpM4YJEiSOrWpYUyCLQmS1B2DBElSp1wnQZLKY5AgSepU4zoJzm4kSZ0xSJAkdcp1EiSpPAYJkqRObW6Y3cjuRpLUHYMESVKnHLgsSeUxSJAkdcrF1CSpPMUECRFxXER8JSLuj4hNEfG9iHh9REz5WyIizouInOR167hzTpzk+PcO/tNKkkY0r5NgkCBJXSliVFhEvAC4FNgCfAG4H3ge8AHgeOCsKV7q6gn2PQ84Eri8Yf81Ded/c4p5S5L6NLxtB9t25E7p8+cFi+YX8zuWJM05nQcJEbEc+DiwHTgxM2+o098OXAmcGREvysxLJrtWZl5Njz/069aIP6rf/k3D6Vdn5nn9ll+SNH0TTX8aES2XRpI0ooSfac4E9gUuGQkQADJzC/C2+u2rdjGP5wAHAtdn5vd28VqSpAHZ2DBo2a5GktStzlsSgJPr7Vd77Ps6sAk4LiIWZ+bQNPP4L/W2qRUB4EkR8VpgOfBT4BuZecs085MkTcGDW7b2TN9rSQlfT5I0d5VQCx9ab28evyMzt0XE7cBTgIOBtf1ePCIOAJ4NbKAa79DkD+rX6HMvBV6Rmb+YYl5rGnYdNpXzJWmueWBT7yBhxdKFLZdEkjRaCd2NVtTbDQ37R9JXTvP6LwfmAxdn5qYe++8B3gI8FdiLquvTs4H/DZwBfDkiSrhPkrTb2bC5d5CwctmilksiSRptIC0JEbEOeEIfp3wuM1881cvX252nv5jsxOqP+5fVb3t2NcrM7wPfH5X0MPDViLgO+A7V7ErPA740WX6ZeVRDOdZQzawkSRqlKUiwJUGSujWo7ka3Uk1fOlXrR/17pKVgRa8DqcYIjD6uH88GHs80Bixn5oMR8XfAW4ETmEKQIEnqzwa7G0lSkQYSJGTmKbtw+k3A0cAqYEyf/ohYADwR2AbcNo1rjwxY/utplu2eervHNM+XJE3AlgRJKlMJfe2vrLen9dh3ArAMuK7fmY0i4rHA71C1QHxxmmU7tt5OJ0CRJE3igc3DPdMNEiSpWyUECauBe4EXRcTRI4kRsQR4V/32o6NPiIhlEXFYRDx+guv+EdWA5c82DFgeudbxvQYmR8SLgd8Hhpl+kCFJmsCGzb3XSVi5zCBBkrrU+RSodd//V1AFC1dHxCXA/cDzqaZHXc3OU5c+E7gKuAY4cfw16z/6J1thecTngHn1QOU7gSXAM+o8tgF/nJnr+v5gkqRJ2d1IksrUeZAAkJmXRcSzqAYJn0H1h/qPgDcCF2ZmvzMb/TbVbEvXZ+aNkxz7UeC3qGYxehTVbEp3ARcBF2Tmd/vMW5I0RRs22d1IkkpURJAAkJnXAs+Z4rFX88jUqL32Xz7R/nHHvg9431SOlSQNVvM6CQYJktSlEsYkSJLmqKYgYbktCZLUKYMESVInduxIxyRIUqEMEiRJnXh4eBs7eow4W7pwPosXzG+/QJKkXzJIkCR1wtWWJalcBgmSpE44aFmSymWQIEnqhIOWJalcBgmSpE480NDdaKVBgiR1ziBBktQJZzaSpHIZJEiSOvHAZldblqRSGSRIkjrhwGVJKpdBgiSpEw/a3UiSimWQIEnqxD0PDfVMX7lsUcslkSSNZ5AgSerEXQ9s6Zm+/4olLZdEkjSeQYIkqRN3b9jcM/2xK5e2XBJJ0ngGCZKk1m0a3tZznYR5AY/ea3EHJZIkjWaQIElq3fqGrkb7LV/Cgvl+NUlS16yJJUmtW/9A765G+9vVSJKKYJAgSWpd03gEBy1LUhkMEiRJrWvqbnSALQmSVASDBElS6xq7G9mSIElFMEiQJLXu7g29WxKc/lSSymCQIElqXVNLgkGCJJXBIEGS1KrMZL0LqUlS0QwSJEmt+sWmrWzZumOn9MUL5rH3soUdlEiSNJ5BgiSpVT/6+cM90x+7cikR0XJpJEm9GCRIklr1vTsf6Jn+5Efv2XJJJElNDBIkSa268a4NPdOPOHBFyyWRJDUxSJAktep7dzYFCStbLokkqYlBgiSpNRs2b+X2ezf23PfUA2xJkKRSGCRIklrz/YauRo/bZyl777Go5dJIkpoYJEiSWvO9xvEIdjWSpJIYJEiSWvONW+7pmX6EXY0kqSgGCZKkVtz1wGauu/W+nvue9jhbEiSpJAYJkqRW/MO37yRz5/RH7bmYo56wd/sFkiQ1MkiQJM24rdt3sHrNnT33vfDpj2XhfL+OJKkkndfKEbEwIs6NiE9HxHciYjgiMiJevgvXPC4ivhIR90fEpoj4XkS8PiLmT3DOcyPi6ojYEBEPR8S3IuIl0y2DJOkRH77qR6y7b1PPfWccdWDLpZEkTWZB1wUA9gAuqP/9M+CnwOOme7GIeAFwKbAF+AJwP/A84APA8cBZPc55LfBB4D7gYmAYOBO4KCKemplvnm55JGku27Ej+dy3fswF/+uWnvt/df/lHL7/8pZLJUmaTAlBwibgOcB3MvPuiDgPeMd0LhQRy4GPA9uBEzPzhjr97cCVwJkR8aLMvGTUOQcBf0kVTBydmevq9HcC/w68KSIuzcx/ndan68D6Bzbz0JZtA7lW0qMD8XSvNbhLDfZahX7GQZsL92yQtz8HWLDBlmuAFxtoyWBo2w42DW1n4/A27nloiFvv2cg3f3QPd9y/ufGcP/pPTxxoGSRJg9F5kJCZw8DlA7rcmcC+wGdGAoQ6jy0R8Tbga8CrgEtGnfMyYDHwvpEAoT7nFxHxHuCTwCuBWRMkvPuf1vJPN97ddTEkaUK/+eRHcfqRB3RdDElSD52PSRiwk+vtV3vs+zpVq8VxEbF4iudcPu4YSdIALF+ygP/3zCOIiK6LIknqofOWhAE7tN7ePH5HZm6LiNuBpwAHA2uncM7dEbERODAilmVm71F3tYhY07DrsKkUXpLmgj0XL+Cvzz6a/Vcs7bookqQGu1uQMLJk54aG/SPpo1ftmco5e9THTRgkSJImtuoxe/LRFx/FIfvu2XVRJEkTGEiQEBHrgCf0ccrnMvPFg8i7TyPt2v2M1pvyOZl5VM8LVC0MR/aRpyTtVn7tgOWcfewT+N1fP5BFC3a3nq6StPsZVEvCrVRTjk7V+gHlO95Ia8CKhv3Lxx038u9H1efcN8E5D+5y6Vqy34olrHrM4H6lCwbXZ7jU7seD7Bc96I84yHs20GsV+lwM9P4X+lwM+v/RoC63YN489lg8nz0WL2CvJQs5YOUSDt53T458/N7st2LJgHKRJLVhIEFCZp4yiOsMwE3A0cAqYMz4gIhYADwR2AbcNu6cR9Xn/Ou4c/an6mp052TjEUry9uf+atdFkCRJ0iy2u7X5XllvT+ux7wRgGXBdZg5N8ZxnjztGkiRJ2u3NyiAhIlZExGH1L/2jrQbuBV4UEUePOn4J8K767UfHnfNpYAh4bb2w2sg5ewN/Vr/92OBKL0mSJJWtiNmNIuItPDJN6NPr7TkR8Z/qf38zMz8x6pTfpfrj/m+Bl44kZuaDEfEKqmDh6oi4hGol5edTTXW6GvjC6Lwz8/aI+K/AhcANEfEFYJhqYbYDgb+aTastS5IkSbuqiCCBqqvPs8alHVe/RnyCKcjMyyLiWcBbgTOAJcCPgDcCF2bmTrMUZeYH6xma3gz8IVULyw+At2Xm3/b3USRJkqTZrYggITNP7PP4i4CLJth/LfCcPq/5ZeDL/ZwjSZIk7Y5m5ZgESZIkSTPHIEGSJEnSGAYJkiRJksYwSJAkSZI0hkGCJEmSpDEMEiRJkiSNYZAgSZIkaQyDBEmSJEljGCRIkiRJGsMgQZIkSdIYkZldl2G3FxH3LV26dJ/DDz+866JIkiRpN7Z27Vo2b958f2b+yq5cxyChBRFxO7AcWNdy1ofV2x+2nO9s5j3rj/erP96v/ni/+uc964/3qz/er/51cc8OAh7MzCfuykUMEnZjEbEGIDOP6ross4X3rD/er/54v/rj/eqf96w/3q/+eL/6N5vvmWMSJEmSJI1hkCBJkiRpDIMESZIkSWMYJEiSJEkawyBBkiRJ0hjObiRJkiRpDFsSJEmSJI1hkCBJkiRpDIMESZIkSWMYJEiSJEkawyBBkiRJ0hgGCZIkSZLGMEiQJEmSNIZBwm4mIp4cEX8aEVdGxB0RMRwRP4uIL0XESZOc+5KI+LeIeDgiNkTE1RHx3LbK3pWIWBgR50bEpyPiO/U9y4h4+QTnvLQ+pun1yjY/Q5umc79GnTsnn7EmEXHQJM/RJV2XsQsRcWBEfCoi1kfEUESsi4gLImLvrstWovr+ND1DP+26fF2IiDMj4oMR8Y2IeLC+FxdPcs5xEfGViLg/IjZFxPci4vURMb+tcnepn3s21+uuiPiViHh5RPxDRPwoIjbX32nfjIg/ioief1/PtmdsQdcF0MD9P8DvAz8AvgLcDxwKPB94fkScm5kXjj8pIv4SeBNwJ/BxYBHwIuDLEfEnmfmhlsrfhT2AC+p//wz4KfC4KZ77JeA7PdJvGEC5SjWt+zXHn7HJfBe4rEf6f7RdkK5FxCHAdcCjqf5//RB4JnAucFpEHJ+Z93VYxFJt4JH/l6M93HZBCvE24GlUn/9O4LCJDo6IFwCXAluAL1B9dz4P+ABwPHDWTBa2EH3ds9pcrbvOAj4K3A1cBfwEeAxwOvAJ4NkRcVaOWrF4Vj5jmelrN3oBLwV+vUf6s4BhYAjYf9y+44AEfgTsPSr9IOA+qgf6oK4/2wzes0XAs0fuC3BefT9ePsl9TuClXZd/ltyvOf2MTXBfDqrvy0Vdl6WUF/DP9T35k3Hp76/TP9Z1GUt7AeuAdV2Xo6QXcBLwZCCAE+tn5+KGY5cDP6+/H48elb6EKmBN4EVdf6bC7tmcrruAk6n+wJ83Ln0/qoAhgTNGpc/KZ8zuRruZzLwoM/93j/RrgKup/sA7btzuka4x787MX4w6Zx3wYWAxcM5MlLcEmTmcmZdn5t1dl2U2mOb9mtPPmKYmIg4GTqX6o/fD43a/A9gInB0Re7RcNM0ymXlVZt6S9V9ikzgT2Be4JDN/2QqcmVuofl0HeNUMFLMofd6zOS0zr8zML2fmjnHpPwU+Vr89cdSuWfmMGSTMLVvr7bZx6SfX26/2OOfyccdorKfX/QnfEhFnR8SBXReoUD5jE3tsRPxxRPxZvT2i6wJ1ZOQZuKLHl+9DwLXAMuDYtgs2CyyOiBfXz9C5EXFSqf2cCzRR/fR1YBNwXEQsbq9Is4Z11856/a01K58xxyTMERHxBOAUqgfx66PS9wAOAB5u+GX4lnq7asYLOTudO+799oj4BPD6+heCOc9nbEr+r/r1SxFxNfCSzPxJJyXqxqH19uaG/bdQtTSsAr7WSolmj/2Az45Luz0izqlbktWs8bnLzG0RcTvwFOBgYG2bBZsFrLtGiYgFwB/Wb0cHBLPyGbMlYQ6oI9PPUXXpOG90dw9gRb3d0HD6SPrKGSrebHU78CdU//H3AB4L/B5VN4k/Bj7VWcnK4zPWbBPVZANHAXvXr2dRDYQ7EfjaHOta47MyPZ+m+hFoP6r66KnAX1P1G788Ip7WXdFmBZ+7/ll39fZe4NeAr2TmP49Kn5XPmEFCgSaZzq7Xq3Fat7q5+bNUI+e/APzlNItVdB/FQd6zqcjMazLzQ5l5c2Zuysy7M/PvqQZ+/QL4v0v+Ym77fk1R0c9Yk125l5n588z888z8dmY+UL++TvVr+beAJwGTTi07h0S9nZXPykzJzPPrPtI/q+uj/8jMV1IN9l5KNbmAps/nbhzrrp1FxOuoZvD7IXB2v6fX26KeMbsblelWqtlepmp9r8Q6QLiYalqtLwIv7jEgaSR6XUFvk0W/pRjIPdtVmXlHRHwF+APgBKrp4UrU5v3aXZ6xJgO/l3Xz8yeAY6ieo/9vmmWbbSZ7VpaPO04T+xjVHy0ndF2QwvncDchcrbsi4jVUn/UHwCmZef+4Q2blM2aQUKDMPGVXr1H3i/s7qgDh74A/zMztPfLaGBF3AQdExP49+ow/ud429REuwiDu2QDdU2+LbWpt837tLs9Ykxm8l8U/RzPgpnrbND5lVj8rHfh5vZ1Lz9B03AQcTfXcrRm9o/4ufSLVINTb2i/arDSn6q6IeD3VWgf/QRUg/LzHYbPyGbO70W4oIhYBq6kChM8AZ/cKEEa5st6e1mPfs8cdo8kdU2+L+s/eMZ+x/o3M4DOXnqOr6u2p41csjYi9qLpNbgaub7tgs9Rv1Nu59AxNx0T10wlUM2pdl5lD7RVpVpszdVdE/ClVgPAd4KSGAAFm6TNmkLCbqQcp/wPwAuCTwDnjpxLsYWRO37dGxN6jrnUQ8BqqxT8+PfDCzmIR8Zs90iIi/jvVF/O99J7qbK7yGeshIo6pg/rx6ScDb6jftjEepAiZeStwBdWA29eM230+1S+Tn8nMjS0XrVgR8ZSI2KdH+hOAkVXM58wzNE2rqersF0XE0SOJEbEEeFf99qNdFKxU1l0QEW+nGqi8hqoF4d4JDp+Vz1i4ZsbuJSI+TbUa8L3AR+g9CObqzLx63Hl/BbyRain21VSLrv0+8CtUK59+aPxFdicR8RYeWYL+6VRL01/HI9NzfjMzPzHq+KTq8vDvwF1U/QyPp5rVYBPwu5l5RTulb1+/96s+Z04/Y73UUwU+hWqhwzvr5CN4ZE7tt2fmu3Y+c/cVEYdQPUuPBr5ENR3gMVSTAtwMHJeZ93VXwrJExHnAW6haYW4HHgIOAX6HajXXr1DVR8NdlbELEfFC4IX12/2A36b6Zfsbddq9mfnmccevphpfdAlwP/B8qhnsVgO/t7svMtbPPZvrdVdEvAS4CNgOfJDeYwnWZeZFo86Zfc/YIJZt9lXOi+o/bE7yOq/h3JdQ/dG7keqL5hrguV1/pkLu20Xjjv8f9f1ZT/UffhPVjAYfAg7u+vOUdr98xhrv4x8B/z/V1LkPU7Wo/IRqJrLf7Lp8Hd6Xx1G1LN0NDAM/phoUuE/XZSvtRTXt5Ofr+ucBqoWc7gH+hWq+9ui6jB3dl/MmqaPW9TjneKqg6hdU3dpupPpVfH7Xn6e0ezbX664p3Kuk+kF2Vj9jtiRIkiRJGsMxCZIkSZLGMEiQJEmSNIZBgiRJkqQxDBIkSZIkjWGQIEmSJGkMgwRJkiRJYxgkSJIkSRrDIEGSJEnSGAYJkiRJksYwSJAkSfo/7dexAAAAAMAgf+vdcyiLgJEEAABgJAEAABhJAAAARhIAAICRBAAAYCQBAAAYSQAAACZlSOBcrInF5AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 250,
       "width": 388
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.lineplot(pts, np.tanh(pts*np.pi)) ;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **ReLU (Rectified linear unit)**\n",
    "\n",
    "\\begin{equation*}\n",
    "f(z) = \\mathrm{max}(0,z)\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvgAAAH0CAYAAABICFkFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xd4HPW59vH7Ubdky71XbOOGiyzRCab3XkyxnBDenCQYTO81piUQEqqN4SQcOMFyo0OoAWxMC0Wy3CvuvVdZdX/vH1qfGDFrVtZqtuj7uS5d4517dvZRYuDWaIo55wQAAAAgMSRFewAAAAAAkUPBBwAAABIIBR8AAABIIBR8AAAAIIFQ8AEAAIAEQsEHAAAAEggFHwAAAEggFHwAAAAggVDwAQAAgARCwQcAAAASCAUfAAAASCAUfAAAACCBUPABAACABELBBwAAABIIBR8AAABIIBR8AAAAIIGkRHuAWGdmSyVlS1oW5VEAAACQ2LpJ2uGcO6guO6Hg/7zsRo0atejbt2+LaA8CAACAxDVv3jzt2bOnzvuh4P+8ZX379m1RWFgY7TkAAACQwPLy8lRUVLSsrvvhHHwAAAAggVDwAQAAgARCwQcAAAASCAUfAAAASCAUfAAAACCBUPABAACABELBBwAAABIIBR8AAABIIBR8AAAAIIHUueCbWUsz+y8ze8PMFpvZHjPbbmZfmNlvzMzzM8zsaDN7z8y2mFmJmc00sxvMLPkAZuhnZpPNbIOZlZrZAjO738wa1fX7AwAAAOJJSgT2MVTSWElrJU2RtEJSW0kXSvq7pDPMbKhzzu19g5mdJ+k1SaWSJknaIukcSU9IOia4z7CY2RGSPpWUKulVSSslnSjpPkknmdlJzrmyOn6PAAAAQFyIRMFfKOlcSe865wJ7V5rZXZK+lXSRqsv+a8H12ZL+JqlK0vHOue+D6+9VdVG/2Mwuc85N/LkPDh7tf1FSpqTznHNvB9cnSZoc/OwbJT0Sge8TAAAAiHl1PkXHOfepc+6dfct9cP06Sc8FXx6/T3SxpNaSJu4t98HtSyXdE3w5IsyPP05SX0nT9pb74L4Ckm4LvrzKzCzM/QEAAABxrb4vsq0ILiv3WXdicPmBx/bTJJVIOtrM0sPYf8h9OeeWqPq3C10ldQ9rWgAAADRobxWv1pw126M9Rp3UW8E3sxRJvwq+3LeA9w4uF9Z8j3OuUtJSVZ86FE4pD7mvoEXBZa8w9gUAAIAGbMbKbbr1lZm64NmvNOm7FdEe54BF4hz8UB6R1F/Se865D/dZ3zS4DPWj0d71zcL4jIjty8wKQ0R9wpgDAAAAcWzr7nJdXVCk8qrqs85vf22Wvl+2VQ+c11+N0mp9k8eoqpcj+GZ2naSbJc2X9Mvavj24dPvdyv99AQAAIAEFAk43Ti7W6m17frT+lcJVuuDZL7V88+4oTXZgIn4E38yukfSUpLmSTnLObamxyd6j6k3lLbvGdvsTsX055/K81geP7OeGMQsAAADi0LNTF2vqgo2e2eqtexSIs0PFET2Cb2Y3SBotabakE4J30qlpQXD5k/Pig+ftH6Tqi3KXhPGRIfcVdHBwGeocfQAAADRgXy7epMf/FboqPjZ0oA5qleXjRHUXsYJvZrer+kFVxaou9xtCbPppcHm6RzZE1fe0/yrMh1OF3JeZdVd18V+u8H5YAAAAQAOybnuprpswPeQR+t8ee5BO79/e36EiICIFP/iQqkckFar6tJxN+9n8VUmbJF1mZofus48MSQ8FX46tsf9MM+tjZl1q7OszSfMkDTGzc/fZPknSo8GXz+37FF0AAACgoiqgkeOLtHl3uWd+WLfmuu30+LzXSp3PwTezKyQ9oOon034u6TqP50otc869JEnOuR1m9ltVF/2pZjZR0hZVPw23d3D9pBrvP1zSFFUX+uP3rnTOVZnZlao+kv+qmb0qaYWkkyQdKulLVf9WAQAAAPg/j7w/X98v3+qZtWqcptHDcpWaXN+PjKofkbjI9qDgMlnSDSG2+UzSS3tfOOfeNLPjJN0t6SJJGZIWS7pJ0tO1OeLunPvGzA6TdL+kUyU1UfVpOQ9IeiTMU30AAADQQLw/a61e+GKpZ5Zk0tOXD1bb7Ayfp4qcOhd859woSaMO4H1fSjozzG2n6j+3vPTK50oaWtsZAAAA0LAs2bhLt746M2R+86m9dXSPVj5OFHnx+XsHAAAAoJb2lFfp6oIi7Sqr9MxP7NNGI47r4fNUkUfBBwAAQMJzzumeN2dr/rqdnnmn5o30+CWDlJQU8qSRuEHBBwAAQMKb+N1KvVa0yjNLS07Ss/m5apaZ5vNU9YOCDwAAgIQ2e/V2/eHtOSHz+87pp4Gdmvk4Uf2i4AMAACBhbS+p0IiCQpVXBjzzCwZ3VP4RNR+1FN8o+AAAAEhIgYDTza8Ua+WWPZ55r7aN9fAF/eXxDKe4RsEHAABAQnp+2hJ9PG+DZ5aVlqyxw/OUmRaJx0LFFgo+AAAAEs7XP2zWYx/OD5k/evFA9Wjd2MeJ/EPBBwAAQELZsKNU106YroDzzn99dDedPbCDv0P5iIIPAACAhFFZFdDICdO1aVeZZ57bpZnuOrOvz1P5i4IPAACAhPHYRwv07dItnlmLrDSNHpartJTErsCJ/d0BAACgwfhozjo9/9kSz8xMeuqyHHVo1sjnqfxHwQcAAEDcW755t25+ZUbI/IaTeunYg1v7OFH0UPABAAAQ10orqjRiXJF2llZ65kN6tda1J/b0earooeADAAAgrv3hrTmau3aHZ9ahaYaevDRHSUmJ9TCr/aHgAwAAIG5N/n6lJn2/0jNLTTaNyc9Vi6w0n6eKLgo+AAAA4tLcNTt075uzQ+b3nNVPg7s093Gi2EDBBwAAQNzZUVqhqwsKVVYZ8MzPGdRBvzqqq89TxQYKPgAAAOKKc063vTJTyzaXeOY9WmfpkQsHyKzhnHe/Lwo+AAAA4srfP1+qD+as88wapSbrueF5ykpP8Xmq2EHBBwAAQNz4btkWPfLB/JD5IxcN0MFtm/g4Ueyh4AMAACAubNxZpmsKilQVcJ75L4/sqvNyOvo8Veyh4AMAACDmVQWcrpswXRt2lnnmgzo11T1n9/V5qthEwQcAAEDMe/xfC/T1ks2eWbPMVI3Jz1V6SrLPU8UmCj4AAABi2ifz1mvMlB9C5k9cmqNOzTN9nCi2UfABAAAQs1ZuKdGNk4pD5tee2FMn9G7j40Sxj4IPAACAmFRaUaWrC4q0o7TSM/9Fz1a64eRePk8V+yj4AAAAiEkP/nOuZq3e7pm1y87QU5flKDmpYT7Man8o+AAAAIg5b0xfpYJvVnhmKUmmMfmD1bJxus9TxQcKPgAAAGLKgnU7defrs0Lmd57ZV3ldW/g4UXyh4AMAACBm7Cqr1IhxhSqtCHjmZw5op/93TDd/h4ozFHwAAADEBOecbn91ppZs2u2Zd2+VpUcvGigzzrvfHwo+AAAAYsJLXy3Tu7PWemYZqUl6dniummSk+jxV/KHgAwAAIOoKl2/Vw+/OC5k/fP4A9WmX7eNE8YuCDwAAgKjavKtMI8cXqTLgPPPLD++ii/I6+TxV/IpIwTezi83sGTP73Mx2mJkzs3Ehtn0pmO/v65MwP7fbz+xnYiS+PwAAANSPqoDTDZOKtXZ7qWfev2O2/nBOP5+nim8pEdrPPZIGSdolaZWkPvvZ9k1Jy0Jkv5TUXdL7tfz8GcH91jS7lvsBAACAj576ZJE+X7TJM8vOSNHY/DxlpCb7PFV8i1TBv1HVxX6xpOMkTQm1oXPuTXmUcTNrJuk2SeWSXqrl5xc750bV8j0AAACIoqkLNuiZTxeFzB+/JEedW2T6OFFiiEjBd879X6Gvw22LfimpkaSJzjnvH+MAAACQEFZv26MbJhXLeZ92rxHH99DJ/dr6O1SCiNQR/Ej4bXD53wfw3g5m9ntJLSVtlvS1c25mxCYDAABAxJRXBnR1QZG2lVR45kd2b6GbT+nl81SJIyYKvpkdJWmApIX7/jagFk4Jfu27z6mSrnDOrQhzhsIQ0f6uJwAAAEAtPfzuXM1Yuc0za9MkXU9fPlgpydzs8UDFyv9yvwsu/1bL95VIelBSnqTmwa+91wAcL+kTM8uK0IwAAACoo7dnrNH/fr3cM0tOMj1z+WC1aZLh81SJJepH8M2sqaRLdAAX1zrnNki6r8bqaWZ2qqQvJB0h6b8kPRXGvvJCzFcoKbc2cwEAAOCnFm/YqTteC30W9W2n9dYR3Vv6OFFiioUj+MMlZUp6PVIX1zrnKiX9PfhySCT2CQAAgAO3u6xSV40rUkl5lWd+ar+2+t2Q7j5PlZhioeDvvbj2+Qjvd2NwySk6AAAAUeSc052vz9LiDbs8864tM/XY0EF1uRsj9hHVgm9mR6j6AVkLnXNTI7z7I4PLJRHeLwAAAGph3L+X6+0Zazyz9JQkPZufq6aNUn2eKnFF+wj+3otr93trTDNramZ9zKx9jfVHmFmax/YnqvrhW5I0LiKTAgAAoNaKV27TA/+cGzJ/8Lz+OqRDUx8nSnwRucjWzM6XdH7wZbvg8igzeyn4503OuVtqvCdb0qWqvrj2f3/mIy6Q9GJwu1/vs/5RSYcEb4m5KrhuoKQTg3++1zn3VW2+FwAAAETG1t3luqagSBVV3k+zGprXSZcc1tnnqRJfpO6ikyPpihrruge/JGm5pFtq5PmqPj++Lk+ufVnV5f8wSWdISpW0XtJkSaOdc58f4H4BAABQB4GA042Ti7V62x7PvG/7bD14fn+fp2oYIlLwnXOjJI2q5XvGShob5rYvyeMWms65FyS9UJvPBQAAQP0bM2Wxpi7Y6Jk1SU/R2PxcZaQm+zxVwxDtc/ABAACQYL5YtEmPf7wwZP7Y0EHq1oobHdYXCj4AAAAiZu32Pbpu4nQ579Pu9bsh3XV6/3beISKCgg8AAICIqKgK6JqCIm3ZXe6ZH9atuW49rbfPUzU8FHwAAABExJ/em6+iFds8s1aN0zV6WK5Sk6mf9Y3/hQEAAFBn781aq//5cqlnlmTS05fnqG12hs9TNUwUfAAAANTJko27dNurM0PmN5/aW0f3aOXjRA0bBR8AAAAHrKS8UiPGFWlXWaVnflKfNhpxXA+fp2rYKPgAAAA4IM453fPGbC1Yv9Mz79S8kR6/JEdJSebzZA0bBR8AAAAHZMK3K/X69NWeWVpyksbm56lpZqrPU4GCDwAAgFqbtWq7Rr09J2T+h3P7aUCnpj5OhL0o+AAAAKiV7SUVGlFQqPKqgGd+4eCOGnZ4F5+nwl4UfAAAAIQtEHC6aXKxVm3d45n3bttED13QX2acdx8tFHwAAACE7blpP+iT+Rs8s8bpKXp2eK4y01J8ngr7ouADAAAgLF/9sEl/+XBByPzRiwaqR+vGPk4ELxR8AAAA/Kz1O0p13YTpCjjv/Mpjuumsge39HQqeKPgAAADYr4qqgK4dP12bdpV75rldmunOM/r6PBVCoeADAABgv/7y4QJ9u2yLZ9YiK01j8nOVlkKtjBX8PwEAAICQPpyzTs9PW+KZmUlPXZaj9k0b+TwV9oeCDwAAAE/LNu3WLZNnhMxvPLmXjj24tY8TIRwUfAAAAPxEaUWVRhQUaWdZpWd+XK/WGnlCT5+nQjgo+AAAAPiJ+96arXlrd3hmHZs10pOX5igpiYdZxSIKPgAAAH5k8ncrNfn7VZ5ZarJpTH6ummel+TwVwkXBBwAAwP+Zs2a77n1rdsj83rP7KadzMx8nQm1R8AEAACBJ2r6nQlcXFKmsMuCZnzuog355ZFefp0JtUfABAAAg55xufWWGlm8u8cx7tmmsP104QGacdx/rKPgAAADQ3z5foo/mrvfMMtOS9dzwXGWlp/g8FQ4EBR8AAKCB+3bpFj36wYKQ+SMXDVTPNk18nAh1QcEHAABowDbsLNXI8UWqCjjP/FdHddW5gzr4PBXqgoIPAADQQFVWBXTdhOnasLPMMx/UuZnuPquvz1Ohrij4AAAADdTj/1qofy/Z4pk1y0zVmGGDlZ6S7PNUqCsKPgAAQAP08dz1enbqD56ZmfTkpTnq1DzT56kQCRR8AACABmbllhLdNLk4ZH7tCT11fO82Pk6ESKLgAwAANCClFVUaUVCoHaWVnvmxB7fS9Sf38nkqRBIFHwAAoAG5/525mr16h2fWLjtDT16ao+QkHmYVzyj4AAAADcRrhas04dsVnllKkmlMfq5aNk73eSpEWkQKvpldbGbPmNnnZrbDzJyZjQuxbbdgHupr4gF8/tFm9p6ZbTGzEjObaWY3mBmXfQMAAEiav26H7n5zVsj8rjP7Kq9rcx8nQn2J1POG75E0SNIuSask9QnjPTMkvemxfnZtPtjMzpP0mqRSSZMkbZF0jqQnJB0jaWht9gcAAJBodpZWaMS4IpVWBDzzswa015XHdPN3KNSbSBX8G1Vd7BdLOk7SlDDeU+ycG1WXDzWzbEl/k1Ql6Xjn3PfB9fdK+lTSxWZ2mXOu1r8VAAAASATOOd3+2kwt3bTbM+/eKkuPXDRAZpx3nygicoqOc26Kc26Rc877Gcf152JJrSVN3Fvug/OUqvq3CpI0wueZAAAAYsaLXy7Te7PWeWYZqUkaOzxPTTJSfZ4K9SlSR/APRAcz+72klpI2S/raOTezlvs4Mbj8wCObJqlE0tFmlu6c834GMwAAQIIqXL5Ff3xvXsj8jxcMUO92TXycCH6IZsE/Jfj1f8xsqqQrnHPel3f/VO/gcmHNwDlXaWZLJR0iqbuk0H+7qz+7MEQUzvUEAAAAMWXTrjJdUzBdlQHvEyyGHdFFF+Z28nkq+CEat8kskfSgpDxJzYNfe8/bP17SJ2aWFea+mgaX20Pke9c3O6BJAQAA4lBVwOmGicVat6PUM+/fMVv3nd3P56ngF9+P4DvnNki6r8bqaWZ2qqQvJB0h6b8kPRWBj9t7tcjPXhvgnMvz3EH1kf3cCMwCAADgi6c+XqgvFm/yzJo2StXY/DxlpHI38UQVMw+6cs5VSvp78OWQMN+29wh90xB5do3tAAAAEtqUBRv09KeLQ+aPXzJInVtk+jgR/BYzBT9oY3AZ7ik6C4LLXjUDM0uRdJCkSklL6j4aAABAbFu1tUQ3TioOmV99fA+d1LetjxMhGmKt4B8ZXIZbyD8NLk/3yIZIypT0FXfQAQAAia6sskrXFBRpW0mFZ35U95a66ZSfHBNFAvK94JvZEWaW5rH+RFU/MEuSxtXImppZHzNrX+Ntr0raJOkyMzt0n+0zJD0UfDk2YsMDAADEqIffnacZq7zPSm7TJF1PXz5YKcmxdmwX9SEiF9ma2fmSzg++bBdcHmVmLwX/vMk5d0vwz49KOiR4S8xVwXUD9Z972t/rnPuqxkdcIOlFSf8r6dd7VzrndpjZb1Vd9Kea2URJWySdq+pbaL4qaVJdvz8AAIBY9lbxav3j6+WeWXKSafSwXLVuku7zVIiWSN1FJ0fSFTXWdQ9+SdJySXsL/suqLuyHSTpDUqqk9ZImSxrtnPu8Nh/snHvTzI6TdLekiyRlSFos6SZJT0fh6boAAAC+WbR+p+58fVbI/PbTe+vwg1r4OBGiLSIF3zk3StKoMLd9QdILtdz/S5Je2k/+paQza7NPAACAeLe7rFIjCopUUl7lmZ92SFv99tjunhkSFydiAQAAxCHnnO54fZYWb9jlmXdtmanHhg6SmXnmSFwUfAAAgDj08r+X650Zazyz9JQkjc3PU3ZGqs9TIRZQ8AEAAOLM9BVb9eA/54bMHzy/v/p1yA6ZI7FR8AEAAOLI1t3luqagSBVV3vcRueTQTrrk0M4+T4VYQsEHAACIE4GA0w2TirVme6ln3q99th44r7/PUyHWUPABAADixDOfLtZnCzd6Zk0yUjR2eK4yUpN9ngqxhoIPAAAQBz5ftFFPfrIwZP6XoYPUtWWWjxMhVlHwAQAAYtyabXt0/cRihXp85++HdNdph7TzdyjELAo+AABADCuvDGjk+CJt2V3umR/erYVuPa23z1MhllHwAQAAYtif3p+nohXbPLNWjdM1ethgpSRT6fAf/G0AAACIUf+cuUYvfrnMM0sy6ZnLB6tNdoa/QyHmUfABAABi0A8bd+n2V2eGzG85rbeO6tHSx4kQLyj4AAAAMaakvFIjxhVqd3mVZ35y3za6akgPn6dCvKDgAwAAxBDnnO5+Y7YWrt/lmXdu0Uh/HZqjpCTzeTLECwo+AABADBn/7Qq9MX21Z5aWkqSx+Xlqmpnq81SIJxR8AACAGDFr1Xbd//bckPn95x6i/h2b+jgR4hEFHwAAIAZsKynXiIJClVcFPPMLczvqssM6+zwV4hEFHwAAIMoCAaebJs/Qqq17PPM+7Zro4fMHyIzz7vHzKPgAAABRNvazH/Tp/A2eWeP0FD2bn6tGack+T4V4RcEHAACIoq9+2KS/frQgZP7niweqe+vGPk6EeEfBBwAAiJL1O0p13YTpCjjv/P8dc5DOHNDe36EQ9yj4AAAAUVBRFdDI8UXatKvcM8/r2lx3ntnH56mQCCj4AAAAUfDnD+bru2VbPbOWWWkaMyxXqclUNdQef2sAAAB89sHstfrb50s9MzPpqcsGq13TDJ+nQqKg4AMAAPho6abduvWVmSHzm07upV8c3MrHiZBoKPgAAAA+Ka2o0ohxhdpZVumZH9+7ta45oafPUyHRUPABAAB8cu+bszV/3U7PrGOzRnrikhwlJfEwK9QNBR8AAMAHk75boVcKV3lmqcmmZ/Nz1TwrzeepkIgo+AAAAPVszprtuvetOSHz+87up0Gdm/k4ERIZBR8AAKAebd9ToRHjilReGfDMz8vpoOFHdvV5KiQyCj4AAEA9cc7plldmaMWWEs+8Z5vG+uMFA2TGefeIHAo+AABAPfnvaUv0r7nrPbPMtGQ9NzxXWekpPk+FREfBBwAAqAffLNmsP3+4IGT+yEUD1bNNEx8nQkNBwQcAAIiwDTtLNXLCdFUFnGd+xVFdde6gDj5PhYYiIgXfzC42s2fM7HMz22FmzszGhdj2YDO73cw+NbOVZlZuZuvN7C0zO6GWn9st+FmhviZG4vsDAAAIV2VVQNeOn66NO8s885zOzXT3Wf18ngoNSaRO+rpH0iBJuyStktRnP9s+KOlSSXMlvSdpi6Teks6VdK6ZXe+ce7qWnz9D0pse62fXcj8AAAB18td/LdQ3S7d4Zs0zUzUmP1dpKZxEgfoTqYJ/o6qL/WJJx0masp9tP5D0qHNu+r4rzew4Sf+S9JiZveKcW1uLzy92zo2q3cgAAACR9a+56zV26g+emZn0xKU56tiskc9ToaGJyI+PzrkpzrlFzjnvE81+vO1LNct9cP1nkqZKSpN0dCTmAgAA8MuKzSW6aXJxyPzaEw/W8b3b+DgRGqpYuy9TRXBZWcv3dTCz30tqKWmzpK+dczMjOhkAAEAIpRVVGlFQqJ2l3hXm2INb6fqTDvZ5KjRUMVPwzayrpJMklUiaVsu3nxL82nd/UyVd4ZxbEZEBAQAAQrj/nTmas2aHZ9a+aYaevDRHyUk8zAr+iImCb2bpkgokpUu6zTm3Ncy3lqj6ot03JS0JrhsoaZSkEyR9YmY5zrndYcxQGCLa3wXDAACggXu1cJUmfLvSM0tJMo0elquWjdN9ngoNWdQv4TazZEkvSzpG0iRJfwn3vc65Dc65+5xzRc65bcGvaZJOlfSNpJ6S/qs+5gYAAJi/bofueXNWyPzus/oqr2tzHycConwEP1jux0kaKmmypOHhXKj7c5xzlWb2d0lHSBoi6akw3pMXYsZCSbl1nQkAACSWnaUVGjGuSKUVAc/8rIHt9euju/k7FKAoHsE3sxRJEyRdJmm8pGHOudpeXLs/G4PLrAjuEwAAQM453fbqTC3d5H0WcPfWWXr0ooEy47x7+C8qR/DNLE3VR+zPk/QPSVc657x//D1wRwaXS/a7FQAAQC298MVSvT97nWfWKDVZzw3PU+P0mLjUEQ2Q70fwgxfUvqHqcv+Cwij3ZtbUzPqYWfsa648I/rBQc/sTVf3wLan6FCAAAICI+H7ZFj3y/vyQ+R8v7K9ebZv4OBHwYxH50dLMzpd0fvBlu+DyKDN7KfjnTc65W4J/fk7SmZI2SVot6T6PX19Ndc5N3ef1BZJelPS/kn69z/pHJR0SvCXmquC6gZJODP75XufcVwf0TQEAANSwaVeZrhlfpMqA9yWD+Ud00QWDO/k8FfBjkfrdUY6kK2qs6x78kqTlkvYW/IOCy1aS7tvPPqeG8bkvq7r8HybpDEmpktar+vSf0c65z8PYBwAAwM+qCjhdP3G61u8o88wHdGyqe8/u5/NUwE9FpOA750ap+t7z4Wx7/AHs/yVJL3msf0HVp/kAAADUqyc/XqgvF2/2zJo2StWz+bnKSE32eSrgp6J+H3wAAIBYN2X+Bj3z6eKQ+ROXDlLnFpk+TgSERsEHAADYj1VbS3TDpOKQ+TUn9NCJfdr6OBGwfxR8AACAEMoqq3R1QZG276nwzI/u0VI3ndLb56mA/aPgAwAAhPDQP+dp5qrtnlmbJul66rLBSk7iYVaILRR8AAAAD28Vr9bL/17umSUnmcbk56p1k3SfpwJ+HgUfAACghoXrd+qO12aFzO88o48O69bCx4mA8FHwAQAA9rGrrFJXjSvUnooqz/z0Q9rpN784yDMDYgEFHwAAIMg5pztem6klG3d75t1aZurPQwfKjPPuEbso+AAAAEH/+Hq5/jlzrWeWnpKkZ/PzlJ2R6vNUQO1Q8AEAACQVrdiqh96dGzJ/6Pz+6tch28eJgANDwQcAAA3elt3lGllQpIoq55lfemhnDT20s89TAQeGgg8AABq0qoDT9ROna832Us+8X/ts3X/eIT5PBRw4Cj4AAGjQnvl0kT5ftMkza5KRorHDc5WRmuzzVMCBo+ADAIAGa9rCjXrqk0Uh878OHaSuLbN8nAioOwo+AABokNZs26PrJ06X8z7tXr8/rrtOPaSdv0MBEUDBBwAADU55ZUBXFxRpa0mFZ374QS1066m9fZ4KiAwKPgAAaHD++N48Fa/c5pm1apyu0ZcPVkoyNQnxib/mI22wAAAgAElEQVS5AACgQXlnxhq99NUyzyzJpNHDBqtNdoa/QwERRMEHAAANxuINu3THazND5ree1kdHdm/p40RA5FHwAQBAg1BSXqmrCwq1u7zKMz+5b1v9fkh3n6cCIo+CDwAAEp5zTne9PksL1+/yzDu3aKS/Dh2kpCTzeTIg8ij4AAAg4RV8s0JvFq/xzNJSkjQ2P09NM1N9ngqoHxR8AACQ0Gau2qYH3pkbMn/g3EPUv2NTHycC6hcFHwAAJKxtJeUaMa5I5VUBz/yi3E669LDOPk8F1C8KPgAASEiBgNONk4q1etsez7xPuyZ66Pz+MuO8eyQWCj4AAEhIYz/7QVMWbPTMGqen6Nn8XDVKS/Z5KqD+UfABAEDC+XLxJv31owUh88cuHqjurRv7OBHgHwo+AABIKOu2l+q6CdMVcN75b35xkM4Y0N7foQAfUfABAEDCqKgKaOT4Im3eXe6ZH9q1ue44o4/PUwH+ouADAICE8ej78/X98q2eWcusNI0elqvUZOoPEht/wwEAQEL4YPZa/f2LpZ5ZkklPXz5Y7Zpm+DwV4D8KPgAAiHtLN+3Wra/MDJnfdEovHdOzlY8TAdFDwQcAAHFtT3mVRowr1M6ySs/8hN6tdfXxPX2eCogeCj4AAIhbzjnd8+ZszV+30zPv2KyRnrg0R0lJPMwKDQcFHwAAxK1J363Ua0WrPLO05CSNHZ6rZplpPk8FRFdECr6ZXWxmz5jZ52a2w8ycmY37mfccbWbvmdkWMysxs5lmdoOZ1fqRcmbWz8wmm9kGMys1swVmdr+ZNTrw7woAAMSy2au3676354TM7z2nnwZ2aubjREBsSInQfu6RNEjSLkmrJO33BrNmdp6k1ySVSpokaYukcyQ9IekYSUPD/WAzO0LSp5JSJb0qaaWkEyXdJ+kkMzvJOVdWy+8HAADEsO17KnR1QZHKKwOe+fk5HTT8iC4+TwXEhkidonOjpF6SsiWN2N+GZpYt6W+SqiQd75z7jXPuVkk5kr6WdLGZXRbOhwaP9r8oKVPSxc65Yc652yUdoeofII4JzgYAABJEIOB08+QZWrGlxDM/uE1j/fHCATLjvHs0TBEp+M65Kc65Rc65EA+F/pGLJbWWNNE59/0++yhV9W8CpJ/5IWEfx0nqK2mac+7tffYVkHRb8OVVxj/hAAAkjP/+fIk+nrfeM8tKS9bY4XnKTIvUSQpA/InGRbYnBpcfeGTTJJVIOtrM0uuyL+fcEkkLJXWV1P0A5gQAADHm30s2688fzA+ZP3LRQPVs09jHiYDYE40fb3sHlwtrBs65SjNbKukQVZfyeQe6r6BFqj51qJekH/a3IzMrDBHt93oCAADgjw07SjVy/HQFQpwv8Ouju+mcQR38HQqIQdE4gt80uNweIt+7PpzL3iO5LwAAEKMqqwIaOWG6Nu3yvm9GTudmuuvMvj5PBcSmWDxBbe/58uGczx+xfTnn8jx3UH1kPzcCswAAgAP02EcL9O3SLZ5Z88xUjcnPVVoKj/cBpOgcwd97VL1piDy7xnZ+7QsAAMSgj+as0/OfLfHMzKSnLhusjs149A2wVzQK/oLgslfNwMxSJB0kqVKS9z/JYe4r6ODgMtQ5+gAAIIYt37xbN78yI2R+/UkHa0iv1j5OBMS+aBT8T4PL0z2yIaq+p/1XYT6cKuS+zKy7qov/coX3wwIAAIghpRVVGjGuSDtLKz3zYw9upWtPPNgzAxqyaBT8VyVtknSZmR26d6WZZUh6KPhy7L5vMLNMM+tjZjUfSfeZqu+0M8TMzt1n+yRJjwZfPhfm/fkBAEAMGfX2HM1du8Mza980Q09dNljJSTzqBqgpIhfZmtn5ks4PvmwXXB5lZi8F/7zJOXeLJDnndpjZb1Vd9Kea2URJWySdq+rbXr4qaVKNjzhc0hRVF/rj9650zlWZ2ZWqPpL/qpm9KmmFpJMkHSrpS0lPROJ7BAAA/nnl+5Wa+N1Kzyw12TQmP1ctstJ8ngqID5G6i06OpCtqrOuu/zxgarmkW/YGzrk3zew4SXdLukhShqTFkm6S9HRtjrg7574xs8Mk3S/pVElNgp/3gKRHwjzVBwAAxIh5a3fonjdnh8zvPrOvcrs093EiIL5EpOA750ZJGlXL93wp6cwwt52q/9zy0iufK2lobT4fAADEnh2lFRoxrlBllQHP/KyB7XXF0d38HQqIM9wwFgAAxATnnG57ZaaWbS7xzHu0ztKjFw2UGefdA/tDwQcAADHhhS+W6oM56zyzRqnJGjs8T43TY/EZnUBsoeADAICo+27ZFv3p/fkh8z9dOEC92jbxcSIgflHwAQBAVG3aVaaR44tUFfC+x8bwI7vo/MEdfZ4KiF8UfAAAEDVVAafrJkzX+h3eN70b2Kmp7j27n89TAfGNgg8AAKLmiX8t1Fc/bPbMmjZK1ZhhuUpPSfZ5KiC+UfABAEBUfDp/vUZPWRwyf/LSHHVukenjREBioOADAADfrdxSohsnzQiZjzyhp07o08bHiYDEQcEHAAC+Kqus0jXji7R9T4VnfnSPlrrxlF4+TwUkDgo+AADw1QPvzNXMVds9s7bZ6Xr68sFKTuJhVsCBouADAADfvDF9lQq+WeGZpSSZxgzLVavG6T5PBSQWCj4AAPDFwvU7ddfrs0Pmd5zRR4d2a+HjREBiouADAIB6t6usUleNK9SeiirP/Iz+7fSbXxzk81RAYqLgAwCAeuWc0+2vzdSSjbs984NaZenPFw+UGefdA5FAwQcAAPXqpa+W6d2Zaz2zjNQkPZufqyYZqT5PBSQuCj4AAKg3hcu36uF354XMHzp/gPq2z/ZxIiDxUfABAEC92LyrTCPHF6ky4Dzzyw7rrIvzOvk8FZD4KPgAACDiqgJON0wq1trtpZ75IR2yNercQ3yeCmgYKPgAACDinv5kkT5ftMkza5KRorH5ecpITfZ5KqBhoOADAICImrpgg57+dFHI/PFLctSlZaaPEwENCwUfAABEzOpte3TjpGI579PuddVxPXRKv7b+DgU0MBR8AAAQEeWVAV1TUKStJRWe+REHtdAtp/byeSqg4aHgAwCAiPjje/NUvHKbZ9a6SbqeGTZYKclUD6C+8U8ZAACos7dnrNFLXy3zzJKTTKMvH6w2TTL8HQpooCj4AACgThZv2Kk7XpsZMr/1tN46ontLHycCGjYKPgAAOGC7yyo1YlyRSsqrPPNT+rXV74d093kqoGGj4AMAgAPinNNdb8zSog27PPMuLTL1l6GDZGY+TwY0bBR8AABwQMZ9s0JvFa/xzNJSkvRsfq6aNkr1eSoAFHwAAFBrM1Zu04PvzA2ZP3jeIerfsamPEwHYi4IPAABqZevucl1dUKTyqoBnfnFeJ11yaGefpwKwFwUfAACELRBwunFysVZv2+OZ92nXRA+e15/z7oEoouADAICwjZmyWFMXbPTMmqSnaOzwPDVKS/Z5KgD7ouADAICwfLFokx7/eGHI/LGhA3VQqywfJwLghYIPAAB+1rrtpbp+4nQ5553/9tiDdHr/9v4OBcATBR8AAOxXRVVA14wv0ubd5Z75Yd2a67bT+/g8FYBQolLwzezXZuZ+5sv7kXg/3dey/exjXX1/LwAAJLpH3p+vwuVbPbNWjdM0eliuUpM5ZgjEipQofW6xpPtDZMdKOlHS+7XY33ZJT3qs9360HgAACMt7s9bqhS+WemZJJj192WC1zc7weSoA+xOVgu+cK1Z1yf8JM/s6+Mf/rsUutznnRtV1LgAA8B9LNu7Sba/ODJnffGpvHd2zlY8TAQhHTP0+zcz6SzpS0mpJ70Z5HAAAGqw95VW6uqBIu8oqPfMT+7TRiON6+DwVgHBE6xSdUH4fXL7gnAvrHPygdDMbLqmLpN2SZkqaVst9AAAASc453f3mLM1ft9Mz79S8kR6/ZJCSkniYFRCLYqbgm1kjScMlBST9vZZvbyfp5RrrlprZlc65z8L8/MIQEbcFAAA0KBO/W6nXi1Z7ZmnJSXo2P1fNMtN8ngpAuGLpFJ1LJDWT9L5zbmUt3veipJNUXfKzJA2Q9LykbpLeN7NBEZ4TAICENXv1dv3h7Tkh8/vO6aeBnZr5OBGA2oqZI/iSfhdcPl+bNznnat6NZ7akq8xsl6SbJY2SdEEY+8nzWh88sp9bm5kAAIhH20sqNKKgUOWVAc/8gsEdlX9EF5+nAlBbMXEE38z6STpa0ipJ70Vot88Fl0MitD8AABJWIOB08yvFWrllj2feq21jPXxBf5lx3j0Q62Ki4OvAL67dnw3BZVaE9gcAQMJ6ftoSfTxvg2eWlZasscPzlJkWS7/4BxBK1Au+mWVI+qWqL659IYK7Piq4XBLBfQIAkHC+/mGzHvtwfsj80YsHqkfrxj5OBKAuol7wJQ2V1FzSe6EurjWzVDPrY2Y9aqw/xMxaeGzfVdLo4MtxkR4YAIBEsWFHqa6dMF0B553/+uhuOntgB3+HAlAnsfC7tr0X1+7vybUdJc2TtFzVd8fZa6ikO8xsiqSlknZK6iHpLEkZqj6f/y8RnhcAgIRQWRXQyPHTtWlXmWc+uEsz3XVmX5+nAlBXUS34ZtZX0i904BfXTpHUW9JgVZ+SkyVpm6QvVH1f/JedcyGOSQAA0LA99uECfbtsi2fWIitNY4blKi0lFn7ZD6A2olrwnXPzJP3s5fjOuWVe2wUfYhXWg6wAAMB/fDhnnZ6f5n2Zmpn01GU56tCskc9TAYgEfiwHAKCBWb55t255ZUbI/IaTeunYg1v7OBGASKLgAwDQgJRWVOmqcUXaWVrpmQ/p1VrXntjT56kARBIFHwCABuQPb83RvLU7PLMOTTP05KU5SkriYVZAPKPgAwDQQEz+fqUmfe95R2qlJpvG5OeqRVaaz1MBiDQKPgAADcDcNTt075uzQ+b3nNVPg7s093EiAPWFgg8AQILbUVqhqwsKVVYZ8MzPGdRBvzqqq89TAagvFHwAABKYc063vjJDyzaXeOY9WmfpkQsHyIzz7oFEQcEHACCB/f3zpfpwznrPrFFqsp4bnqes9Fh4sD2ASKHgAwCQoL5dukWPfDA/ZP7IRQN0cNsmPk4EwA8UfAAAEtDGnWUaOb5IVQHnmf/yyK46L6ejz1MB8AMFHwCABFNZFdB1E6Zrw84yz3xQp6a65+y+Pk8FwC8UfAAAEswTHy/U10s2e2bNMlM1Jj9X6SnJPk8FwC8UfAAAEsgn89ZrzJQfQuZPXJqjTs0zfZwIgN8o+AAAJIiVW0p046TikPm1J/bUCb3b+DgRgGig4AMAkABKK6o0oqBQO0orPfNjerbUDSf38nkqANFAwQcAIAE88M+5mr16h2fWLjtDT102WMlJPMwKaAgo+AAAxLnXi1Zp/DcrPLOUJNOY/MFq1Tjd56kARAsFHwCAOLZg3U7d9caskPmdZ/ZVXtcWPk4EINoo+AAAxKmdpRUaMa5QpRUBz/zMAe30/47p5u9QAKKOgg8AQBxyzumO12ZpyabdnvlBrbL06EUDZcZ590BDQ8EHACAOvfjlMr07a61nlpGapLHDc9UkI9XnqQDEAgo+AABxpnD5Vv3xvXkh84fPH6A+7bJ9nAhALKHgAwAQRzbvKtPI8UWqDDjP/PLDu+iivE4+TwUgllDwAQCIE1UBp+snFmvt9lLPvH/HbP3hnH4+TwUg1lDwAQCIE099skhfLN7kmWVnpGhsfp4yUpN9ngpArKHgAwAQB6Yu2KBnPl0UMn/8khx1bpHp40QAYhUFHwCAGLd62x7dMKlYzvu0e404vodO7tfW36EAxCwKPgAAMaysskpXFxRpW0mFZ35k9xa6+ZRePk8FIJZR8AEAiGEPvztPM1Zu88zaNEnX05cPVkoy/zkH8B/8GwEAgBj1VvFq/ePr5Z5ZcpLpmcsHq02TDJ+nAhDrKPgAAMSgRet36s7XZ4XMbzutt47o3tLHiQDECwo+AAAxZndZpUYUFKmkvMozP7VfW/1uSHefpwIQLyj4AADEEOec7nx9lhZv2OWZd22ZqceGDpKZ+TwZgHhBwQcAIIaM+/dyvT1jjWeWnpKkZ/Nz1bRRqs9TAYgnUSv4ZrbMzFyIr3W13FcnM/sfM1tjZmXBfT9pZs3ra34AACKteOU2PfDPuSHzB8/rr0M6NPVxIgDxKCXKn79d0pMe671/L+nBzHpI+kpSG0lvSZov6XBJ10s63cyOcc5tjsCsAADUm627y3VNQZEqqryfZjU0r5MuOayzz1MBiEfRLvjbnHOj6riPZ1Vd7q9zzj2zd6WZPS7pRkkPS7qqjp8BAEC9CQScbphUrNXb9njmfdo10YPn9/d5KgDxKq7PwTez7pJOlbRM0pga8R8k7Zb0SzPL8nk0AADCNnrKYn22cKNn1iQ9Rc8Nz1NGarLPUwGIV9E+gp9uZsMldVF1GZ8paZpzzvu+YD91YnD5kXMusG/gnNtpZl+q+geAIyV9EqGZAQCImC8WbdITHy8MmT82dJC6teI4FYDwRbvgt5P0co11S83sSufcZ2G8v3dwGerfjItUXfB76WcKvpkVhoj6hDEHAAC1tnb7Hl03cbqc92n3+t2Q7jq9fzt/hwIQ96J5is6Lkk5SdcnPkjRA0vOSukl638wGhbGPvbcS2B4i37u+2YGPCQBA5JVXBnRNQZG27C73zA/r1ly3ntbbMwOA/YnaEXzn3P01Vs2WdJWZ7ZJ0s6RRki6o48fsfQpIiGMjP5onz3MH1Uf2c+s4BwAAP/LI+/NVtGKbZ9aqcZpGD8tVanJcXyoHIEpi8d8czwWXQ8LYdu8R+lA3Bc6usR0AAFH37sy1+p8vl3pmSSY9fflgtc3O8HkqAIkiFgv+huAynCuKFgSXvULkBweXoa9eAgDARz9s3KXbXp0RMr/51N46ukcrHycCkGhiseAfFVwuCWPbKcHlqWb2o+/FzJpIOkbSHkn/jtx4AAAcmJLySl09rki7y71vFndSnzYacVwPn6cCkGiiUvDN7BAza+Gxvquk0cGX4/ZZn2pmfYJPrf0/zrkfJH2k6gtzr6mxu/tV/VuAfzjndkdwfAAAas05p3vemK0F63d65p2aN9Ljl+QoKck8cwAIV7Qush0q6Q4zmyJpqaSdknpIOktShqT3JP1ln+07Sponabmqy/y+rpb0laSnzeyk4HZHSDpB1afm3F1v3wUAAGGa8O1KvT59tWeWlpyksfl5apqZ6vNUABJRtAr+FFXfw36wqk/JyZK0TdIXqr4v/svOhbor8I85534ws0MlPSDpdElnSlor6WlJ9zvntkR+fAAAwjdr1XaNentOyPwP5/bTgE6h7hcBALUTlYIffIhVOA+y2rv9Mv3nlpde+UpJV9Z9MgAAImt7SYVGFBSqvCrgmV84uKOGHd7F56kAJLJYvMgWAICEEAg43TS5WKu27vHMe7VtrIcu6C8zzrsHEDkUfAAA6snYz37QJ/M3eGaN01M0dnieMtOi9sxJAAmKgg8AQD346odN+utHC0Lmj140UD1aN/ZxIgANBQUfAIAIW7+jVNdNmK5AiNtFXHlMN501sL2/QwFoMCj4AABEUEVVQCPHF2nTrnLPPLdLM915Rl+fpwLQkFDwAQCIoMc+XKDvlm31zFpkpWlMfq7SUvjPL4D6w79hAACIkA9mr9N/T1vimZlJT12Wo/ZNG/k8FYCGhoIPAEAELNu0W7e+MiNkfuPJvXTswa19nAhAQ0XBBwCgjkorqjSioEg7yyo98+N6tdbIE3r6PBWAhoqCDwBAHd331mzNW7vDM+vQNENPXpqjpCQeZgXAHxR8AADqYPJ3KzX5+1WeWWqy6dnheWqelebzVAAaMgo+AAAHaM6a7br3rdkh83vP7qeczs18nAgAKPgAAByQ7XsqdHVBkcoqA575uYM66JdHdvV5KgCg4AMAUGvOOd36ygwt31zimfds01h/unCAzDjvHoD/KPgAANTS3z5foo/mrvfMMtOS9dzwXGWlp/g8FQBUo+ADAFAL3yzZrEc/WBAy/9OFA9SzTRMfJwKAH6PgAwAQpg07SzVywnRVBZxn/qujuuq8nI4+TwUAP0bBBwAgDJVVAV03Ybo27izzzAd1bqa7z+rr81QA8FMUfAAAwvDXfy3Uv5ds8cyaZaZqzLDBSk9J9nkqAPgpCj4AAD/j47nrNXbqD56ZmfTkpTnq1DzT56kAwBsFHwCA/VixuUQ3TS4OmV97Qk8d37uNjxMBwP5R8AEACKG0okpXjy/UjtJKz/wXPVvp+pN7+TwVAOwfBR8AgBDuf2euZq/e4Zm1y87QU5flKDmJh1kBiC0UfAAAPLxWuEoTvl3hmaUkmcbk56pl43SfpwKAn0fBBwCghvnrdujuN2eFzO86s6/yujb3cSIACB8FHwCAfewsrdCIcUUqrQh45mcNaK8rj+nm71AAUAsUfAAAgpxzuv21mVq6abdn3r1Vlh65aIDMOO8eQOyi4AMAEPQ/Xy7Te7PWeWYZqUl6dniummSk+jwVANQOBR8AAEnfL9uiP703L2T+xwsGqE+7bB8nAoADQ8EHADR4m3aVaeT46aoMOM982BFddGFuJ5+nAoADQ8EHADRoVQGn6ydO17odpZ55/47Zuu/sfj5PBQAHjoIPAGjQnvp4ob5cvNkzy85I0dj8PGWkJvs8FQAcOAo+AKDBmrJgg57+dHHI/IlLc9S5RaaPEwFA3VHwAQAN0qqtJbpxUnHI/Orje+ikvm19nAgAIoOCDwBocMoqq3RNQZG2lVR45kd1b6mbTunl81QAEBlRKfhm1tLM/svM3jCzxWa2x8y2m9kXZvYbMwt7LjNbZmYuxJf3zYwBAA3aQ/+cpxmrtntmbZqk6+nLByslmWNgAOJTSpQ+d6iksZLWSpoiaYWktpIulPR3SWeY2VDnnPf9yn5qu6QnPdbvisCsAIAE8lbxar387+WeWXKSafSwXLVuku7zVAAQOdEq+AslnSvpXedcYO9KM7tL0reSLlJ12X8tzP1tc86NivSQAIDEsmj9Tt3x2qyQ+e2n99bhB7XwcSIAiLyo/P7ROfepc+6dfct9cP06Sc8FXx7v+2AAgIS1u6xSV40r1J6KKs/81H5t9dtju/s8FQBEXrSO4O/P3iueKmvxnnQzGy6pi6TdkmZKmuac8/63OACgQXHO6Y7XZ+mHjbs9864tM/XY0EEyM58nA4DIi6mCb2Ypkn4VfPlBLd7aTtLLNdYtNbMrnXOfhfnZhSGiPrWYAwAQg/7x9XK9M2ONZ5aekqSx+Xlq2ijV56kAoH7E2i0CHpHUX9J7zrkPw3zPi5JOUnXJz5I0QNLzkrpJet/MBtXDnACAODF9xVY99O7ckPmD5/dXvw7ZPk4EAPUrZo7gm9l1km6WNF/SL8N9n3Pu/hqrZku6ysx2Bfc3StIFYewnL8RchZJyw50HABA7tuwu1zUFRaqo8r4p2yWHdtIlh3b2eSoAqF8xcQTfzK6R9JSkuZJOcM5ticBu916sOyQC+wIAxJlAwOmGScVas73UM+/bPlsPnNff56kAoP5FveCb2Q2SRqv6yPsJwTvpRMKG4DIrQvsDAMSRZz5drGkLN3pmTdJTNDY/VxmpyT5PBQD1L6oF38xul/SEpGJVl/sNP/OW2jgquFwSwX0CAOLAtIUb9eQnC0Pmf7lkkLq14vgPgMQUtYJvZveq+qLaQkknOec27WfbVDPrY2Y9aqw/xMx+8kQSM+uq6t8KSNK4CI4NAIhxa7bt0fUTpyvUs9B/P6S7Tjuknb9DAYCPonKRrZldIekBSVWSPpd0nce9h5c5514K/rmjpHmSlqv67jh7DZV0h5lNkbRU0k5JPSSdJSlD0nuS/lIv3wQAIOaUVwZ0zfgibS2p8MwP79ZCt57W2+epAMBf0bqLzkHBZbKkG0Js85mkl35mP1Mk9ZY0WNWn5GRJ2ibpC1XfF/9l50IdwwEAJJo/vT9P01ds88xaNU7X6GH/v707D7ajqhM4/v1lJYQQFolAYIgJISACIaFEyIgsiigIbggaEBitkUWIojMuiIYatRxHBVmEGVEoZVMyhQwWmyWbQikaNlnCmsgSICZhyR6SnPmjO/J89H0vL3mvu2+/76eqq9/r033e7546r+/v9j19ek8GDaz89jNJ6lOVJPgppelk01eu6/5zgDdc4s8fYrVOD7KSJDXbrx+YyyV3ziksGxBw3sf3ZNSmG5UblCRVwMsYkqS298S8xXxpxgMty7/43gnsM27LEiOSpOqY4EuS2trSlas4+fKZLFm5urD83buM4sT9xhWWSVITmeBLktpWSokzrnmQx15cXFi+/RbD+P6RExkw4A2jPCWpsUzwJUlt64q7n+aae58rLBsyaAAXTp3MyI0HlxyVJFXLBF+S1JYeePZlzvq/h1uWT//Arrxt9MgSI5KkejDBlyS1nZeXruSky+5h5eo1heUfnjSaj799+5KjkqR6MMGXJLWVNWsSp//yfp57eVlh+YQ3j+BbH9yNggcoSlK/YIIvSWorF97+JLfMmldYtsnQQVx4zCSGDRlYclSSVB8m+JKktnHXE/P5/s2Ptiz/7kd3Z+xWm5QYkSTVjwm+JKktvPDKck676l7WpOLyf5nyFt6/2zblBiVJNWSCL0mqvddWr+GzV9zD/MUrC8sn77A5X3n/ziVHJUn1ZIIvSaq97944iz//9aXCsi2GD+H8T+zJ4IG+pUkSmOBLkmruxgef58e/m11YFgHnHr0n24wcVnJUklRfJviSpNqaPX8J/3b1Ay3LT3/3Tvzz+DeVGJEk1Z8JviSplpatXM1Jl81k0YpVheX7T9iKUw7YseSoJKn+TPAlSbX09WsfZNYLiwrLRm82jLM/NpEBA3yYlSR1ZoIvSaqdX/zpaa6e+Wxh2eCBwQVTJ7H58CElRyVJ7cEEX5JUKw8+9wpnXvtQy/KvH/ZWJm6/WYkRSVJ7McGXJI1e+vcAAA+ASURBVNXGK8te4+TL72HlqjWF5UdM3JZj3rFDyVFJUnsxwZck1UJKiS9efT9PL1xaWL7jqE349od2I8Jx95LUFRN8SVIt/M8dT/Gbh18sLNt4yEAuOmYSw4cOKjkqSWo/JviSpMr98akFfPemR1uWf+cju7PjqBElRiRJ7csEX5JUqXmvLuezV97L6jWpsPy4fXbg8D22LTkqSWpfJviSpMqsWr2GU6+8l78tWlFYPnH7zTjj0LeWHJUktTcTfElSZb5382P8cfbCwrLNNx7MBVMnMWSQb1WS1BOeNSVJlfjNwy9y0e1PFpZFwNlHTWT0ZsNKjkqS2p8JviSpdE8vWMrpv7yvZfmpB45n/wmjSoxIkprDBF+SVKrlr63mpMtnsmj5qsLyd45/E9MOGl9yVJLUHCb4kqRSnXXdQzw099XCsm1GbsQ5R01k4AAfZiVJ68sEX5JUmhkzn+XKu58pLBs0IDj/E5PYcpOhJUclSc1igi9JKsUjz7/K1371l5blZxy6C5N32LzEiCSpmUzwJUl97tXlr3Hy5few/LU1heWH7r4Nx+87ptygJKmhTPAlSX0qpcS/X/0As+cvKSwfu9Vw/vMjuxPhuHtJ6g0m+JKkPvWT38/mxodeKCwbNnggF06dzCZDB5UclSQ1lwm+JKnP/HnOQr5zw6yW5d/+8NuYsPWIEiOSpOarNMGPiO0i4qcRMTciVkTEnIg4JyJ6dJdVRGyRHzcnr2duXu92fRW7JKlr8xev4JQr7mHVmlRYPnXvf+JDe3qalqTeVtl3ohExDrgLGAVcC8wC3g5MAw6JiCkppQXrUM+WeT07AbcAVwE7AycAh0bEPimlp/rmVUiSiqxek5h21b28+OqKwvLdRo/kzMPeWnJUktQ/VDno8Udkyf1pKaXz1m6MiB8Anwe+BZy4DvV8myy5PzuldHqHek4Dfpj/nUN6Me4+9fLSlcxbVPyGKEntYsbMZ7nzieJrNCOHDeZHUyex0eCBJUclSf1DJQl+RIwFDgbmABd0Kv4G8K/AsRHxhZRS8bQLWT3DgWOBJflxHZ1P9kHhvRExtl2u4l93/1zOvPahqsOQpD5z9lF7sP0WG1cdhiQ1VlVj8A/M1zenlP5hUuSU0iLgTmBj4B3d1LMPMAy4Mz+uYz1rgJvzXw/Y4IglSRvslAPGceDOb646DElqtKqG6EzI14+1KH+c7Ar/TsBvN7Ae8nq6FBEzWxTt3N2xkqTu7TtuS05/z4Tud5QkbZCqruCPzNevtChfu32zkuqRJPWhUSOG8sOj92TgAB9mJUl9ra5PFln7DlA8t1of1JNSmlxYQXZlf9IGxiFJ/dawwQO5YOokthoxtOpQJKlfqCrBX3tlfWSL8k077dfX9dTGpsMGM37UJlWHIUkbbEAEE7YewWfeNZZdt211mpYk9baqEvxH83WrsfHj83WrsfW9XU9tHDFxNEdMHF11GJIkSWpTVY3BvzVfHxwR/xBDRIwApgDLgD90U88f8v2m5Md1rGcA2Y26Hf+eJEmS1GiVJPgppSfJprAcA5zSqfgsYDjws45z4EfEzhHxDzPapJQWAz/P95/eqZ7P5vXf1C5z4EuSJEkbqsqbbE8G7gLOjYiDgEeAvcnmrH8MOKPT/o/k685TMHwV2B84PSImAncDuwBHAPN44wcISZIkqbGqGqKz9ir+XsClZIn9F4BxwLnAPiml4mecv7GeBWQPvDoX2DGvZ2/gEmBy/nckSZKkfqHSaTJTSs8AJ6zjvi0nT04pLQSm5YskSZLUb1V2BV+SJElS7zPBlyRJkhrEBF+SJElqEBN8SZIkqUFM8CVJkqQGMcGXJEmSGsQEX5IkSWoQE3xJkiSpQUzwJUmSpAYxwZckSZIaJFJKVcdQaxGxYNiwYVvssssuVYciSZKkBnvkkUdYtmzZwpTSlhtSjwl+NyJiNrApMKeCP79zvp5Vwd9uR7ZXz9hePWN79Zxt1jO2V8/YXj1nm/VMFe01Bng1pfSWDanEBL/GImImQEppctWxtAPbq2dsr56xvXrONusZ26tnbK+es816pp3byzH4kiRJUoOY4EuSJEkNYoIvSZIkNYgJviRJktQgJviSJElSgziLjiRJktQgXsGXJEmSGsQEX5IkSWoQE3xJkiSpQUzwJUmSpAYxwZckSZIaxARfkiRJahATfEmSJKlBTPBrJiLGR8SXIuKWiHgmIlZGxIsRcW1EHNDNscdFxN0RsTgiXomI2yLisLJir0JEDI6IaRFxSUTcl7dXiohPd3HM8fk+rZYTy3wNZVqf9upwbL/rX12JiDHd9KOrqo6xKhGxXUT8NCLmRsSKiJgTEedExOZVx1Y3edu06kMvVB1fVSLioxFxXkT8LiJezdvjsm6O2Tciro+IhRGxNCIeiIjPRcTAsuKuSk/ay3MXRMSWEfHpiLgmIp6IiGX5+9rvI+JTEVGYH7dTHxtUdQB6g/8AjgIeBq4HFgITgMOBwyNiWkrp3M4HRcT3gC8AzwI/BoYARwPXRcSpKaXzS4q/bMOBc/KfXwReALZfx2OvBe4r2P7nXoirrtarvfpx/1oX9wO/Ktj+YNmB1EFEjAPuAkaR/Y/NAt4OTAMOiYgpKaUFFYZYR6/w+v9lR4vLDqRGvgbsQdYGzwI7d7VzRBwB/C+wHPgF2XvnB4CzgSnAkX0ZbA30qL1y/fncdSRwIfA8cCvwNPBm4MPAxcD7IuLI1OFpsG3Xx1JKLjVagOOBPQu2vwtYCawAtulUti+QgCeAzTtsHwMsIOuMY6p+bX3UXkOA961tE2B63haf7qaNE3B81fG3SXv12/7VTVuOydvl0qpjqdMC3JS3y6mdtv8g335R1THWaQHmAHOqjqNuC3AAMB4IYP+871zWYt9NgXn5++NeHbZvRPZhMwFHV/2aatRe/f7cBRxIlpwP6LR9a7JkPwEf6bC97fqYQ3RqJqV0aUrp3oLttwO3kSVo+3YqXjuk5FsppZc6HDMHuAAYCpzQF/FWLaW0MqV0Q0rp+apjaQfr2V79tn+pZyJiLHAwWdJ6QafibwBLgGMjYnjJoanNpJRuTSk9nvIsqhsfBbYCrkop/f0b2JTScrIr2wAn9UGYtdHD9ur3Ukq3pJSuSymt6bT9BeCi/Nf9OxS1XR8zwW8vr+XrVZ22H5ivbyw45oZO++h1E/Oxc1+OiGMjYruqA6op+1fXto2Iz0TEV/P17lUHVKG1/eDmgjfORcCdwMbAO8oOrOaGRsQxeR+aFhEH1HFMb411dY66A1gK7BsRQ8sLqS147ipWlGu1XR9zDH6biIgdgIPIOtEdHbYPB0YDi1tclX08X+/U50G2n2mdfl8dERcDn8s/lfd79q918p58+buIuA04LqX0dCURVWdCvn6sRfnjZFf4dwJ+W0pE7WFr4Oedts2OiBPyb2/VtZb9LqW0KiJmA7sCY4FHygys5jx3dRIRg4BP5r92TObbro95Bb8N5J8ILycbCjG94zAJYGS+fqXF4Wu3b9ZH4bWj2cCpZP+ww4FtgY+RDSv4DPDTyiKrH/tXa0vJboqfDGyeL+8iu2Frf+C3/XAoiv2l5y4hu3izNdn5aDfgv8nGSd8QEXtUF1rbsN/1jOeu1r4DvA24PqV0U4ftbdfHTPD7QDfTnhUtLaf+yr+m/TnZHdq/AL63nmHVdlxeb7bXukgp3Z5SOj+l9FhKaWlK6fmU0tVkNym9BHy8zm+qZbfXOqpt/+rKhrRlSmleSunrKaV7Ukov58sdZFeo/wjsCHQ7/Wg/E/m6LftLX0gpnZWPB34xPx89mFI6keym5GFkN8Jrw9jvOvDcVSwiTiObLW4WcGxPD8/XteljDtHpG0+SzSyyruYWbcyT+8vIpl76JXBMwQ00az81jqRYd58666BX2mtDpZSeiYjrganAfmRTiNVRme3VhP7VlV5vy/zr2ouBvcn60Q/XM7Z21F1/2bTTfmrtIrJkY7+qA2kD9rte0J/PXRFxCtnrfRg4KKW0sNMubdfHTPD7QErpoA2tIx8HdgVZcn8F8MmU0uqCv7UkIp4DRkfENgXjpMfn61ZjYivXG+3Vi/6Wr2v79WSZ7dWE/tWVPmzL2vejPvJovm51T0Zb95eSzcvX/a0PrY9Hgb3I+t3MjgX5e+lbyG6YfKr80NpOvzt3RcTnyOayf5AsuZ9XsFvb9TGH6NRQRAwBZpAl9z8Dji1K7ju4JV8fUlD2vk77qGt75+va/JPWgP2r59bOEtPf+tGt+frgzk+CjIgRZEMNlwF/KDuwNrRPvu5vfWh9dHWO2o9s5qa7UkorygupbfWrc1dEfIksub8POKBFcg9t2MdM8Gsmv6H2GuAI4CfACZ2nmyuwds7WM6LDo+AjYgxwCtmDGS7p9WDbVES8s2BbRMRXyN5U51M8FVZ/Zf8qEBF75x/GO28/EPh8/msZ9z/URkrpSeBmshtET+lUfBbZVcGfpZSWlBxaLUXErhGxRcH2HYC1T4fuV31oPc0gO28fHRF7rd0YERsB38x/vbCKwOrIc1cmIs4ku6l2JtmV+/ld7N52fSx8JkK9RMQlZE9anQ/8iOIbNm5LKd3W6bjvA6eTPaJ6BtkDsY4CtiR7ouT5nStpioj4Mq8/lnsi2eO67+L1KRx/n1K6uMP+iWyIwJ+A58jG1E0hu3N+KfChlNLN5URfvp62V35Mv+1freTTye1K9gC6Z/PNu/P6fMlnppS++cYjmy0ixpH1p1HAtWRTxu1NdhP7Y8C+KaUF1UVYHxExHfgy2Tcfs4FFwDjgULInZF5Pdj5aWVWMVYmIDwIfzH/dGngv2VXl3+Xb5qeUvthp/xlk99RcBSwEDiebLW0G8LEmPwSqJ+3luQsi4jjgUmA1cB7FY+fnpJQu7XBMe/Wx3ngcrkvvLWT/cKmbZXqLY48jS1qXkL1R3A4cVvVrqkGbXdpp///K22Yu2T/qUrK75s8Hxlb9eurWXv29f3XRjp8Cfk02vepism8yniab7eqdVcdXcdtsT/atzvPASuCvZDewbVF1bHVayKYmvDI//7xM9oCdvwG/IZuLO6qOscK2md7NeWpOwTFTyD4UvUQ2FOwvZFekB1b9eurUXp671qm9EtnF1LbtY17BlyRJkhrEMfiSJElSg5jgS5IkSQ1igi9JkiQ1iAm+JEmS1CAm+JIkSVKDmOBLkiRJDWKCL0mSJDWICb4kSZLUICb4kiRJUoOY4EuSJEkNYoIvSZIkNYgJviRJktQgJviSJElSg5jgS5IkSQ1igi9JkiQ1iAm+JEmS1CAm+JIkSVKD/D9wd0ToG3hZTQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 250,
       "width": 380
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pts_relu=[max(0,i) for i in pts];\n",
    "plt.plot(pts, pts_relu) ;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "are some of the commonly used as activation functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "<p><i class=\"fa fa-warning\"></i>&nbsp;\n",
    "ReLU is very popular and is widely used nowadays. There also exist other variations of ReLU, e.g. \"leaky ReLU\".\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<p><i class=\"fa fa-warning\"></i>&nbsp;\n",
    "Why don't we just use a simple linear activation function?\n",
    "    \n",
    "Linear activations are **NOT** used because it can be mathematically shown that if they are used then the output is just a linear function of the input. So we cannot learn interesting and complex functions by adding any number of hidden layers.\n",
    "\n",
    "The only exception when we do want to use a linear activation is for the output layer of a network when solving a regression problem.\n",
    "\n",
    "</p>\n",
    "</div>\n",
    "\n",
    "\n",
    "\n",
    "Non-linear activation functions allow the network to learn complex representations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is Keras?\n",
    "\n",
    "* It is a high level API to create and work with neural networks\n",
    "* Supports multiple backends such as TensorFlow from Google, Theano (Although Theano is dead now) and CNTK (Microsoft Cognitive Toolkit)\n",
    "* Very good for creating neural nets very quickly and hides away a lot of tedious work\n",
    "* Has been incorporated into official TensorFlow (which obviously only works with tensforflow) and as of TensorFlow 2.0 this will the main api to use TensorFlow (check reference)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Say hello to keras\n",
    "\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation\n",
    "\n",
    "# Creating a model\n",
    "model = Sequential()\n",
    "\n",
    "# Adding layers to this model\n",
    "# 1st Hidden layer\n",
    "# A Dense/fully-connected layer which takes as input a \n",
    "# feature array of shape (samples, num_features)\n",
    "# Here input_shape = (8,) means that the layer expects an input with num_features = 8 \n",
    "# and the sample size could be anything\n",
    "# Then we specify an activation function\n",
    "model.add(Dense(units=4, input_shape=(8,)))\n",
    "model.add(Activation(\"relu\"))\n",
    "\n",
    "# 2nd Hidden layer\n",
    "# This is also a fully-connected layer and we do not need to specify the\n",
    "# shape of the input anymore (We need to do that only for the first layer)\n",
    "# NOTE: Now we didn't add the activation seperately. Instead we just added it\n",
    "# while calling Dense(). This and the way used for the first layer are Equivalent!\n",
    "model.add(Dense(units=4, activation=\"relu\"))\n",
    "\n",
    "          \n",
    "# The output layer\n",
    "model.add(Dense(units=1))\n",
    "model.add(Activation(\"sigmoid\"))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### XOR using neural networks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a network to solve the XOR problem\n",
    "\n",
    "# Loading and plotting the data\n",
    "xor = pd.read_csv(\"xor.csv\")\n",
    "\n",
    "# Using x and y coordinates as featues\n",
    "features = xor.iloc[:, :-1]\n",
    "# Convert boolean to integer values (True->1 and False->0)\n",
    "labels = xor.iloc[:, -1].astype(int)\n",
    "\n",
    "colors = [[\"steelblue\", \"chocolate\"][i] for i in xor[\"label\"]]\n",
    "plt.figure(figsize=(5, 5))\n",
    "plt.xlim([-2, 2])\n",
    "plt.ylim([-2, 2])\n",
    "plt.title(\"Blue points are False\")\n",
    "\n",
    "\n",
    "plt.scatter(features[\"x\"], features[\"y\"], color=colors, marker=\"o\") ;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Building a Keras model\n",
    "\n",
    "def a_simple_NN():\n",
    "    \n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(Dense(4, input_shape = (2,), activation = \"relu\"))\n",
    "\n",
    "    model.add(Dense(4, activation = \"relu\"))\n",
    "\n",
    "    model.add(Dense(1, activation = \"sigmoid\"))\n",
    "\n",
    "    model.compile(loss=\"binary_crossentropy\", optimizer=\"rmsprop\", metrics=[\"accuracy\"])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiating the model\n",
    "model = a_simple_NN()\n",
    "\n",
    "# Splitting the dataset into training (70%) and validation sets (30%)\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    features, labels, test_size=0.3)\n",
    "\n",
    "# Setting the number of passes through the entire training set\n",
    "num_epochs = 300\n",
    "\n",
    "# We can pass validation data while training\n",
    "model_run = model.fit(X_train, y_train, epochs=num_epochs,\n",
    "                      validation_data=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Looking at the loss and accuracy on the training and validation sets during the training\n",
    "# This can be done by using Keras callback \"history\" which is applied by default\n",
    "history_model = model_run.history\n",
    "\n",
    "print(\"The history has the following data: \", history_model.keys())\n",
    "\n",
    "# Plotting the training and validation accuracy during the training\n",
    "plt.plot(np.arange(1, num_epochs+1), history_model[\"acc\"], \"blue\") ;\n",
    "\n",
    "plt.plot(np.arange(1, num_epochs+1), history_model[\"val_acc\"], \"red\") ;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Here we dont't really see a big difference between the training and validation data because the function we are trying to fit is quiet simple and there is not too much noise. We will come back to these curves in a later example**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example above we splitted our dataset into a 70-30 train-validation set. We know from previous chapters that to more robustly calculate accuracy we can use **K-fold crossvalidation**.\n",
    "This is even more important when we have small datasets and cannot afford to reserve a validation set!\n",
    "\n",
    "One way to do the cross validation here would be to write our own function to do this. However, we also know that **SciKit learn** provides several handy functions to evaluate and tune the models. So the question is:\n",
    "\n",
    "Can we somehow use these **Scikit learn** functions or ones we wrote ourselves for **Scikit learn** models to evaluate and tune our Keras models?\n",
    "\n",
    "The Answer is **YES !**\n",
    "\n",
    "We show how to do this in the following section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using SciKit learn functions on Keras models\n",
    "\n",
    "Keras offers 2 wrappers which allow its Sequential models to be used with SciKit learn. \n",
    "\n",
    "There are: **KerasClassifier** and **KerasRegressor**.\n",
    "\n",
    "For more information:\n",
    "https://keras.io/scikit-learn-api/\n",
    "\n",
    "**Now lets see how this works!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We wrap the Keras model we created above with KerasClassifier\n",
    "from keras.wrappers.scikit_learn import KerasClassifier\n",
    "from sklearn.model_selection import cross_val_score\n",
    "# Wrapping Keras model\n",
    "# NOTE: We pass verbose=0 to suppress the model output\n",
    "num_epochs = 400\n",
    "model_scikit = KerasClassifier(\n",
    "    build_fn=a_simple_NN, **{\"epochs\": num_epochs, \"verbose\": 0})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's reuse the function to visualize the decision boundary which we saw in chapter 2 with minimal change\n",
    "\n",
    "def list_flatten(list_of_list):\n",
    "    flattened_list = [i for j in list_of_list for i in j]\n",
    "    return flattened_list\n",
    "\n",
    "def plot_points(plt=plt, marker='o'):\n",
    "    colors = [[\"steelblue\", \"chocolate\"][i] for i in labels]\n",
    "    plt.scatter(features.iloc[:, 0], features.iloc[:, 1], color=colors, marker=marker);\n",
    "\n",
    "def train_and_plot_decision_surface(\n",
    "    name, classifier, features_2d, labels, preproc=None, plt=plt, marker='o', N=400\n",
    "):\n",
    "\n",
    "    features_2d = np.array(features_2d)\n",
    "    xmin, ymin = features_2d.min(axis=0)\n",
    "    xmax, ymax = features_2d.max(axis=0)\n",
    "\n",
    "    x = np.linspace(xmin, xmax, N)\n",
    "    y = np.linspace(ymin, ymax, N)\n",
    "    points = np.array(np.meshgrid(x, y)).T.reshape(-1, 2)\n",
    "\n",
    "    if preproc is not None:\n",
    "        points_for_classifier = preproc.fit_transform(points)\n",
    "        features_2d = preproc.fit_transform(features_2d)\n",
    "    else:\n",
    "        points_for_classifier = points\n",
    "\n",
    "    classifier.fit(features_2d, labels, verbose=0)\n",
    "    predicted = classifier.predict(features_2d)\n",
    "    \n",
    "    if name == \"Neural Net\":\n",
    "        predicted = list_flatten(predicted)\n",
    "    \n",
    "    \n",
    "    if preproc is not None:\n",
    "        name += \" (w/ preprocessing)\"\n",
    "    print(name + \":\\t\", sum(predicted == labels), \"/\", len(labels), \"correct\")\n",
    "    \n",
    "    if name == \"Neural Net\":\n",
    "        classes = np.array(list_flatten(classifier.predict(points_for_classifier)), dtype=bool)\n",
    "    else:\n",
    "        classes = np.array(classifier.predict(points_for_classifier), dtype=bool)\n",
    "    plt.plot(\n",
    "        points[~classes][:, 0],\n",
    "        points[~classes][:, 1],\n",
    "        \"o\",\n",
    "        color=\"steelblue\",\n",
    "        markersize=1,\n",
    "        alpha=0.01,\n",
    "    )\n",
    "    plt.plot(\n",
    "        points[classes][:, 0],\n",
    "        points[classes][:, 1],\n",
    "        \"o\",\n",
    "        color=\"chocolate\",\n",
    "        markersize=1,\n",
    "        alpha=0.04,\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_, ax = plt.subplots(figsize=(6, 6))\n",
    "\n",
    "train_and_plot_decision_surface(\"Neural Net\", model_scikit, features, labels, plt=ax)\n",
    "plot_points(plt=ax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Applying K-fold cross-validation\n",
    "# Here we pass the whole dataset, i.e. features and labels, instead of splitting it.\n",
    "num_folds = 5\n",
    "cross_validation = cross_val_score(\n",
    "    model_scikit, features, labels, cv=num_folds, verbose=0)\n",
    "\n",
    "print(\"The acuracy on the \", num_folds, \" validation folds:\", cross_validation)\n",
    "print(\"The Average acuracy on the \", num_folds, \" validation folds:\", np.mean(cross_validation))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NOTE: The above code took quiet long even though we used only 5  CV folds and the neural network and data size are very small!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We know from chapter 6 that there are 2 types of parameters which need to be tuned for a machine learning model.\n",
    "* Internal model parameters (weights) which can be learned for e.g. by gradient-descent\n",
    "* Hyperparameters\n",
    "\n",
    "In the model which we created above we made some arbitrary choices like which optimizer we use, what is its learning rate, number of hidden units and so on ...\n",
    "\n",
    "Now that we have the keras model wrapped as a scikit model we can use the grid search functions we have seen in chapter 6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "# Just to remember\n",
    "model_scikit = KerasClassifier(\n",
    "    build_fn=a_simple_NN, **{\"epochs\": num_epochs, \"verbose\": 0})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HP_grid = {'epochs' : [300, 500, 1000]}\n",
    "search = GridSearchCV(estimator=model_scikit, param_grid=HP_grid)\n",
    "search.fit(features, labels)\n",
    "print(search.best_score_, search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HP_grid = {'epochs' : [10, 15, 30], \n",
    "           'batch_size' : [10, 20, 30] }\n",
    "search = GridSearchCV(estimator=model_scikit, param_grid=HP_grid)\n",
    "search.fit(features, labels)\n",
    "print(search.best_score_, search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A more general model for further Hyperparameter optimization\n",
    "from keras import optimizers\n",
    "\n",
    "def a_simple_NN(activation='relu', num_hidden_neurons=[4, 4], learning_rate=0.01):\n",
    "\n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(Dense(num_hidden_neurons[0],\n",
    "                    input_shape=(2,), activation=activation))\n",
    "\n",
    "    model.add(Dense(num_hidden_neurons[1], activation=activation))\n",
    "\n",
    "    model.add(Dense(1, activation=\"sigmoid\"))\n",
    "\n",
    "    model.compile(loss=\"binary_crossentropy\", optimizer=optimizers.rmsprop(\n",
    "        lr=learning_rate), metrics=[\"accuracy\"])\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: \n",
    "* Look at the model above and choose a couple of hyperparameters to optimize. \n",
    "* **(OPTIONAL:)** What function from SciKit learn other than GridSearchCV can we use for hyperparameter optimization? Use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Create a neural network to classify the 2d points example from chapter 2 learned \n",
    "(Optional: As you create the model read a bit on the different keras commands we have used)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "circle = pd.read_csv(\"2d_points.csv\")\n",
    "# Using x and y coordinates as featues\n",
    "features = circle.iloc[:, :-1]\n",
    "# Convert boolean to integer values (True->1 and False->0)\n",
    "labels = circle.iloc[:, -1].astype(int)\n",
    "\n",
    "colors = [[\"steelblue\", \"chocolate\"][i] for i in circle[\"label\"]]\n",
    "plt.figure(figsize=(5, 5))\n",
    "plt.xlim([-2, 2])\n",
    "plt.ylim([-2, 2])\n",
    "\n",
    "plt.scatter(features[\"x\"], features[\"y\"], color=colors, marker=\"o\");\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Insert Code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The examples above are not the ideal use problems one should use neural networks for. They are too simple and can be easily solved by classical machine learning algorithms. Below we show examples which are the more common applications of Neural Networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handwritten Digits Classification\n",
    "### MNIST Dataset\n",
    "\n",
    "MNIST datasets is a very common dataset used in machine learning. It is widely used to train and validate models.\n",
    "\n",
    "\n",
    ">The MNIST database of handwritten digits, available from this page, has a training set of 60,000 examples, and a >test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size->normalized and centered in a fixed-size image.\n",
    ">It is a good database for people who want to try learning techniques and pattern recognition methods on real-world >data while spending minimal efforts on preprocessing and formatting.\n",
    ">source: http://yann.lecun.com/exdb/mnist/\n",
    "\n",
    "The problem we want to solve using this dataset is: multi-class classification (FIRST TIME)\n",
    "This dataset consists of images of handwritten digits between 0-9 and their corresponsing labels. We want to train a neural network which is able to predict the correct digit on the image. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading the dataset in keras\n",
    "# Later you can explore and play with other datasets with come with Keras\n",
    "from keras.datasets import mnist\n",
    "\n",
    "# Loading the train and test data\n",
    "\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Looking at the dataset\n",
    "print(X_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can see that the training set consists of 60,000 images of size 28x28 pixels\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "i=np.random.randint(0,X_train.shape[0])\n",
    "plt.imshow(X_train[i], cmap=\"gray_r\") ;\n",
    "print(\"This digit is: \" , y_train[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at the data values for a couple of images\n",
    "print(X_train[0].min(), X_train[1].max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data consists of values between 0-255 representing the **grayscale level**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The labels are the digit on the image\n",
    "print(y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scaling the data\n",
    "# It is important to normalize the input data to (0-1) before providing it to a neural net\n",
    "# We could use the previously introduced function from SciKit learn. However, here it is sufficient to\n",
    "# just divide the input data by 255\n",
    "X_train_norm = X_train/255.\n",
    "X_test_norm = X_test/255.\n",
    "\n",
    "# Also we need to reshape the input data such that each sample is a vector and not a 2D matrix\n",
    "X_train_prep = X_train_norm.reshape(X_train_norm.shape[0],28*28)\n",
    "X_test_prep = X_test_norm.reshape(X_test_norm.shape[0],28*28)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**IMPORTANT: One-Hot encoding**\n",
    "\n",
    "**TODO: Better frame the explaination**\n",
    "\n",
    "In such problems the labels are provided as something called **One-hot encodings**. What this does is to convert a categorical label to a vector.\n",
    "\n",
    "For the MNIST problem where we have **10 categories** one-hot encoding will create a vector of length 10 for each of the labels. All the entries of this vector will be zero **except** for the index which is equal to the integer value of the label.\n",
    "\n",
    "For example:\n",
    "if label is 4. The one-hot vector will look like **[0 0 0 0 1 0 0 0 0 0]**\n",
    "\n",
    "Fortunately, we don't have to code this ourselves because Keras has a built-in function for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils.np_utils import to_categorical\n",
    "\n",
    "y_train_onehot = to_categorical(y_train, num_classes=10)\n",
    "y_test_onehot = to_categorical(y_test, num_classes=10)\n",
    "\n",
    "print(y_train_onehot.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Building the keras model\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "\n",
    "def mnist_model():\n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(Dense(64, input_shape=(28*28,), activation=\"relu\"))\n",
    "\n",
    "    model.add(Dense(64, activation=\"relu\"))\n",
    "\n",
    "    model.add(Dense(10, activation=\"softmax\"))\n",
    "\n",
    "    model.compile(loss=\"categorical_crossentropy\",\n",
    "                  optimizer=\"rmsprop\", metrics=[\"accuracy\"])\n",
    "    return model\n",
    "\n",
    "model = mnist_model()\n",
    "\n",
    "model_run = model.fit(X_train_prep, y_train_onehot, epochs=20,\n",
    "                      batch_size=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The [loss, accuracy] on test dataset are: \" , model.evaluate(X_test_prep, y_test_onehot))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional exercise: Run the model again with validation dataset, plot the accuracy as a function of epochs, play with number of epochs and observe what is happening."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution:\n",
    "num_epochs = 20\n",
    "model_run = model.fit(X_train_prep, y_train_onehot, epochs=num_epochs,\n",
    "                      batch_size=512, validation_data=(X_test_prep, y_test_onehot))\n",
    "# Evaluating the model on test dataset\n",
    "#print(\"The [loss, accuracy] on test dataset are: \" , model.evaluate(X_test_prep, y_test_onehot))\n",
    "history_model = model_run.history\n",
    "print(\"The history has the following data: \", history_model.keys())\n",
    "\n",
    "# Plotting the training and validation accuracy during the training\n",
    "plt.plot(np.arange(1, num_epochs+1), history_model[\"acc\"], \"blue\")\n",
    "\n",
    "plt.plot(np.arange(1, num_epochs+1), history_model[\"val_acc\"], \"red\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding l2 regularization\n",
    "# Building the keras model\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.regularizers import l2\n",
    "\n",
    "def mnist_model():\n",
    "    \n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(Dense(64, input_shape=(28*28,), activation=\"relu\", \n",
    "                   kernel_regularizer=l2(0.01)))\n",
    "\n",
    "    model.add(Dense(64, activation=\"relu\", \n",
    "                   kernel_regularizer=l2(0.01)))\n",
    "\n",
    "    model.add(Dense(10, activation=\"softmax\"))\n",
    "\n",
    "    model.compile(loss=\"categorical_crossentropy\",\n",
    "                  optimizer=\"rmsprop\", metrics=[\"accuracy\"])\n",
    "    return model\n",
    "\n",
    "model = mnist_model()\n",
    "\n",
    "num_epochs = 50\n",
    "model_run = model.fit(X_train_prep, y_train_onehot, epochs=num_epochs,\n",
    "                      batch_size=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The [loss, accuracy] on test dataset are: \" , model.evaluate(X_test_prep, y_test_onehot))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Another way to add regularization and to make the network more robust we can add something called \"Dropout\". When we add dropout to a layer a specified percentage of units in that layer are switched off. \n",
    "(MAKING MODEL SIMPLER)\n",
    "\n",
    "### Exercise: Add dropout instead of l2 regularization in the network above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding dropout is easy in keras\n",
    "# We import a layer called Dropout and add as follows\n",
    "# model.add(Dropout(0.5)) to randomly drop 50% of the hidden units\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution\n",
    "# Adding Dropout\n",
    "# Building the keras model\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout\n",
    "\n",
    "def mnist_model():\n",
    "    \n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(Dense(64, input_shape=(28*28,), activation=\"relu\"))\n",
    "              \n",
    "    model.add(Dropout(0.4))\n",
    "\n",
    "    model.add(Dense(64, activation=\"relu\"))\n",
    "\n",
    "    model.add(Dense(10, activation=\"softmax\"))\n",
    "\n",
    "    model.compile(loss=\"categorical_crossentropy\",\n",
    "                  optimizer=\"rmsprop\", metrics=[\"accuracy\"])\n",
    "              \n",
    "    return model\n",
    "\n",
    "model = mnist_model()\n",
    "\n",
    "num_epochs = 50\n",
    "model_run = model.fit(X_train_prep, y_train_onehot, epochs=num_epochs,\n",
    "                      batch_size=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The [loss, accuracy] on test dataset are: \" , model.evaluate(X_test_prep, y_test_onehot))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network Architecture\n",
    "\n",
    "The neural networks which we have seen till now are the simplest kind of neural networks.\n",
    "There exist more sophisticated network architectures especially designed for specific applications.\n",
    "Some of them are as follows:\n",
    "\n",
    "###  Convolution Neural Networks (CNNs)\n",
    "\n",
    "These networks are used mostly for computer vision (EXAMPLES) like tasks. \n",
    "One of the old CNN networks is shown below.\n",
    "\n",
    "<center>\n",
    "<figure>\n",
    "<img src=\"./images/neuralnets/CNN_lecun.png\" width=\"800\"/>\n",
    "<figcaption>source: LeCun et al., Gradient-based learning applied to document recognition (1998).</figcaption>\n",
    "</figure>\n",
    "</center>\n",
    "\n",
    "CNNs consist of new type of layers like convolution layer and pooling layers.\n",
    "\n",
    "###  Recurrent Neural Networks (RNNs)\n",
    "\n",
    "These are used for time-series data, speech recognition, translation etc.\n",
    "\n",
    "IMAGE HERE\n",
    "\n",
    "### Generative adversarial networks (GANs)\n",
    "\n",
    "GANs consist of 2 parts, a generative network and a discriminative network. The generative network produces data which is then fed to the discriminative network which judges if the new data belongs to a specified dataset. Then via feedback loops the generative network becomes better and better at creating images similar to the dataset the discriminative network is judging against. At the same time the discriminative network get better and better at identifyig **fake** instances which are not from the reference dataset. \n",
    "\n",
    "IMAGE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CNN example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this example we will work with a dataset called fashion-MNIST which is quite similar to the MNIST data above.\n",
    "> Fashion-MNIST is a dataset of Zalando's article images—consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. We intend Fashion-MNIST to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same image size and structure of training and testing splits.\n",
    "source: https://github.com/zalandoresearch/fashion-mnist\n",
    "\n",
    "The 10 classes of this dataset are:\n",
    "\n",
    "| Label| Item |\n",
    "| --- | --- |\n",
    "| 0 |\tT-shirt/top |\n",
    "| 1\t| Trouser |\n",
    "|2|\tPullover|\n",
    "|3|\tDress|\n",
    "|4|\tCoat|\n",
    "|5|\tSandal|\n",
    "|6|\tShirt|\n",
    "|7|\tSneaker|\n",
    "|8|\tBag|\n",
    "|9|\tAnkle boot|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading the dataset in keras\n",
    "# Later you can explore and play with other datasets with come with Keras\n",
    "from keras.datasets import fashion_mnist\n",
    "\n",
    "# Loading the train and test data\n",
    "\n",
    "(X_train, y_train), (X_test, y_test) = fashion_mnist.load_data()\n",
    "\n",
    "items =['T-shirt/top', 'Trouser', \n",
    "        'Pullover', 'Dress', \n",
    "        'Coat', 'Sandal', \n",
    "        'Shirt', 'Sneaker',\n",
    "        'Bag', 'Ankle boot']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can see that the training set consists of 60,000 images of size 28x28 pixels\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "i=np.random.randint(0,X_train.shape[0])\n",
    "plt.imshow(X_train[i], cmap=\"gray_r\") ; \n",
    "print(\"This item is a: \" , items[y_train[i]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Also we need to reshape the input data such that each sample is a 4D matrix of dimension\n",
    "# (num_samples, width, height, channels). Even though these images are grayscale we need to add\n",
    "# channel dimension as this is expected by the Conv function\n",
    "X_train_prep = X_train.reshape(X_train.shape[0],28,28,1)/255.\n",
    "X_test_prep = X_test.reshape(X_test.shape[0],28,28,1)/255.\n",
    "\n",
    "from keras.utils.np_utils import to_categorical\n",
    "\n",
    "y_train_onehot = to_categorical(y_train, num_classes=10)\n",
    "y_test_onehot = to_categorical(y_test, num_classes=10)\n",
    "\n",
    "print(y_train_onehot.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a CNN similar to the one shown in the figure from LeCun paper\n",
    "# In the original implementation Average pooling was used. However, we will use maxpooling as this \n",
    "# is what us used in the more recent architectures and is found to be a better choice\n",
    "# Convolution -> Pooling -> Convolution -> Pooling -> Flatten -> Dense -> Dense -> Output layer\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Conv2D, MaxPool2D, Flatten, Dropout, BatchNormalization\n",
    "\n",
    "def simple_CNN():\n",
    "    \n",
    "    model = Sequential()\n",
    "    \n",
    "    model.add(Conv2D(6, (3,3), input_shape=(28,28,1), activation='relu'))\n",
    "    \n",
    "    model.add(MaxPool2D((2,2)))\n",
    "    \n",
    "    model.add(Conv2D(16, (3,3), activation='relu'))\n",
    "    \n",
    "    model.add(MaxPool2D((2,2)))\n",
    "    \n",
    "    model.add(Flatten())\n",
    "    \n",
    "    model.add(Dense(120, activation='relu'))\n",
    "    \n",
    "    model.add(Dense(84, activation='relu'))\n",
    "    \n",
    "    model.add(Dense(10, activation='softmax'))\n",
    "    \n",
    "    model.compile(loss=\"categorical_crossentropy\", optimizer=\"rmsprop\", metrics=[\"accuracy\"])\n",
    "    \n",
    "    return model\n",
    "\n",
    "model = simple_CNN()\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_epochs = 10\n",
    "model_run = model.fit(X_train_prep, y_train_onehot, epochs=num_epochs, \n",
    "                      batch_size=64, validation_data=(X_test_prep, y_test_onehot))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Use the above model or improve it (change number of filters, add more layers etc. on the MNIST example and see if you can get a better accuracy than what we achieved with a vanilla neural network)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Load and play with the CIFAR10 dataset also included with Keras and build+train a simple CNN using it"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}