Newer
Older
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<style>\n",
" \n",
" @import url('http://fonts.googleapis.com/css?family=Source+Code+Pro');\n",
" \n",
" @import url('http://fonts.googleapis.com/css?family=Kameron');\n",
" @import url('http://fonts.googleapis.com/css?family=Crimson+Text');\n",
" \n",
" @import url('http://fonts.googleapis.com/css?family=Lato');\n",
" @import url('http://fonts.googleapis.com/css?family=Source+Sans+Pro');\n",
" \n",
" @import url('http://fonts.googleapis.com/css?family=Lora'); \n",
"\n",
" \n",
" body {\n",
" font-family: 'Lora', Consolas, sans-serif;\n",
" \n",
" -webkit-print-color-adjust: exact important !;\n",
" \n",
" \n",
" \n",
" }\n",
" \n",
" .alert-block {\n",
" width: 95%;\n",
" margin: auto;\n",
" }\n",
" \n",
" .rendered_html code\n",
" {\n",
" color: black;\n",
" background: #eaf0ff;\n",
" background: #f5f5f5; \n",
" padding: 1pt;\n",
" font-family: 'Source Code Pro', Consolas, monocco, monospace;\n",
" }\n",
" \n",
" p {\n",
" line-height: 140%;\n",
" }\n",
" \n",
" strong code {\n",
" background: red;\n",
" }\n",
" \n",
" .rendered_html strong code\n",
" {\n",
" background: #f5f5f5;\n",
" }\n",
" \n",
" .CodeMirror pre {\n",
" font-family: 'Source Code Pro', monocco, Consolas, monocco, monospace;\n",
" }\n",
" \n",
" .cm-s-ipython span.cm-keyword {\n",
" font-weight: normal;\n",
" }\n",
" \n",
" strong {\n",
" background: #f5f5f5;\n",
" margin-top: 4pt;\n",
" margin-bottom: 4pt;\n",
" padding: 2pt;\n",
" border: 0.5px solid #a0a0a0;\n",
" font-weight: bold;\n",
" color: darkred;\n",
" }\n",
" \n",
" \n",
" div #notebook {\n",
" # font-size: 10pt; \n",
" line-height: 145%;\n",
" }\n",
" \n",
" li {\n",
" line-height: 145%;\n",
" }\n",
"\n",
" div.output_area pre {\n",
" background: #fff9d8 !important;\n",
" padding: 5pt;\n",
" \n",
" -webkit-print-color-adjust: exact; \n",
" \n",
" }\n",
" \n",
" \n",
" \n",
" h1, h2, h3, h4 {\n",
" font-family: Kameron, arial;\n",
"\n",
"\n",
" }\n",
" \n",
" div#maintoolbar {display: none !important;}\n",
"</style>\n",
" <script>\n",
"IPython.OutputArea.prototype._should_scroll = function(lines) {\n",
" return false;\n",
"}\n",
" </script>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# IGNORE THIS CELL WHICH CUSTOMIZES LAYOUT AND STYLING OF THE NOTEBOOK !\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n",
"%config InlineBackend.figure_format = 'retina'\n",
"import warnings\n",
"warnings.filterwarnings('ignore', category=FutureWarning)\n",
"from IPython.core.display import HTML; HTML(open(\"custom.html\", \"r\").read())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Chapter 1: General Introduction to machine learning (ML)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A \"model\" allows us to explain observations and to answer questions. For example:\n",
"\n",
" 1. Where will my car at given velocity stop if I apply break now?\n",
" 2. Where on the night sky will I see the moon tonight?\n",
" 3. Is the email I received spam?\n",
"- The first two questions can be answered based on existing physical models (formulas). \n",
"\n",
"- For the questions 3 and 4 it is difficult to develop explicitly formulated models. \n",
"- We have a vague understanding of the problem domain, e.g. we know that some words are specific to spam emails and others are specific to my personal and work-related emails.\n",
"- We have enough example data, as my mailbox is full of both spam and non-spam emails.\n",
"\n",
"\n",
"We could handcraft a personal spam classifier by hard coding rules, like \"mail contains 'no prescription' and comes from russia or china\" plus some statistics which would be very tedious\n",
"\n",
"<div class=\"alert alert-block alert-info\">\n",
"<i class=\"fa fa-info-circle\"></i>\n",
" Systems with such hard coded rules are called <strong>expert systems</strong>\n",
"</div>\n",
"\n",
"**In such cases machine learning offers approaches to automatically build predictive models based on example data.**\n",
"<div class=\"alert alert-block alert-info\">\n",
"<i class=\"fa fa-info-circle\"></i>\n",
"The closely-related concept of <strong>data mining</strong> usually means use of predictive machine learning models to explicitly discover previously unknown knowledge from a specific data set, such as, for instance, association rules between customer and article types in the Problem 4 above.\n",
"\n",
"\n",
"\n",
"## ML: what is \"learning\" ?\n",
"\n",
"To create a predictive model, we must first **train** such a model on given data. \n",
"<div class=\"alert alert-block alert-info\">\n",
"<i class=\"fa fa-info-circle\"></i>\n",
"Alternative names for \"to train\" a model are \"to <strong>fit</strong>\" or \"to <strong>learn</strong>\" a model.\n",
"</div>\n",
"\n",
"All ML algorithms have in common that they rely on internal data structures and/or parameters. Learning then builds up such data structures or adjusts parameters based on the given data. After that such models can be used to explain observations or to answer questions.\n",
"\n",
"The important difference between explicit models and models learned from data:\n",
"\n",
"- Explicit models usually offer exact answers to questions\n",
"- Models we learn from data usually come with inherent uncertainty."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Some parts of ML are older than you might think. This is a rough time line with a few selected achievements from this field:\n",
" 1805: Least squares regression\n",
" 1812: Bayes' rule\n",
" 1957-65: \"k-means\" clustering algorithm\n",
" 1959: Term \"machine learning\" is coined by Arthur Samuel, an AI pioneer\n",
" 1969: Book \"Perceptrons\": Limitations of Neural Networks\n",
" 1974-86: Neural networks learning breakthrough: backpropagation method\n",
" 1984: Book \"Classification And Regression Trees\"\n",
" 1995: Randomized Forests and Support Vector Machines methods\n",
" 1998: Public appearance: first ML implementations of spam filtering methods; naive Bayes Classifier method\n",
" 2006-12: Neural networks learning breakthrough: deep learning\n",
" \n",
"So the field is not as new as one might think, but due to \n",
"\n",
"- more available data\n",
"- more processing power \n",
"- development of better algorithms \n",
"\n",
"more applications of machine learning appeared during the last 15 years."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Machine learning with Python\n",
"\n",
"Currently (as of 2019) `Python` is the dominant programming language for ML. Especially the advent of deep-learning pushed this forward. First versions of frameworks such as `TensorFlow` or `PyTorch` got early `Python` releases.\n",
"\n",
"The prevalent packages in the Python eco-system used for ML include:\n",
"\n",
"- `pandas` for handling tabular data\n",
"- `matplotlib` and `seaborn` for plotting\n",
"- `scikit-learn` for classical (non-deep-learning) ML\n",
"- `TensorFlow`, `PyTorch` and `Keras` for deep-learning.\n",
"\n",
"`scikit-learn` is very comprehensive and the online-documentation itself provides a good introducion into ML."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## ML lingo: What are \"features\" ?\n",
"A typical and very common situation is that our data is presented as a table, as in the following example:"
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>alcohol_content</th>\n",
" <th>bitterness</th>\n",
" <th>darkness</th>\n",
" <th>fruitiness</th>\n",
" <th>is_yummy</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>3.739295</td>\n",
" <td>0.422503</td>\n",
" <td>0.989463</td>\n",
" <td>0.215791</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4.207849</td>\n",
" <td>0.841668</td>\n",
" <td>0.928626</td>\n",
" <td>0.380420</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>4.709494</td>\n",
" <td>0.322037</td>\n",
" <td>5.374682</td>\n",
" <td>0.145231</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>4.684743</td>\n",
" <td>0.434315</td>\n",
" <td>4.072805</td>\n",
" <td>0.191321</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>4.148710</td>\n",
" <td>0.570586</td>\n",
" <td>1.461568</td>\n",
" <td>0.260218</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" alcohol_content bitterness darkness fruitiness is_yummy\n",
"0 3.739295 0.422503 0.989463 0.215791 0\n",
"1 4.207849 0.841668 0.928626 0.380420 0\n",
"2 4.709494 0.322037 5.374682 0.145231 1\n",
"3 4.684743 0.434315 4.072805 0.191321 1\n",
"4 4.148710 0.570586 1.461568 0.260218 0"
]
},
"metadata": {},
"output_type": "execute_result"
}
],
"features = pd.read_csv(\"data/beers.csv\")\n",
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<i class=\"fa fa-warning\"></i> <strong>Definitions</strong>\n",
"<ul>\n",
" <li>every row of such a matrix is called a <strong>sample</strong> or <strong>feature vector</strong>;</li>\n",
" <li>the cells in a row are <strong>feature values</strong>;</li>\n",
" <li>every column name is called a <strong>feature name</strong> or <strong>attribute</strong>.</li>\n",
"</ul>\n",
"\n",
"Features are also commonly called <strong>variables</strong>.\n",
"</div>"
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The feature names are `alcohol_content`, `bitterness`, `darkness`, `fruitiness` and `is_yummy`.\n",
"<div class=\"alert alert-block alert-warning\">\n",
"<i class=\"fa fa-warning\"></i> <strong>More definitions</strong>\n",
"<ul>\n",
" <li>The first four features have continuous numerical values within some ranges - these are called <strong>numerical features</strong>,</li>\n",
" <li>the <code>is_yummy</code> feature has only a finite set of values (\"categories\"): <code>0</code> (\"no\") and <code>1</code> (\"yes\") - this is called a <strong>categorical feature</strong>.</li>\n",
"</ul>\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A straight-forward application of machine-learning on the previous beer dataset is: **\"can we predict `is_yummy` from the other features\"** ?\n",
"<div class=\"alert alert-block alert-warning\">\n",
"<i class=\"fa fa-warning\"></i> <strong>Even more definitions</strong>\n",
"\n",
"In context of the question above we call:\n",
"<ul>\n",
" <li>the <code>alcohol_content</code>, <code>bitterness</code>, <code>darkness</code>, <code>fruitiness</code> features our <strong>input features</strong>, and</li>\n",
" <li>the <code>is_yummy</code> feature our <strong>target/output feature</strong> or a <strong>label</strong> of our data samples.\n",
" <ul>\n",
" <li>Values of categorical labels, such as <code>0</code> (\"no\") and <code>1</code> (\"yes\") here, are often called <strong>classes</strong>.</li>\n",
" </ul>\n",
" </li>\n",
"</ul>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Most of the machine learning algorithms require that every sample is represented as a vector containing numbers. \n",
"\n",
"Let's look now at two examples of how one can create feature vectors from data which is not naturally given as vectors:\n",
"\n",
"1. Feature vectors from images\n",
"2. Feature vectors from text.\n",
"### 1st Example: How to represent images as feature vectors ?\n",
"In order to simplify our explanations we only consider grayscale images in this section. \n",
"Computers represent images as matrices. Every cell in the matrix represents one pixel, and the numerical value in the matrix cell its gray value.\n",
"So how can we represent images as vectors?\n",
"To demonstrate this we will now load a sample dataset that is included in `scikit-learn`:"
"from sklearn.datasets import load_digits\n",
"\n",
"execution_count": 4,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Optical recognition of handwritten digits dataset\n",
"--------------------------------------------------\n",
"\n",
"**Data Set Characteristics:**\n",
"\n",
" :Number of Instances: 5620\n",
" :Number of Attributes: 64\n",
" :Attribute Information: 8x8 image of integer pixels in the range 0..16.\n",
" :Missing Attribute Values: None\n",
" :Creator: E. Alpaydin (alpaydin '@' boun.edu.tr)\n",
" :Date: July; 1998\n",
"\n",
"This is a copy of the test set of the UCI ML hand-written digits datasets\n",
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
"http://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits\n",
"\n",
"The data set contains images of hand-written digits: 10 classes where\n",
"each class refers to a digit.\n",
"\n",
"Preprocessing programs made available by NIST were used to extract\n",
"normalized bitmaps of handwritten digits from a preprinted form. From a\n",
"total of 43 people, 30 contributed to the training set and different 13\n",
"to the test set. 32x32 bitmaps are divided into nonoverlapping blocks of\n",
"4x4 and the number of on pixels are counted in each block. This generates\n",
"an input matrix of 8x8 where each element is an integer in the range\n",
"0..16. This reduces dimensionality and gives invariance to small\n",
"distortions.\n",
"\n",
"For info on NIST preprocessing routines, see M. D. Garris, J. L. Blue, G.\n",
"T. Candela, D. L. Dimmick, J. Geist, P. J. Grother, S. A. Janet, and C.\n",
"L. Wilson, NIST Form-Based Handprint Recognition System, NISTIR 5469,\n",
"1994.\n",
"\n",
".. topic:: References\n",
"\n",
" - C. Kaynak (1995) Methods of Combining Multiple Classifiers and Their\n",
" Applications to Handwritten Digit Recognition, MSc Thesis, Institute of\n",
" Graduate Studies in Science and Engineering, Bogazici University.\n",
" - E. Alpaydin, C. Kaynak (1998) Cascading Classifiers, Kybernetika.\n",
" - Ken Tang and Ponnuthurai N. Suganthan and Xi Yao and A. Kai Qin.\n",
" Linear dimensionalityreduction using relevance weighted LDA. School of\n",
" Electrical and Electronic Engineering Nanyang Technological University.\n",
" 2005.\n",
" - Claudio Gentile. A New Approximate Maximal Margin Classification\n",
" Algorithm. NIPS. 2000.\n"
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's plot the first ten digits from this data set:"
"image/png": "iVBORw0KGgoAAAANSUhEUgAACPMAAAEZCAYAAAD4nfIqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3X+UJGZd5/vPN4lIgJAJYAD5kU6yAlHWDASRq14zAcKiqAl60SvrboYVxbMqyR7wEnFdEj27BPdKhouuhlVIxNXdzXGdiKJAkOBVBNeYyV5+hIhDByH80JAZQgi/kuf+UTXShvQkpOvpeqrq9TpnTk13ld9+TPOerq7+dlW11gIAAAAAAAAAAMzfEfM+AAAAAAAAAAAAMGGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZhy9TVQ+rqldV1d9U1Wer6uNV9Yaqetq8zwaroKqOqarvqaqfr6o/rKq/r6o2/fO4eZ8PVkVVPbqqzpt+DfxQVX2uqm6pqmur6qKqevi8zwiroKqeNP2a+EdV9YGqOjjt8SNVdUVVnT3vM8IqqqoHVNXfbrifunveZ4JVUFW7N3S32Z9Pz/ucsEqq6rFV9eqqen9V3Tq9v/q+qnptVZ0+7/PBsroHXw83/tEidFZVR1TV86rqyqr6u6r6QlUdqKp3VdXPVNUx8z4jrIKa+MGqektV3TR9HHW9qn6tqh4z7/PxlanW2rzPwECq6huT/HGSB0/f9akkD8hk8asleWlr7aI5HQ9WwvSHkr+7ydWntNau287zwCqqqkcluSFJbXj3p5LcP8mR07dvTvJ9rbW3bfPxYKVU1a8mecGGd306yVFJ7rvhfb+T5Adba1/YzrPBKquqPUnO3fCu57XWLp3TcWBlTBfnXpfkC0k+ucnNbm2tnbxth4IVVlUvTPIfk9xn+q4731f99dba8+dxNlh2VfWxu7nJA5McneTzSb62tXZT/1PBaqqq+yV5Q5Knbnj3wUw6PPT46g1Jntpa27/Nx4OVUVX3SfLfkhz65ccvJrklyXHTt29L8n+21n5vDsfjXvDMPPyDqjo6ye9lsshzTZLHt9aOzSTwX8zkC+5/qKpnzO+UsDI+keSNSS5M8qNzPgusokMLO3+Q5DlJHjT9mni/JN+Z5IOZfH3cW1UPm88RYWX8eZJ/k+S0JMe01o5prR2d5NGZ/OAkSb4vyflzOh+snKp6YpKfSPKueZ8FVtg7WmsP2+SPRR7YBlX1giSvymR55xVJTthwX/XhSf5lknfM8Yiw1A7zdfBhrbWHJbl+etPft8gD3f1sJos8LclPJ9nRWtuRyXLrDyY5kOSEJL82txPCargok0WeL2byC1gPbK09KMmjMvllyKOT/Neq8j3jgvDMPPyDqjovycWZ/AbJ41prH7nT9b+byT8Af9VaO20OR4SVUFVHttZu3/D2WiaLA4ln5oFtUVXHJllrrV27yfWPy2Tx9b5JLmitXbid5wO+pKpen+SHkuz3w0vor6qOyGSJ5wlJvinJX02v8sw8sA02PDPP21tru+Z7Glhd08dq3pPJL3z8aGvtP8/1QMA/UlU7M3ncJknO8gwE0FdV3ZDJL129trX2w3dx/e5M7sMmk1+avHkbjwcroaqOT/K3mTxj5Mtbay+90/VHZXL/9TFJfqu19s+3/5R8pTwzDxsdiva37rzIM3XoN5+fWFWP3aYzwcrZuMgDzEdr7eBmizzT669L8s7pmxZcYb7+5/Tya+d6ClgdP5nkSUl+pbV2zd3dGACW1LmZLPK8yyIPDOmc6eWhZz8H+nro9HKz7xGv3vD3+3U+C6yqp+ZLL/26585Xtta+mOTV0ze/t6oesF0H496zzEOSpKqOyZd+GPmmTW72zkxe4zJJntb9UAAwtkNP0XzkYW8F9PYt08sPHvZWwJZV1SOS/HySjyf5t3M+DgDM03Onl78911MAX2b6zAOHGv2t6Q8vgb7Wp5dP2OT6Qz9//PgmTyYAbN0J08sDrbVPbHKbQ6/8cd8k39b/SGyVZR4OOSVJTf/+nru6QWvtjiTvn7759dtxKAAY0fSBoW+dvvnueZ4FVlFVPaCqvrGqfjnJD0zf/UvzPBOsiFcnOSbJi1trB+/uxkBX31BV76mq26rqlqp6d1VdXFUnzvtgsOyq6uQkx0/fvKaqnlJVb6iqm6ZNXldV/3H6UgfA9vuOfKnRy+Z5EFghh56l7nlVdX5VHZskVXWfqvqBJBcnaUlePK8Dwgpo08vD/fLxURv+/g0dz8KMWObhkIdv+PuNh7ndoesefpjbAMCy+/EkD0tyRzwwBNuiqh5ZVa2qWpJbklyb5F8n+WySn22t/ae5HhCWXFV9d5JnJ7mqtfab8z4PkIdk8otZn8nktyq/Icl5Sd5TVc893P8hsGVft+Hvu5L8aZLvSvJVmfwQ5bGZ/LByX1X5IQlsv93Ty2tba/vmeRBYIXuS/HImTxrw8iQHqupAktuS/NdMng3ke3wvCV3dML08pqoeucltNj5Zh5/1LwDLPBxy/w1/v+0wt/vM9NLr6AGwkqrqGzP5pjRJfqm19t55ngdWyO2ZvLTPx5N8fvq+L2bS4y/P61CwCqrq/pk8+9UXMlloBebnxiQvS/L4JPdtrT04k8donpXkvUmOTnJZVX37/I4IS2/Hhr+/LMn1SZ7SWntgJj1+Z5JPZPIDkt+ZPrMrsA2q6kGZLNclfvkKtk1r7fZMFstflMljNUlybL70c+hjknzNHI4Gq+SqTB63SZKfuvOVVXXfJC/c8K5jtuFMbJFlHgCAe6iqHp5kbyY/JLk6yUvmeyJYHa21j7bWHtZae1gmDT42yW8kuTB+6xl6+7kkj05ysSVWmK/W2ptbaz/XWntPa+3z0/d9rrX2xiTfkuQDmTx1+kXzPCcsuY2Pqbckz26tvStJWmt3tNb+MMm/ml7/2CTfu83ng1X2g0nuk8kywX+Z81lgZVTVw5L8WZJfzKS9UzNZcP26JD+d5KQkr62ql286BNiS1trHk1wyffMnq+rnq+oRVfVVVfXEJH+Q5IR8aeHujnmck6+MZR4OuXXD348+zO3uN738dMezAMBwpr/d9eYkJyb56yTPaq19dr6ngtU0/SHJ9a21H07yykyWDF5fVb6/gRmrqp1Jzk3yt5ks9QCDaq0dTPIfpm8+paoeMs/zwBLb+LjoH7XW3n/nG7TW/iCTZ+xJkqdty6mAJDlnevmHrbVPzPUksFp+I8mTk/x6a213a+1/tdZuba19oLV2UZIXTG/3f/llLOjqp5L8YSYvefdvk3w4k2c4vzrJU5P8bJKbp7c9MI8D8pXxYDeH3Ljh7197mNsduu6jHc8CAEOpqmOTvCmTlzP4UJKnTzfdgfl79fTyCdM/wGy9KsmRSX4mSVXVAzb+2XC7r56+7353PQbYJu+aXlYmS+jA7G18HPXLFnnu4rpHdTwLMFVVpyT5pumbXmILtklVfX2SM6dvXnxXt2mtvT7JTZn8XPq7t+losHKmv3z8XUmem+T3k/zN9M/vJflnSV6RL71k7F/P44x8ZbxeL4dcl8nTwlaSb8hdfCM6/U3nx07f9NTqAKyEqrp/kjcmeVKSj2WyyPOh+Z4K2OAjG/5+cia/aQLMzgnTy9+4m9v96vTPDUnWeh4IAObsvZm8LME9/UXZ1vEswJfsnl5+Mskb5ngOWDWnbPj7Bw9zu/1JHhzfL0JXrbU7kvz29M8/Mn25ra+avvnn23ku7h3PzEOSpLV2S5K/nL555iY3++Ykx07//tbuhwKAOauqozN5AOhbMvntkae31mysw1g2PuuAl4IFYNV984a/r8/rELDMWmufyZd++PHYw9z00HXrXQ8EpKqOTPJD0zd/u7X2+XmeB1bMHRv+/ujD3O7QL4rc0vEswOH94PRyX2vtfXM9CfeIZR42+q3p5T+vqoffxfUvnl5efVevBQ0Ay6Sq7pPkfyQ5I5PXj31Ga+098z0VrJaqOrKq6m5u9lPTyy/Gb5TAzLXW1lprtdmfDTd93vR9a/M6Kyy7u/uaWFUPTHL+9M2/aK39Xf9Twco69Ix1z6yqL1voqapnJXnM9M03btupYHU9PcnXTv/uJbZge1274e8/clc3qKrvTnL89M133dVtgL6q6tQkPzF98+XzPAv3nGUeNrokk6dEPybJ709f5zJVdUxV/UKS753e7qVzOh+sjKp6yKE/SY7bcNWOjddNX/4OmLHpb3T9VpJnZvLbIt/RWvur+Z4KVtKjkvxlVf2rqnrkoXdW1RFVtbOq/kuS50/f/erW2s1zOSUAbI8TquqdVfXDVfUPv/VcVfepqmcm+bNMlgfuSPLT8zokrIjXZvJyW0cm+R9V9eTkH+6nPjPJr09v985Y5oHtcM708r2ttf8515PAimmt7U/y5umb51XVy6vq+CSpqgdU1e4kl06vX0/ye9t9RlgVVXVGVb2oqv7J9Gccqapjq+oFSf44yX2T/PfW2n+f60G5x6o1L9nLl0y38t6ayetWJsmnkjwgk8WvluSlrbWL5nQ8WBlVdU//cT6xtbbe8yywiqrq25O8ffrmZ5McPMzN/7a19k39TwWrp6rW8o9fb/2zmbyU1jFJvnrD+y9N8iOttS9u19mAiQ33W5/XWrt0nmeBZbfJ18VbkzwwyVdN3/eZJD/WWnv9th4OVlBVnZTkqkwW0JPJL4IcmeR+07ffm8kzvH5k+08Hq2P6zHQfS3J0kpe01n5hzkeClTN9tY+3Jjllw7tvyeTxm0M+nskvTF6znWeDVTJdnnvd9M0vZtLhjiSHnuX1t5Oc01r7wvafjnvjqHkfgLG01q6tqsdn8htc35XkEUluSvIXSS5urb11nucDgG2y8Vmv7jv9s5nPdj4LrLIbk/xAkqcleXKSh2eydP7ZJH+Tyctqva619mdzOyEAbJ+PJ3lhkm9LcmqSr0lybCYLPX+dyQ9QfqW1dsPcTggrpLW2v6r+aSYv+/rsJCdm8sxYf5Xk8kyeOfLWOR4RVsX3Z7LIc0eS35zzWWAltdY+WlWnJfnRTF7l4/GZ3E/9VJIPJPmDTL4uehlY6OtPk+xJ8u1JTshkoe4jmTyG+uuttTfN8WzcC56ZBwAAAAAAAAAABnHE3d8EAAAAAAAAAADYDpZ5AAAAAAAAAABgEJZ5AAAAAAAAAABgEJZ5AAAAAAAAAABgEJZ5AAAAAAAAAABgEJZ5AAAAAAAAAABgEJZ5AAAAAAAAAABgEJZ5AAAAAAAAAABgEJZ5AAAAAAAAAABgEEfN+wB3p6o+mOSBSdbnfBTYLmtJPtVaO3HeB9lIi6ygtWgRRrAWLcII1qJFGMFatAgjWIsWYQRr0SKMYC1ahBGsRYswgrXMqMVqrW39OB1V1U1JHjTvc9xbO3bs6Dr/EY94RNf5t9xyS9f5H/nIR7rNvv3227vN3g6ttZr3GTZa9BZ7e8xjHtN1/pFHHtl1/o033tht9sGDB7vN3g5aXCwPeMADus4/+eSTu86/7bbbus2+/vrru83eDlqcrYc+9KFd5/e+j/q5z32u6/zrrruu22z3UWdr0Vvsrfd9yBNOOKHr/P3793edv8i0OFu9v5/r/XXrhhtu6DqfzWlxsSz6Yzfve9/7us5fZFqcreOPP77r/N6t9P6ZzNFHH91tdu/vF9/97nd3mXv77benqnLHHXdocYYe+chHdp3fu5Wbbrqp6/xPfOIT3WZ77Ga2Fr3Fk046qev8o47q+9wri/5zgUU2ixaHf2aeTLb0FjbwM888s+v8iy66qOv8K6+8suv8888/v9vsm2++udvsFbWeBW6xt9e85jVd5/e+Y/2yl72s2+wrrrii2+wVtR4tbupJT3pS1/l79+7tOn/fvn3dZu/atavb7BW1ngVucffu3V3n976P2vsH/D3/LXEfdebWs8At9nbMMcd0nf/KV76y6/yzzz6763xmaj0L3GLv7+fW19e7zu/9dZ2Fsp4FbrG3RX/sZufOnV3nM1PrWeAWn/vc53ad37uV3vchTz311G6ze/9S5NraWpe5vX8pfAvWs8AtvvjFL+46v3crl156adf5e/bs6Tb7wIED3WavqPUscIu9H/vo/XXRzwUW2xHzPgAAAAAAAAAAADBhmQcAAAAAAAAAAAZhmQcAAAAAAAAAAAZhmQcAAAAAAAAAAAZhmQcAAAAAAAAAAAYxs2WeqnpkVb22qm6sqs9V1XpV7amq42b1MYC7p0UYgxZhDFqEMWgRxqBFGIMWYQxahDFoEcagRRjPUbMYUlUnJ3lHkuOTXJHkuiRPTnJukmdW1be21m6axccCNqdFGIMWYQxahDFoEcagRRiDFmEMWoQxaBHGoEUY06yemec/ZRL3C1trZ7fWzm+tPTXJxUkem+Tfz+jjAIenRRiDFmEMWoQxaBHGoEUYgxZhDFqEMWgRxqBFGNCWl3mmm3rPSLKe5JfvdPXLktya5F9U1f23+rGAzWkRxqBFGIMWYQxahDFoEcagRRiDFmEMWoQxaBHGNYuX2Tpjevnm1todG69ord1SVX+WyT8AT0ny1s2GVNXVm1z1uBmcEVaBFmEMWoQxaBHGoEUYgxZhDFqEMWgRxqBFGIMWYVCzeJmtx04vr9/k+r+eXj5mBh8L2JwWYQxahDFoEcagRRiDFmEMWoQxaBHGoEUYgxZhULN4Zp5jp5cHN7n+0Pt3HG5Ia+20u3r/dIvviffuaLBStAhj0CKMQYswBi3CGLQIY9AijEGLMAYtwhi0CIOaxTPzAAAAAAAAAAAAMzCLZZ5D23jHbnL9ofcfmMHHAjanRRiDFmEMWoQxaBHGoEUYgxZhDFqEMWgRxqBFGNQslnneP73c7HXyvm56udnr7AGzoUUYgxZhDFqEMWgRxqBFGIMWYQxahDFoEcagRRjULJZ53ja9fEZV/aN5VXVMkm9N8pkk75zBxwI2p0UYgxZhDFqEMWgRxqBFGIMWYQxahDFoEcagRRjUlpd5Wmt/k+TNSdaS/Pidrr4wyf2TvL61dutWPxawOS3CGLQIY9AijEGLMAYtwhi0CGPQIoxBizAGLcK4jprRnH+d5B1J/p+qelqS9yX55iRnZPKUWz8zo48DHJ4WYQxahDFoEcagRRiDFmEMWoQxaBHGoEUYgxZhQLN4ma1DG3tPSnJpJmG/KMnJSV6V5CmttZtm8XGAw9MijEGLMAYtwhi0CGPQIoxBizAGLcIYtAhj0CKMaVbPzJPW2t8med6s5gH3jhZhDFqEMWgRxqBFGIMWYQxahDFoEcagRRiDFmE8M3lmHgAAAAAAAAAAYOss8wAAAAAAAAAAwCBm9jJb3LWLLrqo6/yTTjqp6/zjjjuu6/xPfvKT3WZ///d/f7fZSXL55Zd3nc9iOXDgQNf5p59+etf5Z5xxRrfZV1xxRbfZLJ6dO3d2nf+2t72t6/yDBw92nb+2ttZ1Poul5/3I5zznOd1mJ8kLXvCCrvMvueSSrvNPO+20brOvvPLKbrPhznbv3t11/r59+7rOh+3S+z5Y7+/nzjnnnK7zb7jhhm6z3f9lo7POOqvr/N4tXnjhhV3nw7Lo/Tjqeeedt7Dzd+zY0W120v+/PbPV+3HU3np/P7pr166FnE0fPb+v6H0ftbfWWtf51157bbfZi/7v4Cx4Zh4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABiEZR4AAAAAAAAAABjEUfM+wAhOO+20brNPOumkbrOT5OSTT+46f//+/V3nv+Utb+k2u+fnNUkuv/zyrvOZrZ07d3adv2vXrq7ze9u3b9+8j8CKOPvss7vOv/baa7vO37t3b9f5L3vZy7rOZ7G85jWv6Tb7Fa94RbfZSfKXf/mXXef3vo965ZVXdp0Ph+zYsaPr/N27d3edv2fPnq7z19bWus7vaX19fd5H4Ctw4MCBrvNPOOGErvMPHjzYdf5VV13VbXbvfwd7f26ZrQsvvHDeR9iS3t8vwnbpfR+vtwsuuKDr/J73URf9MWZmq/dj9r2/Z+n9/WjP+3m9W+x5/3pV9f6+oqe3v/3tXef3bt3Xrr48Mw8AAAAAAAAAAAzCMg8AAAAAAAAAAAzCMg8AAAAAAAAAAAzCMg8AAAAAAAAAAAzCMg8AAAAAAAAAAAzCMg8AAAAAAAAAAAxiy8s8VfXgqnp+Vf1uVX2gqm6rqoNV9adV9cNVZWEItoEWYQxahDFoEcagRRiDFmEMWoQxaBHGoEUYgxZhXEfNYMZzkvxKko8meVuSDyV5aJLvTfJrSb6jqp7TWmsz+FjA5rQIY9AijEGLMAYtwhi0CGPQIoxBizAGLcIYtAiDmsUyz/VJvifJH7TW7jj0zqp6aZK/SPJ9mcT+OzP4WMDmtAhj0CKMQYswBi3CGLQIY9AijEGLMAYtwhi0CIPa8tNitdb+uLX2ho1xT9//sSS/On1z11Y/DnB4WoQxaBHGoEUYgxZhDFqEMWgRxqBFGIMWYQxahHH1fo27L0wvv9j54wCHp0UYgxZhDFqEMWgRxqBFGIMWYQxahDFoEcagRZijWbzM1l2qqqOS/Mvpm390D25/9SZXPW5mh4IVpEUYgxZhDFqEMWgRxqBFGIMWYQxahDFoEcagRZi/ns/Mc1GSxyd5Y2vtTR0/DnB4WoQxaBHGoEUYgxZhDFqEMWgRxqBFGIMWYQxahDnr8sw8VfXCJC9Kcl2Sf3FP/m9aa6dtMuvqJE+c3elgdWgRxqBFGIMWYQxahDFoEcagRRiDFmEMWoQxaBHGMPNn5qmqn0jyqiTvTXJGa+2Ts/4YwN3TIoxBizAGLcIYtAhj0CKMQYswBi3CGLQIY9AijGOmyzxVdV6SVyd5dyZxf2yW84F7RoswBi3CGLQIY9AijEGLMAYtwhi0CGPQIoxBizCWmS3zVNVLklycZF8mcX9iVrOBe06LMAYtwhi0CGPQIoxBizAGLcIYtAhj0CKMQYswnpks81TVzya5KMnVSZ7WWvv7WcwFvjJahDFoEcagRRiDFmEMWoQxaBHGoEUYgxZhDFqEMR211QFVdU6Sn0tye5L/N8kLq+rON1tvrV261Y8FbE6LMAYtwhi0CGPQIoxBizAGLcIYtAhj0CKMQYswri0v8yQ5cXp5ZJLzNrnN25NcOoOPBWxOizAGLcIYtAhj0CKMQYswBi3CGLQIY9AijEGLMKgtv8xWa+2C1lrdzZ9dMzgrcBhahDFoEcagRRiDFmEMWoQxaBHGoEUYgxZhDFqEcW15mQcAAAAAAAAAAJgNyzwAAAAAAAAAADCIo+Z9gBEcd9xx3WZfffXV3WYnyf79+7vO7633fx8Wy3nnbfZSnFt3wQUXdJudJMcee2zX+b1dddVV8z4CK2LPnj1d56+vr3ed3/v8V1xxRdf5LJae9/NOOumkbrO3Y/6VV17ZdX7P7w9uvvnmbrNZPLt37+46f21trev8Sy+9tOv8nl93Dxw40G120v/7D2ar933IU089tev83t+P7tu3r9vs3i2yWHbs2NF1/rXXXtt1fs9W4M527dq1kLO3Q8/HmHs7++yzu87vff+d2er9+brmmmu6zu/9/WjP+5G9vz9g9hb5c9b73/69e/d2nd/7Pvyq88w8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwCMs8AAAAAAAAAAAwiKPmfYARHHfccd1mX3nlld1mL4Oe/+1vvvnmbrPpY8+ePd1mX3rppd1mJ4v/v7cdO3bM+wgMpOf/Hs4777xus5Pk7LPP7jq/t927d8/7CKyI/fv3d53/oAc9qOv8t7zlLQs7/8wzz+w2O1n8+yQjOuuss7rNvvjii7vNTpLLLrus6/zezj333G6zn/e853WbzeLpfR9y165dXefv3Lmz6/ze/1b11PNxBmav92MT6+vrXef3/n5379693Wb3/m/D7PX8nPX+utL762JvPe83XHXVVd1ms3gW/TH7008/vev8E088sdtsXxcXz4EDB7rNvvbaa7vNTvo/lveqV72q6/ye9xvW1ta6zU4Wo3XPzAMAAAAAAAAAAIOwzAMAAAAAAAAAAIOwzAMAAAAAAAAAAIOwzAMAAAAAAAAAAIOwzAMAAAAAAAAAAIPotsxTVT9UVW365/m9Pg5weFqEMWgRxqBFGIMWYQxahPnTIYxBizAGLcIYtAhj6LLMU1WPSvJLST7dYz5wz2gRxqBFGIMWYQxahDFoEeZPhzAGLcIYtAhj0CKMY+bLPFVVSV6X5KYkvzrr+cA9o0UYgxZhDFqEMWgRxqBFmD8dwhi0CGPQIoxBizCWHs/M88IkT03yvCS3dpgP3DNahDFoEcagRRiDFmEMWoT50yGMQYswBi3CGLQIA5lCobgpAAATwklEQVTpMk9VnZLkoiSvaq39ySxnA/ecFmEMWoQxaBHGoEUYgxZh/nQIY9AijEGLMAYtwniOmtWgqjoqyeuTfCjJS+/F//3Vm1z1uK2cC1aNFmEMWoQxaBHGoEUYgxZh/rba4XSGFmGLtAhj0CKMQYswppkt8yT5d0mekOTbWmu3zXAu8JXRIoxBizAGLcIYtAhj0CLMnw5hDFqEMWgRxqBFGNBMlnmq6psz2dL7xdban9+bGa210zaZfXWSJ27heLAytAhj0CKMQYswBi3CGLQI8zeLDhMtwlZpEcagRRiDFmFcR2x1wPRpt34jyfVJfnbLJwLuFS3CGLQIY9AijEGLMAYtwvzpEMagRRiDFmEMWoSxbXmZJ8kDkjwmySlJPltV7dCfJC+b3uY/T9+3ZwYfD7hrWoQxaBHGoEUYgxZhDFqE+dMhjEGLMAYtwhi0CAObxctsfS7Jr29y3RMzeX29P03y/iT3+qm5gLulRRiDFmEMWoQxaBHGoEWYPx3CGLQIY9AijEGLMLAtL/O01m5L8vy7uq6qLsgk8staa7+21Y8FbE6LMAYtwhi0CGPQIoxBizB/OoQxaBHGoEUYgxZhbLN4mS0AAAAAAAAAAGAGLPMAAAAAAAAAAMAgui7ztNYuaK2Vp96C+dIijEGLMAYtwhi0CGPQIsyfDmEMWoQxaBHGoEWYP8/MAwAAAAAAAAAAg7DMAwAAAAAAAAAAg7DMAwAAAAAAAAAAgzhq3gcYwc0339xt9mmnndZt9nY47rjjus7v+d/n8ssv7zYbls3OnTu7zd63b1+32fRxwQUXdJt97rnndpu9Hc4+++yu8w8cONB1PmyXnvevk+TMM8/sOv+SSy7pNvslL3lJt9lJcv7553edv4oOHjy4kLOT5Jxzzuk6v+d9yN727t077yOwQq666qp5H2FYa2tr8z4CA1lfX+86//TTT+86f8eOHV3nX3zxxd1mP+EJT+g2O/HYUA89e+n92Edrrev83uf3dZ2Nen5P9La3va3b7CS58MILu87vfT+v5/d0vf8d6X2fh9nq/dhH7/mLfD9sz549Xef3bn0WPDMPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAMwjIPAAAAAAAAAAAM4qh5H2AE+/fv7zb7tNNO6zY7SZ7znOcs9PyeXvGKV8z7CAAL6dJLL+02e9euXd1mJ8mpp57adf7evXu7zr/iiiu6zX7d617XbXbS9+zM3kUXXdR1/pVXXtl1/nHHHdd1/tOf/vRusy+//PJus+njqquu6jZ7x44d3WYnyc6dO7vO7/nfJkkuu+yybrMPHDjQbTaL56yzzuo6/+DBg13nX3DBBV3n99T7/jWLpef3okly8cUXd52/vr7edf7a2lq32WeffXa32Umyb9++rvOZrT179nSd3/vr4tvf/vau82Gjnv/2926ld+s9v24lyTXXXNNt9u7du7vNThb7/juz1/t+Uu/We/bS+z7qIvDMPAAAAAAAAAAAMAjLPAAAAAAAAAAAMAjLPAAAAAAAAAAAMAjLPAAAAAAAAAAAMAjLPAAAAAAAAAAAMAjLPAAAAAAAAAAAMIiZLvNU1dOq6ner6mNV9bmqurGq3lRV3znLjwMcnhZhDFqEMWgRxqBFGIMWYQxahDFoEcagRRiDFmEsR81qUFX9QpKfSvLhJL+X5O+TfE2S05LsSvLGWX0sYHNahDFoEcagRRiDFmEMWoQxaBHGoEUYgxZhDFqE8cxkmaeqfiSTuC9L8qOttc/f6fqvmsXHAQ5PizAGLcIYtAhj0CKMQYswBi3CGLQIY9AijEGLMKYtv8xWVX11kn+f5EO5i7iTpLX2ha1+HODwtAhj0CKMQYswBi3CGLQIY9AijEGLMAYtwhi0COOaxTPznJnJU2ztSXJHVT0ryeOTfDbJX7TW/vyeDKmqqze56nEzOCOsAi3CGLQIY9AijEGLMAYtwhi0CGPQIoxBizAGLcKgZrHM803Ty88muSaTuP9BVf1Jkv+jtfZ3M/hYwOa0CGPQIoxBizAGLcIYtAhj0CKMQYswBi3CGLQIg5rFMs/x08ufSvLeJP97kn1JTkzyfyd5RpLLk+w63JDW2ml39f7pFt8TZ3BOWHZahDFoEcagRRiDFmEMWoQxaBHGoEUYgxZhDFqEQR0xwxlfTPI9rbU/ba19urX2/yV5dpIPJzm9qv63GXwsYHNahDFoEcagRRiDFmEMWoQxaBHGoEUYgxZhDFqEQc1imefA9PKa1tr6xitaa59J8qbpm0+ewccCNqdFGIMWYQxahDFoEcagRRiDFmEMWoQxaBHGoEUY1CyWed4/vTywyfU3Ty+PnsHHAjanRRiDFmEMWoQxaBHGoEUYgxZhDFqEMWgRxqBFGNQslnnemqQl+fqquqt5j59efnAGHwvYnBZhDFqEMWgRxqBFGIMWYQxahDFoEcagRRiDFmFQW17maa3dkOQNSR6d5NyN11XVM5L8s0w2+f5oqx8L2JwWYQxahDFoEcagRRiDFmEMWoQxaBHGoEUYgxZhXEfNaM6PJ3lCkldW1bOSXJPkxCRnJ7k9yfNbawdn9LGAzWkRxqBFGIMWYQxahDFoEcagRRiDFmEMWoQxaBEGNJNlntbah6vqtCT/Lsn3JPn2JJ/KZIvv5a21v5jFxwEOT4swBi3CGLQIY9AijEGLMAYtwhi0CGPQIoxBizCmWT0zT1prf5fkJ6d/gDnRIoxBizAGLcIYtAhj0CKMQYswBi3CGLQIY9AijOeIeR8AAAAAAAAAAACYsMwDAAAAAAAAAACDmNnLbC2y/fv3d5t9/vnnd5udJBdddFHX+VdffXXX+U960pO6zodDDhw40HX+FVdc0XX+WWed1XX+rl27us2+9NJLu82mj3379nWbvXPnzm6zt2P+BRdc0HV+z9bX19e7zU76/zvIbN18881d519yySVd5/d2+eWXd5v9ghe8oNtsuLPe94GPPfbYrvPdj2S7nHHGGV3nn3vuuV3n93bZZZd1m33VVVd1m83i6f3v/traWtf5u3fv7jq/Zy979+7tNpvF0/NxwiQ555xzus7vfR8YNur5v7fe95N6PzZ08ODBrvN7Pha5Z8+ebrNZPL3/99D7Zxo7duzoOr/n/YaeP6taFJ6ZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABmGZBwAAAAAAAAAABlGttXmf4bCq6qYkD5r3Oe6thzzkIV3nP/ShD+06/zOf+UzX+R/84Ae7zl9krbWa9xk2WvQWezvppJO6zt+xY0fX+TfddFO32TfccEO32dtBi4vl6KOP7jr/4Q9/eNf5PVv/xCc+0W12knz4wx/uOl+Ls9X7PmTv+8C93XLLLd1mf+hDH+o2eztocbHc5z736Tr/8Y9/fNf5119/fbfZn/70p7vN3g5anK1HPvKRXecff/zxXef35vvFzWlxto488siu83t/P/fgBz+46/ye91E/+tGPdpudJLfddlvX+VqcrVNOOaXr/BtvvLHr/IMHD3adz+a0OFuL/jOH22+/vev8nl8Xe99H7f3fRouz1fv7xfvd735d5/e+j72+vt5tdu/7kL3NosVFWOb5YJIHJlm/h/8nj5teXtflQMzLKn1e15J8qrV24rwPspEWmVqlz+tatMi4VunzuhYtMq5V+ryuRYuMa5U+r2vRIuNapc/rWrTIuFbp87oWLTKuVfq8rkWLjGuVPq9r0SLjWqXP61pm1OLwyzxfqaq6Oklaa6fN+yzMjs/r4vE5W04+r4vH52w5+bwuHp+z5eTzunh8zpaTz+vi8TlbTj6vi8fnbDn5vC4en7Pl5PO6eHzOlpPP6+LxOVtOPq/3zhHzPgAAAAAAAAAAADBhmQcAAAAAAAAAAAZhmQcAAAAAAAAAAAZhmQcAAAAAAAAAAAZhmQcAAAAAAAAAAAZRrbV5nwEAAAAAAAAAAIhn5gEAAAAAAAAAgGFY5gEAAAAAAAAAgEFY5gEAAAAAAAAAgEFY5gEAAAAAAAAAgEFY5gEAAAAAAAAAgEFY5gEAAAAAAAAAgEFY5gEAAAAAAAAAgEFY5gEAAAAAAAAAgEEszTJPVT2yql5bVTdW1eeqar2q9lTVcfM+G/fO9HPYNvnzsXmfj7umxeWjxcWkxeWjxcWkxeWjxcWkxeWjxcWkxeWjxcWkxeWjxcWkxeWjxcWkxeWjxcWkxeWjxa07at4HmIWqOjnJO5Icn+SKJNcleXKSc5M8s6q+tbV20xyPyL13MMmeu3j/p7f7INw9LS41LS4QLS41LS4QLS41LS4QLS41LS4QLS41LS4QLS41LS4QLS41LS4QLS41LS4QLS41LW5BtdbmfYYtq6o3JXlGkhe21l694f2vTPJvklzSWvuxeZ2Pe6eq1pOktbY235NwT2lxOWlx8WhxOWlx8WhxOWlx8WhxOWlx8WhxOWlx8WhxOWlx8WhxOWlx8WhxOWlx8WhxOWlx6xZ+mWe6qfeBJOtJTm6t3bHhumOSfDRJJTm+tXbrXA7JvSLwxaLF5aXFxaLF5aXFxaLF5aXFxaLF5aXFxaLF5aXFxaLF5aXFxaLF5aXFxaLF5aXFxaLF5aXFrVuGl9k6Y3r55o1xJ0lr7Zaq+rNMNvmekuSt2304tuyrq+qHkjw6ya1J/leSP2mt3T7fY3EXtLjctLg4tLjctLg4tLjctLg4tLjctLg4tLjctLg4tLjctLg4tLjctLg4tLjctLg4tLjctLgFy7DM89jp5fWbXP/XmQT+mAh8ET0syevv9L4PVtXzWmtvn8eB2JQWl5sWF4cWl5sWF4cWl5sWF4cWl5sWF4cWl5sWF4cWl5sWF4cWl5sWF4cWl5sWF4cWl5sWt+CIeR9gBo6dXh7c5PpD79+xDWdhtl6X5GmZRH7/JP80ySVJ1pL8YVWdOr+jcRe0uLy0uFi0uLy0uFi0uLy0uFi0uLy0uFi0uLy0uFi0uLy0uFi0uLy0uFi0uLy0uFi0uLy0uEXL8Mw8LKnW2oV3ete7k/xYVX06yYuSXJDk2dt9Llg1WoQxaBHGoEUYgxZhDFqEMWgRxqBFGIMWYQxa3LpleGaeQ9t4x25y/aH3H9iGs7A9fnV6+e1zPQV3psXVo8UxaXH1aHFMWlw9WhyTFlePFsekxdWjxTFpcfVocUxaXD1aHJMWV48Wx6TF1aPFe2gZlnneP718zCbXf930crPX2WPx/N308v5zPQV3psXVo8UxaXH1aHFMWlw9WhyTFlePFsekxdWjxTFpcfVocUxaXD1aHJMWV48Wx6TF1aPFe2gZlnneNr18RlX9o/9/quqYJN+a5DNJ3rndB6Obp0wv98/1FNyZFlePFsekxdWjxTFpcfVocUxaXD1aHJMWV48Wx6TF1aPFMWlx9WhxTFpcPVockxZXjxbvoYVf5mmt/U2SNydZS/Ljd7r6wkw2ul7fWrt1m4/GFlTVKVX1Zdt4VbWW5Jemb/7mdp6Jw9PictLi4tHictLi4tHictLi4tHictLi4tHictLi4tHictLi4tHictLi4tHictLi4tHictLibFRrbd5n2LKqOjnJO5Icn+SKJO9L8s1JzsjkKbe+pbV20/xOyFeqqi5I8qIkf5LkhiS3JDk5ybOS3DfJG5M8u7X2+XmdkS+nxeWjxcWkxeWjxcWkxeWjxcWkxeWjxcWkxeWjxcWkxeWjxcWkxeWjxcWkxeWjxcWkxeWjxdlYimWeJKmqRyX5uSTPTPLgJB9N8rtJLmyt3TzPs/GVq6rTk/xYkickeVgmW5cHkuxL8vpMNjCX43+8S0aLy0WLi0uLy0WLi0uLy0WLi0uLy0WLi0uLy0WLi0uLy0WLi0uLy0WLi0uLy0WLi0uLy0WLs7E0yzwAAAAAAAAAALDojpj3AQAAAAAAAAAAgAnLPAAAAAAAAAAAMAjLPAAAAAAAAAAAMAjLPAAAAAAAAAAAMAjLPAAAAAAAAAAAMAjLPAAAAAAAAAAAMAjLPAAAAAAAAAAAMAjLPAAAAAAA/3+7diwAAAAAMMjfeho7iiMAAACYkHkAAAAAAAAAAGBC5gEAAAAAAAAAgAmZBwAAAAAAAAAAJmQeAAAAAAAAAACYkHkAAAAAAAAAAGBC5gEAAAAAAAAAgAmZBwAAAAAAAAAAJmQeAAAAAAAAAACYCGYPMOpm/dk/AAAAAElFTkSuQmCC\n",
"<Figure size 1440x360 with 10 Axes>"
"image/png": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"import matplotlib.pyplot as plt\n",
"\n",
"# dd.images: list of 8 x 8 images\n",
"# dd.target: label\n",
"\n",
" plt.subplot(1, N, i + 1).set_title(dd.target[i])\n",
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The data is a set of 8 x 8 matrices with values 0 to 15 (black to white). The range 0 to 15 is fixed for this specific data set. Other formats allow e.g. values 0..255 or floating point values in the range 0 to 1."
"data": {
"text/plain": [
"array([0, 1, 2, ..., 8, 9, 8])"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"images[0].shape: (8, 8)\n",
"images[0]:\n",
" [[ 0. 0. 5. 13. 9. 1. 0. 0.]\n",
" [ 0. 0. 13. 15. 10. 15. 5. 0.]\n",
" [ 0. 3. 15. 2. 0. 11. 8. 0.]\n",
" [ 0. 4. 12. 0. 0. 8. 8. 0.]\n",
" [ 0. 5. 8. 0. 0. 9. 8. 0.]\n",
" [ 0. 4. 11. 0. 1. 12. 7. 0.]\n",
" [ 0. 2. 14. 5. 10. 12. 0. 0.]\n",
"print(\"images[0].shape:\", dd.images[0].shape) # dimensions of a first sample array\n",
"print()\n",
"print(\"images[0]:\\n\", dd.images[0]) # first sample array"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To transform such an image to a feature vector we just have to flatten the matrix by concatenating the rows to one single vector of size 64:"
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"image_vector.shape: (64,)\n",
"image_vector: [ 0. 0. 5. 13. 9. 1. 0. 0. 0. 0. 13. 15. 10. 15. 5. 0. 0. 3.\n",
" 15. 2. 0. 11. 8. 0. 0. 4. 12. 0. 0. 8. 8. 0. 0. 5. 8. 0.\n",
" 0. 9. 8. 0. 0. 4. 11. 0. 1. 12. 7. 0. 0. 2. 14. 5. 10. 12.\n",
" 0. 0. 0. 0. 6. 13. 10. 0. 0. 0.]\n"
]
}
],
"source": [
"image_vector = dd.images[0].flatten()\n",
"print(\"image_vector.shape:\", image_vector.shape)\n",
"print(\"image_vector:\", image_vector)"
"### 2nd Example: How to present textual data as feature vectors?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we start a machine learning project for texts, we first have to choose a dictionary (a set of words) for this project. The words in the dictionary are enumerated. The final representation of a text as a feature vector depends on this dictionary.\n",
"\n",
"Such a dictionary can be very large, but for the sake of simplicity we use a very small enumerated dictionary to explain the overall procedure:\n",
"\n",
"\n",
"| Word | Index |\n",
"|----------|-------|\n",
"| like | 0 |\n",
"| dislike | 1 |\n",
"| american | 2 |\n",
"| italian | 3 |\n",
"| beer | 4 |\n",
"| pizza | 5 |\n",
"\n",
"To \"vectorize\" a given text we count the words in the text which also exist in the vocabulary and put the counts at the given `Index`.\n",
"\n",
"E.g. `\"I dislike american pizza, but american beer is nice\"`:\n",
"\n",
"| dislike | 1 | 1 |\n",
"| american | 2 | 2 |\n",
"| italian | 3 | 0 |\n",
"| beer | 4 | 1 |\n",
"| pizza | 5 | 1 |\n",
"\n",
"The respective feature vector is the `Count` column, which is:\n",
"In real case scenarios the dictionary is much bigger, which often results in vectors with only few non-zero entries (so called **sparse vectors**)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Below you find is a short code example to demonstrate how text feature vectors can be created with `scikit-learn`.\n",
"<div class=\"alert alert-block alert-info\">\n",
"<i class=\"fa fa-info-circle\"></i>\n",
"Such vectorization is usually not done manually. Actually there are improved but more complicated procedures which compute multiplicative weights for the vector entries to emphasize informative words such as, e.g., <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html\">\"term frequency-inverse document frequency\" vectorizer</a>.\n",
"</div>"
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[0 1 2 0 1 1]\n"
]
}
],
"source": [
"from sklearn.feature_extraction.text import CountVectorizer\n",
"\n",
"vocabulary = {\n",
" \"like\": 0,\n",
" \"dislike\": 1,\n",
" \"american\": 2,\n",
" \"italian\": 3,\n",
" \"beer\": 4,\n",
" \"pizza\": 5,\n",
"}\n",
"# this how one can create a count vector for a given piece of text:\n",
"vector = vectorizer.fit_transform([\n",
" \"I dislike american pizza. But american beer is nice\"\n",
"]).toarray().flatten()\n",
"print(vector)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Most applications of ML belong to two categories: **supervised** and **unsupervised** learning.\n",
"In supervised learning the data comes with an additional target/label value that we want to predict. Such a problem can be either \n",
"\n",
"- **classification**: we want to predict a categorical value.\n",
"- **regression**: we want to predict numbers in a given range.\n",
"Examples of supervised learning:\n",
"- Classification: predict the class `is_yummy` based on the attributes `alcohol_content`,\t`bitterness`, \t`darkness` and `fruitiness` (a standard two-class problem).\n",
"\n",
"- Classification: predict the digit-shown based on a 8 x 8 pixel image (a multi-class problem).\n",
"\n",
"- Regression: predict temperature based on how long sun was shining in the last 10 minutes.\n",
"\n",
"<table>\n",
" <tr>\n",
" <td><img src=\"./images/classification-svc-2d-poly.png\" width=400px></td>\n",
" <td><img src=\"./images/regression-lin-1d.png\" width=400px></td>\n",
" </tr>\n",
" <tr>\n",
" <td><center>Classification</center></td>\n",
" <td><center>Linear regression</center></td>\n",
" </tr>\n",
"</table>\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Unsupervised learning \n",
"\n",
"In unsupervised learning the training data consists of samples without any corresponding target/label values and the aim is to find structure in data. Some common applications are:\n",
"- Density estimation, novelty detection: find a probability distribution in your data.\n",
"- Dimension reduction (e.g. PCA): find latent structures in your data.\n",
"\n",
"Examples of unsupervised learning:\n",
"- Can we split up our beer data set into sub-groups of similar beers?\n",
"- Can we reduce our data set because groups of features are somehow correlated?\n",
" <td><img src=\"./images/cluster-image.png/\" width=400px></td>\n",
" <td><img src=\"./images/nonlin-pca.png/\" width=400px></td>\n",
" </tr>\n",
" <tr>\n",
" <td><center>Clustering</center></td>\n",
" <td><center>Dimension reduction: detecting 2D structure in 3D data</center></td>\n",
" </tr>\n",
"</table>\n",
"\n",
"\n",
"\n",
"This course will only introduce concepts and methods from **supervised learning**."
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How to apply machine learning in practice?\n",
"\n",
"Application of machine learning in practice consists of several phases:\n",
"\n",
"1. Understand and clean your data\n",
"2. Analyze model for its quality / performance\n",
"2. Apply this model to new incoming data\n",
"\n",
"In practice steps 1. and 2. are iterated for different machine learning algorithms with different configurations until performance is optimal or sufficient. \n",
"\n",
"\n",
"<div class=\"alert alert-block alert-warning\">\n",
"<i class=\"fa fa-warning\"></i> <strong>Garbage in / garbage out</strong>\n",
"\n",
"The principle of \"garbage in, garbage out\" also applies in machine learning.\n",
"\n",
"Cleaning data to remove strong outliers or erroneous entries is crucial in real-world problems and can be the most time-consuming part.\n",
"\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"./images/303yin.jpg\" width=35%/>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our example beer data set reflects the very personal opinion of one of the tutors which beer he likes and which not. To learn a predictive model and to understand influential factors all beers went through some lab analysis to measure alcohol content, bitterness, darkness and fruitiness."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1. Load the data and show the overall structure using `pandas`"
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(225, 5)\n"
]
}
],
"source": [
"import pandas as pd\n",
"\n",
"# read some data\n",
"beer_data = pd.read_csv(\"data/beers.csv\")\n",
"print(beer_data.shape)"
]
},
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>alcohol_content</th>\n",
" <th>bitterness</th>\n",
" <th>darkness</th>\n",
" <th>fruitiness</th>\n",
" <th>is_yummy</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>3.739295</td>\n",
" <td>0.422503</td>\n",
" <td>0.989463</td>\n",
" <td>0.215791</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>4.207849</td>\n",
" <td>0.841668</td>\n",
" <td>0.928626</td>\n",
" <td>0.380420</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>4.709494</td>\n",
" <td>0.322037</td>\n",
" <td>5.374682</td>\n",
" <td>0.145231</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>4.684743</td>\n",
" <td>0.434315</td>\n",
" <td>4.072805</td>\n",
" <td>0.191321</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>4.148710</td>\n",
" <td>0.570586</td>\n",
" <td>1.461568</td>\n",
" <td>0.260218</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" alcohol_content bitterness darkness fruitiness is_yummy\n",
"0 3.739295 0.422503 0.989463 0.215791 0\n",
"1 4.207849 0.841668 0.928626 0.380420 0\n",
"2 4.709494 0.322037 5.374682 0.145231 1\n",
"3 4.684743 0.434315 4.072805 0.191321 1\n",
"4 4.148710 0.570586 1.461568 0.260218 0"
]
},
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# show first 5 rows\n",
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>alcohol_content</th>\n",
" <th>bitterness</th>\n",