02_classification.ipynb

   "source": [
    "### Tuning classifiers in a difficult problem\n",
    "\n",
    "Load the data from `\"data/spiral.csv\"`, plot points and train both Logistic Regression classfier with polynomial features and the Support Vector Classifier `sklearn.svm.SVC` with no preprocessing. Compare the decision surfaces.\n",
    "\n",
    "Try different values of degree of the polynomial features and of the hyperparameter `C` (applicable to both classifiers).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"data/spiral.csv\")\n",
    "\n",
    "features = df.iloc[:, :-1]\n",
    "labels = df.iloc[:, -1]\n",
    "\n",
    "x = features.iloc[:, 0]\n",
    "y = features.iloc[:, 1]\n",
    "color = [[\"steelblue\", \"chocolate\"][l] for l in labels]\n",
    "plt.figure(figsize=(6, 6))\n",
    "plt.scatter(x, y, color=color)\n",
    "\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "clf = LogisticRegression(C=1, max_iter=1000)\n",
    "preproc = PolynomialFeatures(4, include_bias=False)\n",
    "\n",
    "plt.figure(figsize=(6, 6))\n",
    "train_and_plot_decision_surface(\"Logistic regression\", clf, features, labels, preproc=preproc)\n",
    "\n",
    "\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "clf = SVC(C=500)\n",
    "\n",
    "plt.figure(figsize=(6, 6))\n",
    "train_and_plot_decision_surface(\"SVC\", clf, features, labels, preproc=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparison of decision surfaces for different classifiers and datasets\n",
    "\n",
    "Compare decision surfaces for different classifiers listed below for both `\"data/xor.csv\"` and `\"data/circle.csv\"` (circle) datasets. For which classifiers does it help to add polynomial features? How many degrees suffice?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import LinearSVC, SVC\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "# ...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import LinearSVC, SVC\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "def try_dataset(dataset, preproc):\n",
    "    \n",
    "    df = pd.read_csv(dataset)\n",
    "    features = df.iloc[:, :-1]\n",
    "    labels = df.iloc[:, -1]\n",
    "    \n",
    "    plt.figure(figsize=(11, 28))\n",
    "    \n",
    "    clf = LogisticRegression()\n",
    " \n",
    "    plt.subplot(5, 2, 1)\n",
    "    train_and_plot_decision_surface(\"LogisticRegression\", clf, features, labels, preproc=None, N=300)\n",
    "    \n",
    "    plt.subplot(5, 2, 2)\n",
    "    train_and_plot_decision_surface(\"LogisticRegression\", clf, features, labels, preproc=preproc, N=300)\n",
    "\n",
    "    clf = LinearSVC()\n",
    "\n",
    "    plt.subplot(5, 2, 3)\n",
    "    train_and_plot_decision_surface(\"LinearSVC\", clf, features, labels, preproc=None, N=300)\n",
    "    \n",
    "    plt.subplot(5, 2, 4)\n",
    "    train_and_plot_decision_surface(\"LinearSVC\", clf, features, labels, preproc=preproc, N=300)\n",
    "    \n",
    "    \n",
    "    clf = SVC()\n",
    "    plt.subplot(5, 2, 5)\n",
    "    train_and_plot_decision_surface(\"SVC\", clf, features, labels, preproc=None, N=300)\n",
    "    plt.subplot(5, 2, 6)\n",
    "    train_and_plot_decision_surface(\"SVC\", clf, features, labels, preproc=preproc, N=300)\n",
    "\n",
    "    \n",
    "    clf = DecisionTreeClassifier()\n",
    "    plt.subplot(5, 2, 7)\n",
    "    train_and_plot_decision_surface(\"DecisionTreeClassifier\", clf, features, labels, preproc=None, N=300)\n",
    "    plt.subplot(5, 2, 8)\n",
    "    train_and_plot_decision_surface(\"DecisionTreeClassifier\", clf, features, labels, preproc=preproc, N=300)\n",
    "\n",
    "    clf = KNeighborsClassifier()\n",
    "    plt.subplot(5, 2, 9)\n",
    "    train_and_plot_decision_surface(\"KNeighborsClassifier\", clf, features, labels, preproc=None, N=300)\n",
    "    \n",
    "    plt.subplot(5, 2, 10)\n",
    "    train_and_plot_decision_surface(\"KNeighborsClassifier\", clf, features, labels, preproc=preproc, N=300)\n",
    "\n",
    "\n",
    "try_dataset(\"data/xor.csv\", PolynomialFeatures(2, include_bias=False))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "try_dataset(\"data/circle.csv\", PolynomialFeatures(4, include_bias=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## But.. what if there are more than two classes?\n",
    "\n",
    "\n",
    "Previous and the following examples in this script consider two class problems.\n",
    "Before we dig deeper into classification, let's say a few words on how to handle more than two classes.\n",
    "\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\"><p><i class=\"fa fa-warning\"></i>&nbsp;\n",
    "    The general idea for <code>n > 2</code> classes is to build multiple 2-class classifiers and determine a winning class by applying all of them:\n",
    "<ul>\n",
    "    <li>in the <strong>one-vs-rest</strong> (OvR; or <strong>one-vs-all</strong>, OvA) approach build <code>n</code> classifiers for \"label n vs. the rest\";</li>\n",
    "    <br/>\n",
    "    \n",
    "  <li>in the <strong>one-vs-one</strong> (OvO) approach builds  classifiers for `label i vs label j` (in total <code>n x (n - 1) / 2</code> classifiers).</li>\n",
    "</ul>\n",
    "</p></div>\n",
    "\n",
    "For new incoming data then all classifiers (`n` or `n x (n -1) / 2`) are applied and the overall winning class gives the final result.\n",
    "\n",
    "For instance, to classify images of digits:\n",
    "\n",
    "- we could build 10 classifiers `is it 0 or other digit`, `is it 1 or other digit`, etc.\n",
    "  \n",
    "  A new image then would hopefully yield `True` for exactly one of the classifier, in other situations the result is unclear.\n",
    "   \n",
    "   \n",
    "- we could build 45 classifiers `is it 0 or 1`, `is it 0 or 2`, etc.\n",
    "\n",
    "  For a new image we could choose the final outcome based on which class \"wins\" most often.\n",
    "\n",
    "\n",
    "<div class=\"alert alert-block alert-info\"><p><i class=\"fa fa-info-circle\"></i>&nbsp;\n",
    "    In <code>scikit-learn</code> many classifiers support multi-class problems out of the box and also offer functionalities to implement <strong>one-vs-rest</strong> or <strong>one-vs-one</strong> in some cases (cf. <a href=\"https://scikit-learn.org/stable/modules/multiclass.html\"><code>scikit-learn</code> multiclass and multilabel algorithms</a>).\n",
    "</p></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (C) 2019-2022 ETH Zurich, SIS ID"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}