08_e-neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# IGNORE THIS CELL WHICH CUSTOMIZES LAYOUT AND STYLING OF THE NOTEBOOK !\n",
    "from numpy.random import seed\n",
    "\n",
    "seed(42)\n",
    "import tensorflow as tf\n",
    "\n",
    "tf.random.set_seed(42)\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "sns.set(style=\"darkgrid\")\n",
    "mpl.rcParams[\"lines.linewidth\"] = 3\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "%config IPCompleter.greedy=True\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings('ignore')\n",
    "warnings.filterwarnings(\"ignore\", category=FutureWarning)\n",
    "from IPython.core.display import HTML\n",
    "\n",
    "HTML(open(\"custom.html\", \"r\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 8e: Sequence modeling: Natural language processing\n",
    "## What is Natural language processing?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the name suggests, it refers to processing of data such as text and speech. This involves tasks such as:\n",
    "\n",
    "- Automatic document processing\n",
    "- Topic modeling\n",
    "- Language translation\n",
    "- sentiment analysis\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we all know, computers cannot process data in text format. They need numbers. So we need some mechanism to convert our text to numbers.\n",
    "\n",
    "**Important to know libraries:**\n",
    "- [Natural language toolkit](https://www.nltk.org/)\n",
    "- [Gensim](https://radimrehurek.com/gensim/)\n",
    "- [Tomotopy](https://bab2min.github.io/tomotopy/v0.12.3/en/)\n",
    "- [fastext](https://fasttext.cc/)\n",
    "\n",
    "## Text prepocessing\n",
    "\n",
    "### Tokenization\n",
    "\n",
    "Text -> tokens\n",
    "\n",
    "The process of reducing a piece of text to tokens is called tokenization. It is genrally done at a word level but can also be done at other levels such as a sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "nltk.download(\"all\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"Is Monty a python or a group of pythons in a flying circus? What about swimming circuses?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "print(word_tokenize(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lemmatization and Stemming\n",
    "\n",
    "Most of the time we want to also reduce the inflectional forms of the same word. For example, consider a text that has (organization, organizational, organizations)\n",
    "\n",
    "`Stemming`: This is a process of reducing a word to a stem form based on some pre-defined rules. The resulting stem might be a non-sensical word.\n",
    "\n",
    "`Lemmatization`: This is a process of reducing a word to a lemma or the dictionary form of the word. This follows lexicon rules and is much more comprehensive than `stemming`. However, it is also more computationally expensive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import PorterStemmer, WordNetLemmatizer\n",
    "from nltk.tokenize import word_tokenize\n",
    "from prettytable import PrettyTable\n",
    "\n",
    "words = word_tokenize(text)\n",
    "print(\"Tokens \\n\")\n",
    "print(words)\n",
    "\n",
    "stemmer = PorterStemmer()\n",
    "\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "table = PrettyTable([\"Word\", \"Stem\", \"Lemma\"])\n",
    "\n",
    "for w in words:\n",
    "    table.add_row([w, stemmer.stem(w), lemmatizer.lemmatize(w)])\n",
    "\n",
    "print(table)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lemmatizer.lemmatize(\"swimming\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lemmatizer.lemmatize?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lemmatizer.lemmatize(\"swimming\", \"v\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Automatically find POS tag\n",
    "from nltk.corpus import wordnet\n",
    "\n",
    "\n",
    "def get_wordnet_pos(word):\n",
    "    \"\"\"Map POS tag to first character lemmatize() accepts\"\"\"\n",
    "    tag = nltk.pos_tag([word])[0][1][0].upper()\n",
    "    tag_dict = {\n",
    "        \"J\": wordnet.ADJ,\n",
    "        \"N\": wordnet.NOUN,\n",
    "        \"V\": wordnet.VERB,\n",
    "        \"R\": wordnet.ADV,\n",
    "    }\n",
    "\n",
    "    return tag_dict.get(tag, wordnet.NOUN)\n",
    "\n",
    "\n",
    "words = word_tokenize(text)\n",
    "\n",
    "table = PrettyTable([\"Word\", \"Stem\", \"Lemma\"])\n",
    "\n",
    "for w in words:\n",
    "    table.add_row([w, stemmer.stem(w), lemmatizer.lemmatize(w, get_wordnet_pos(w))])\n",
    "\n",
    "print(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other:\n",
    "\n",
    "- Text to lower case\n",
    "- Remove punctuations\n",
    "- Remove stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Text to lower case\n",
    "text = text.lower()\n",
    "print(text)\n",
    "\n",
    "# Remove punctuations\n",
    "import string\n",
    "\n",
    "text = text.translate(str.maketrans(\"\", \"\", string.punctuation))\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove stopwords\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "print(stopwords.words(\"english\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = word_tokenize(text)\n",
    "\n",
    "filtered_text = [w for w in words if not w in set(stopwords.words(\"english\"))]\n",
    "\n",
    "print(filtered_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokens to Vectors\n",
    "\n",
    "Once we have cleaned up our text we have different ways in which we can tokenize them:\n",
    "\n",
    "### Bag-of-Words (BoW)\n",
    "\n",
    "Imagine that all the unique words in our text corpus are put together in one big bag. \n",
    "\n",
    "All or a subset of this bag is then considered as our `vocabulary`.\n",
    "\n",
    "Each unit (document/line/...) in our corpus can now be represented as a vector of length equal to our vocabulary size with each index of the vector representing a word from our `vocabulary`.\n",
    "\n",
    "We count the number of occurences of each word in a unit of text and put this number at the corresponding location in this vector. If the word does not exist in the unit we enter 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's consider each sentence of our example text as a document/unit we want to process\n",
    "import numpy as np\n",
    "\n",
    "text = [\n",
    "    \"Is Monty a python or a group of pythons in a flying circus?\",\n",
    "    \"What about swimming circuses?\",\n",
    "]\n",
    "\n",
    "for index, value in enumerate(text):\n",
    "    text[index] = value.lower().translate(str.maketrans(\"\", \"\", string.punctuation))\n",
    "\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "unique_words = {}\n",
    "\n",
    "bow_text = []\n",
    "\n",
    "for index, value in enumerate(text):\n",
    "    words = word_tokenize(value)\n",
    "    words = [w for w in words if not w in set(stopwords.words(\"english\"))]\n",
    "    words = [lemmatizer.lemmatize(w) for w in words]\n",
    "    print(words)\n",
    "    for token in words:\n",
    "        if token not in unique_words.keys():\n",
    "            unique_words[token] = 1\n",
    "        else:\n",
    "            unique_words[token] += 1\n",
    "    bow_text.append(words)\n",
    "\n",
    "print(unique_words)\n",
    "\n",
    "unique_words = list(unique_words.keys())\n",
    "\n",
    "bow_vectors = np.zeros((len(unique_words), len(text)))\n",
    "\n",
    "for column, value in enumerate(bow_text):\n",
    "    for _, word in enumerate(value):\n",
    "        if word in unique_words:\n",
    "            bow_vectors[unique_words.index(word), column] += 1\n",
    "print(bow_vectors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much better way of doing this is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from string import punctuation\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# CountVectorizer automatically makes the text lowercase\n",
    "\n",
    "text = [\n",
    "    \"Is Monty a python or a group of python in a flying circus?\",\n",
    "    \"What about swimming circuses?\",\n",
    "]\n",
    "\n",
    "\n",
    "class LemmaTokenizer:\n",
    "    def __init__(self):\n",
    "        self.wnl = WordNetLemmatizer()\n",
    "\n",
    "    def __call__(self, doc):\n",
    "        return [self.wnl.lemmatize(t) for t in word_tokenize(doc)]\n",
    "\n",
    "\n",
    "vectorizer = CountVectorizer(\n",
    "    stop_words=list(set(stopwords.words(\"english\")).union(set(punctuation))),\n",
    "    tokenizer=LemmaTokenizer(),\n",
    ")\n",
    "\n",
    "bow_vectors = vectorizer.fit_transform(text)\n",
    "\n",
    "print(f\"The vocabulary of our corpus is: \\n {vectorizer.vocabulary_}\\n\")\n",
    "\n",
    "print(f\"Vectorizer from Scikit learn creates sparse matrices: {type(bow_vectors)} \\n\")\n",
    "\n",
    "print(f\"The created vectors are: {bow_vectors.todense()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Other tokenizers\n",
    "from string import punctuation\n",
    "\n",
    "from nltk.tokenize import TweetTokenizer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "tokenizer = TweetTokenizer()\n",
    "\n",
    "text = [\n",
    "    \"Is Monty a python or a group of python's in a flying circus?\",\n",
    "    \"What about swimming circuses?\",\n",
    "]\n",
    "\n",
    "\n",
    "class LemmaTokenizer:\n",
    "    def __init__(self):\n",
    "        self.wnl = WordNetLemmatizer()\n",
    "\n",
    "    def __call__(self, doc):\n",
    "        return [self.wnl.lemmatize(t) for t in tokenizer.tokenize(doc)]\n",
    "\n",
    "\n",
    "vectorizer = CountVectorizer(\n",
    "    stop_words=list(set(stopwords.words(\"english\")).union(set(punctuation))),\n",
    "    tokenizer=LemmaTokenizer(),\n",
    ")\n",
    "\n",
    "bow_vectors = vectorizer.fit_transform(text)\n",
    "\n",
    "print(f\"The vocabulary of our corpus is: \\n {vectorizer.vocabulary_}\\n\")\n",
    "\n",
    "print(f\"Vectorizer from Scikit learn creates sparse matrices: {type(bow_vectors)} \\n\")\n",
    "\n",
    "print(f\"The created vectors are: {bow_vectors.todense()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Term frequency inverse document frequency (Tf-idf)\n",
    "\n",
    "A numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus.\n",
    "\n",
    "A survey conducted in 2015 showed that 83% of text-based recommender systems in digital libraries use tf–idf(*)\n",
    "\n",
    "*[Research-paper recommender systems : a literature survey](https://link.springer.com/article/10.1007/s00799-015-0156-0)\n",
    "\n",
    "$TF-IDF = TF * IDF$\n",
    "\n",
    "**TF = Term frequency**\n",
    "\n",
    "**IDF = Inverse document/text frequency**\n",
    "\n",
    "$T_{t',d}$ = Number of occurences of a particular term ($t'$) in a document ($d$).\n",
    "\n",
    "$\\sum_{t' \\in d} T_{t',d}$ : Total number of terms in the document\n",
    "\n",
    "$N_T$ = Total number of documents/text samples.\n",
    "\n",
    "$N_{t'}$ = Number of documents/text samples that contain the term $t'$-\n",
    "\n",
    "$TF-IDF = \\dfrac{T_{t',d}}{\\sum_{t' \\in d} T_{t',d}} * \\dfrac{N_T}{N_{t'}}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IMDB dataset\n",
    "\n",
    "Let's have a look at a sample dataset.\n",
    "\n",
    "IMDB dataset comprises of 50,000 movie reviews. Each of them has a label `0` or `1` representing a bad or a good review, respectively.\n",
    "\n",
    "`Note`: This dataset is also contained in tensorflow.keras.datasets, however that data is already preprocessed. Therefore, we import it from tensorflow_datasets instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Explore the IMDB dataset and vectorize the tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "\n",
    "train_data, test_data = tfds.load(\n",
    "    name=\"imdb_reviews\", split=[\"train\", \"test\"], batch_size=-1, as_supervised=True\n",
    ")\n",
    "\n",
    "X_train, y_train = tfds.as_numpy(train_data)\n",
    "X_test, y_test = tfds.as_numpy(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Number of: training samples - {len(y_train)}, test_samples - {len(y_test)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(X_train[:5])\n",
    "print(y_train[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Apply tokenization and vectorization (e.g. CountVectorizer) to the imdb dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a vectorizer e.g. CountVectorizer\n",
    "# Pass maximum features=10000 to the vectorizer to avoid running out of memory\n",
    "\n",
    "\n",
    "# train it on the training set (HINT: one can pass an array of texts)\n",
    "\n",
    "\n",
    "# Look at the resulting vocabulary\n",
    "# vectorizer.vocabulary_\n",
    "\n",
    "\n",
    "# Transform the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [],
   "source": [
    "# Build a 3 layer simple vanilla neural network\n",
    "# Dont forget to add dropout\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "model.compile(optimizer=\"adam\", loss=\"binary_crossentropy\", metrics=[\"accuracy\"])\n",
    "\n",
    "# We need to convert the sparse vector to dense\n",
    "results = model.fit(\n",
    "    train.todense(),\n",
    "    y_train,\n",
    "    epochs=10,\n",
    "    batch_size=512,\n",
    "    validation_data=(test.todense(), y_test),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Solution\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "vectorizer = CountVectorizer(\n",
    "    stop_words=list(set(stopwords.words(\"english\")).union(set(punctuation))),\n",
    "    tokenizer=LemmaTokenizer(),\n",
    "    max_features=20000,\n",
    ")\n",
    "\n",
    "train = vectorizer.fit_transform(X_train)\n",
    "\n",
    "#print(vectorizer.vocabulary_)\n",
    "\n",
    "test = vectorizer.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Build a 3 layer simple vanilla neural network\n",
    "# Dont forget to add dropout\n",
    "\n",
    "from tensorflow.keras.layers import Dense, Dropout\n",
    "from tensorflow.keras.models import Sequential\n",
    "\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(50, activation=\"relu\", input_shape=(test.shape[1],)))\n",
    "# Hidden - Layers\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(30, activation=\"relu\"))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(20, activation=\"relu\"))\n",
    "# Output- Layer\n",
    "model.add(Dense(1, activation=\"sigmoid\"))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer=\"adam\", loss=\"binary_crossentropy\", metrics=[\"accuracy\"])\n",
    "\n",
    "# We need to convert the sparse vector to dense\n",
    "results = model.fit(\n",
    "    train.todense(),\n",
    "    y_train,\n",
    "    epochs=10,\n",
    "    batch_size=512,\n",
    "    validation_data=(test.todense(), y_test),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word embeddings: Featurized representation of words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "<figure>\n",
    "<img src=\"./images/neuralnets/word_embedding.png\" width=\"700\"/>\n",
    "<figcaption>Embedding words in a higher dimensional feature space</figcaption>\n",
    "</figure>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| <div style=\"width:150px\"></div>  | <div style=\"width:150px\"></div> Apple | <div style=\"width:150px\"></div> Orange  | <div style=\"width:150px\"></div> Pants | <div style=\"width:150px\"></div> Tiger |\n",
    "| :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n",
    "| Animal |0.01 |0.015 |0.006 | 0.96 |\n",
    "| Fruit | 0.99 | 0.97 | -0.001 | -0.01 |\n",
    "| Clothing | 0.02 | 0.07 | 0.97 | 0.002 |\n",
    "| FeatureX | - | - | - | - |\n",
    "| FeatureY | - | - | - | - |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some models to compute word embeddings:\n",
    "- Word2Vec\n",
    "- GloVe\n",
    "- fastText\n",
    "- BERT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pretrained embeddings\n",
    "\n",
    "Example:\n",
    "https://fasttext.cc/docs/en/crawl-vectors.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fasttext\n",
    "\n",
    "ft = fasttext.load_model(\"./data/cc.en.100.bin\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = [\"cat\", \"dog\", \"cream\", \"pizza\", \"car\", \"tractor\"]\n",
    "\n",
    "word_vectors = {}\n",
    "for word in words:\n",
    "    word_vectors[word] = ft.get_word_vector(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy import spatial\n",
    "\n",
    "\n",
    "def compute_similarity(a, b):\n",
    "    \"\"\"This function computes cosine similarity between two vectors\"\"\"\n",
    "    return 1 - spatial.distance.cosine(a, b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# similarities = np.zeros([len(words)]*2)\n",
    "similarities = pd.DataFrame(columns=words, index=words)\n",
    "for word_a, vec_a in word_vectors.items():\n",
    "    for word_b, vec_b in word_vectors.items():\n",
    "        similarities.at[word_a, word_b] = compute_similarity(vec_a, vec_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similarities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Recurrent Neural Networks (RNNs)\n",
    "\n",
    "RNNs are used for problems such as time-series data, speech recognition and translation.\n",
    "\n",
    "<center>\n",
    "<figure>\n",
    "<img src=\"./images/neuralnets/RNNs.png\" width=\"700\"/>\n",
    "<figcaption>Recurrent neural network</figcaption>\n",
    "</figure>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are newer variants that overcome some issues with a vanilla RNN:\n",
    "- Gated Recurrent Unit (GRU)\n",
    "- Long Short Term Memory (LSTM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example walkthrough** : https://keras.io/examples/vision/video_classification/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Transformer models\n",
    "\n",
    "Transformers are models based on an encoder-decoder architecture and mainly using the attention.\n",
    "\n",
    "<center>\n",
    "<table>\n",
    "    <tr><td>\n",
    "        <figure>\n",
    "        <img src=\"./images/neuralnets/transformer.png\" width=\"400\"/>\n",
    "        <figcaption>Transformer architecture</figcaption>\n",
    "        </figure>\n",
    "    </td></tr>\n",
    "    <tr><td><center><sub>Source: <a href=\"https://arxiv.org/abs/1706.03762/\">\"Attention is all you need\": https://arxiv.org/abs/1706.03762/</a></sub></center></td></tr>\n",
    "</table>\n",
    "</center>ce"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Putting it all together\n",
    "\n",
    "https://paperswithcode.com/sota/sentiment-analysis-on-imdb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}