starter_notebook_into_English_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Igc5itf-xMGj"
   },
   "source": [
    "# Masakhane - Reverse Machine Translation for African Languages (Using JoeyNMT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ## NB\n",
    ">### - The purpose of this Notebook is to build models that translate African languages(target language) *into* English(source language). This will allow us to in future be able to make translations from one African language to the other. If you'd like to translate *from* English, please use [this](https://github.com/masakhane-io/masakhane-mt/blob/master/starter_notebook.ipynb) starter notebook instead.\n",
    "\n",
    ">### - We call this reverse training because normally we build models that make translations from the source language(English) to the target language. But in this case we are doing the reverse; building models that make translations from the target language to the source(English)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "x4fXCKCf36IK"
   },
   "source": [
    "## Note before beginning:\n",
    "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
    "\n",
    "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
    "\n",
    "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
    "\n",
    "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
    "\n",
    "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to masakhanetranslation@gmail.com\n",
    "\n",
    "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l929HimrxS0a"
   },
   "source": [
    "## Retrieve your data & make a parallel corpus\n",
    "\n",
    "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
    "\n",
    "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "oGRmDELn7Az0",
    "outputId": "807c3318-4a3e-483b-fed0-142c2b9f926a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
     ]
    }
   ],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "Cn3tgQLzUxwn"
   },
   "outputs": [],
   "source": [
    "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
    "# These will also become the suffix's of all vocab and corpus files used throughout\n",
    "import os\n",
    "source_language = \"en\"\n",
    "target_language = \"sw\" \n",
    "lc = False  # If True, lowercase the data.\n",
    "seed = 42  # Random seed for shuffling.\n",
    "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
    "\n",
    "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
    "os.environ[\"tgt\"] = target_language\n",
    "os.environ[\"tag\"] = tag\n",
    "\n",
    "# This will save it to a folder in our gdrive instead!\n",
    "!mkdir -p \"/content/drive/My Drive/masakhane/$tgt-$src-$tag\"\n",
    "os.environ[\"gdrive_path\"] = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (target_language, source_language, tag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "kBSgJHEw7Nvx",
    "outputId": "3bdb0703-874f-4d53-e5b2-7802a937720b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/content/drive/My Drive/masakhane/en-sw-baseline\n"
     ]
    }
   ],
   "source": [
    "!echo $gdrive_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "gA75Fs9ys8Y9",
    "outputId": "e415db4b-66fe-4785-b80f-6e375809c1bf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: opustools-pkg in /usr/local/lib/python3.6/dist-packages (0.0.52)\n"
     ]
    }
   ],
   "source": [
    "# Install opus-tools\n",
    "! pip install opustools-pkg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 208
    },
    "id": "xq-tDZVks7ZD",
    "outputId": "c19ee6c0-e652-486c-a1fd-0c5fd265b204"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/en-sw.xml.gz not found. The following files are available for downloading:\n",
      "\n",
      "   8 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en-sw.xml.gz\n",
      " 263 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en.zip\n",
      "  94 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/sw.zip\n",
      "\n",
      " 365 MB Total size\n",
      "./JW300_latest_xml_en-sw.xml.gz ... 100% of 8 MB\n",
      "./JW300_latest_xml_en.zip ... 100% of 263 MB\n",
      "./JW300_latest_xml_sw.zip ... 100% of 94 MB\n"
     ]
    }
   ],
   "source": [
    "# Downloading our corpus\n",
    "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
    "\n",
    "# extract the corpus file\n",
    "! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 590
    },
    "id": "n48GDRnP8y2G",
    "outputId": "e27c546b-8b90-43c3-9123-738d3473f12a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-10-28 06:26:47--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 277791 (271K) [text/plain]\n",
      "Saving to: ‘test.en-any.en.1’\n",
      "\n",
      "\r",
      "test.en-any.en.1      0%[                    ]       0  --.-KB/s               \r",
      "test.en-any.en.1    100%[===================>] 271.28K  --.-KB/s    in 0.03s   \n",
      "\n",
      "2020-10-28 06:26:47 (9.82 MB/s) - ‘test.en-any.en.1’ saved [277791/277791]\n",
      "\n",
      "--2020-10-28 06:26:47--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-sw.en\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 206336 (202K) [text/plain]\n",
      "Saving to: ‘test.en-sw.en’\n",
      "\n",
      "test.en-sw.en       100%[===================>] 201.50K  --.-KB/s    in 0.03s   \n",
      "\n",
      "2020-10-28 06:26:47 (7.07 MB/s) - ‘test.en-sw.en’ saved [206336/206336]\n",
      "\n",
      "--2020-10-28 06:26:48--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-sw.sw\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 214836 (210K) [text/plain]\n",
      "Saving to: ‘test.en-sw.sw’\n",
      "\n",
      "test.en-sw.sw       100%[===================>] 209.80K  --.-KB/s    in 0.03s   \n",
      "\n",
      "2020-10-28 06:26:48 (7.22 MB/s) - ‘test.en-sw.sw’ saved [214836/214836]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Download the global test set.\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
    "  \n",
    "# And the specific test set for this language pair.\n",
    "os.environ[\"trg\"] = target_language \n",
    "os.environ[\"src\"] = source_language \n",
    "\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
    "! mv test.en-$trg.en test.en\n",
    "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
    "! mv test.en-$trg.$trg test.$trg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "NqDG-CI28y2L",
    "outputId": "20dc514e-b0d4-4c53-c492-705e42fbbe7c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 3571 global test sentences to filter from the training/dev data.\n"
     ]
    }
   ],
   "source": [
    "# Read the test data to filter from train and dev splits.\n",
    "# Store english portion in set for quick filtering checks.\n",
    "en_test_sents = set()\n",
    "filter_test_sents = \"test.en-any.en\"\n",
    "j = 0\n",
    "with open(filter_test_sents) as f:\n",
    "  for line in f:\n",
    "    en_test_sents.add(line.strip())\n",
    "    j += 1\n",
    "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 161
    },
    "id": "3CNdwLBCfSIl",
    "outputId": "fc8f80b7-2f12-4f4b-ca94-3eb0258de6ee"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded data and skipped 6478/979526 lines since contained in test set.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source_sentence</th>\n",
       "      <th>target_sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>“ Look !</td>\n",
       "      <td>“ Tazama !</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I Am Making All Things New ”</td>\n",
       "      <td>Mimi Ninafanya Vitu Vyote Kuwa Vipya ”</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The above is a promise from God that has been ...</td>\n",
       "      <td>Iliyopo juu ni ahadi itokayo kwa Mungu ambayo ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     source_sentence                                    target_sentence\n",
       "0                                           “ Look !                                         “ Tazama !\n",
       "1                       I Am Making All Things New ”             Mimi Ninafanya Vitu Vyote Kuwa Vipya ”\n",
       "2  The above is a promise from God that has been ...  Iliyopo juu ni ahadi itokayo kwa Mungu ambayo ..."
      ]
     },
     "execution_count": 10,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# TMX file to dataframe\n",
    "source_file = 'jw300.' + source_language\n",
    "target_file = 'jw300.' + target_language\n",
    "\n",
    "source = []\n",
    "target = []\n",
    "skip_lines = []  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
    "with open(source_file) as f:\n",
    "    for i, line in enumerate(f):\n",
    "        # Skip sentences that are contained in the test set.\n",
    "        if line.strip() not in en_test_sents:\n",
    "            source.append(line.strip())\n",
    "        else:\n",
    "            skip_lines.append(i)             \n",
    "with open(target_file) as f:\n",
    "    for j, line in enumerate(f):\n",
    "        # Only add to corpus if corresponding source was not skipped.\n",
    "        if j not in skip_lines:\n",
    "            target.append(line.strip())\n",
    "    \n",
    "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
    "    \n",
    "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
    "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
    "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YkuK3B4p2AkN"
   },
   "source": [
    "## Pre-processing and export\n",
    "\n",
    "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
    "\n",
    "In addition we will split our data into dev/test/train and export to the filesystem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 191
    },
    "id": "M_2ouEOH1_1q",
    "outputId": "b30fcc33-5a31-46fe-a224-c7d9c4e5bcb5"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  import sys\n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:8: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "# drop duplicate translations\n",
    "df_pp = df.drop_duplicates()\n",
    "\n",
    "# drop conflicting translations\n",
    "# (this is optional and something that you might want to comment out \n",
    "# depending on the size of your corpus)\n",
    "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
    "df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
    "\n",
    "# Shuffle the data to remove bias in dev set selection.\n",
    "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 278
    },
    "id": "Z_1BwAApEtMk",
    "outputId": "d99a471a-6fb6-4941-950c-7c90f2a0789d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting fuzzywuzzy\n",
      "  Downloading https://files.pythonhosted.org/packages/43/ff/74f23998ad2f93b945c0309f825be92e04e0348e062026998b5eefef4c33/fuzzywuzzy-0.18.0-py2.py3-none-any.whl\n",
      "Installing collected packages: fuzzywuzzy\n",
      "Successfully installed fuzzywuzzy-0.18.0\n",
      "Collecting python-Levenshtein\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/42/a9/d1785c85ebf9b7dfacd08938dd028209c34a0ea3b1bcdb895208bd40a67d/python-Levenshtein-0.12.0.tar.gz (48kB)\n",
      "\u001b[K     |████████████████████████████████| 51kB 2.8MB/s \n",
      "\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from python-Levenshtein) (50.3.0)\n",
      "Building wheels for collected packages: python-Levenshtein\n",
      "  Building wheel for python-Levenshtein (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for python-Levenshtein: filename=python_Levenshtein-0.12.0-cp36-cp36m-linux_x86_64.whl size=144796 sha256=a672b4c48e25c303e96476a57daff44ea0e5e6d99efc41a4a1c494ca2eefcd39\n",
      "  Stored in directory: /root/.cache/pip/wheels/de/c2/93/660fd5f7559049268ad2dc6d81c4e39e9e36518766eaf7e342\n",
      "Successfully built python-Levenshtein\n",
      "Installing collected packages: python-Levenshtein\n",
      "Successfully installed python-Levenshtein-0.12.0\n"
     ]
    }
   ],
   "source": [
    "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n",
    "# test and training sets.\n",
    "! pip install fuzzywuzzy\n",
    "! pip install python-Levenshtein\n",
    "import time\n",
    "from fuzzywuzzy import process\n",
    "import numpy as np\n",
    "from os import cpu_count\n",
    "from functools import partial\n",
    "from multiprocessing import Pool\n",
    "\n",
    "\n",
    "# reset the index of the training set after previous filtering\n",
    "df_pp.reset_index(drop=False, inplace=True)\n",
    "\n",
    "# Remove samples from the training data set if they \"almost overlap\" with the\n",
    "# samples in the test set.\n",
    "\n",
    "# Filtering function. Adjust pad to narrow down the candidate matches to\n",
    "# within a certain length of characters of the given sample.\n",
    "def fuzzfilter(sample, candidates, pad):\n",
    "  candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n",
    "  if len(candidates) > 0:\n",
    "    return process.extractOne(sample, candidates)[1]\n",
    "  else:\n",
    "    return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "92EsgTaY3B4H"
   },
   "outputs": [],
   "source": [
    "# start_time = time.time()\n",
    "# ### iterating over pandas dataframe rows is not recomended, let use multi processing to apply the function\n",
    "\n",
    "# with Pool(cpu_count()-1) as pool:\n",
    "#     scores = pool.map(partial(fuzzfilter, candidates=list(en_test_sents), pad=5), df_pp['source_sentence'])\n",
    "# hours, rem = divmod(time.time() - start_time, 3600)\n",
    "# minutes, seconds = divmod(rem, 60)\n",
    "# print(\"done in {}h:{}min:{}seconds\".format(hours, minutes, seconds))\n",
    "\n",
    "# # Filter out \"almost overlapping samples\"\n",
    "# df_pp = df_pp.assign(scores=scores)\n",
    "# df_pp = df_pp[df_pp['scores'] < 95]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 836
    },
    "id": "hxxBOCA-xXhy",
    "outputId": "e009b507-337c-4c33-e07a-172790f9755f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==> train.en <==\n",
      "Besides , being single gives one the opportunity to do so many great things !\n",
      "A Logical Conclusion\n",
      "I still feel the pain of the loss , but thanking Jehovah for having had a good marriage and for the privilege of serving him with someone who deeply loved him has improved my outlook . ”\n",
      "If we do his will , he blesses and supports us in this life and offers us the certain hope of an even better life to come .\n",
      "We are determined to continue providing a rich supply of enlightening and appealing information ​ — both in print and online — ​ to benefit our many readers who respect the Bible and who want to know what it really teaches .\n",
      "Well , in the past decade , the number of active Witnesses increased from slightly fewer than 3,800,000 to almost 6,000,000 .\n",
      "Actually , according to the Bible , there is no spiritual part of a person that survives the death of the body .\n",
      "The enemy may aim a blow by attacking Bible truths that are fundamental to your faith .\n",
      "When Saul returned to Damascus , “ the Jews took counsel together to do away with him . ”\n",
      "To settle the issue of Godship , Elijah now proposes a contest .\n",
      "\n",
      "==> train.sw <==\n",
      "Isitoshe , kwa kuwa sijaanzisha urafiki kama huo nina nafasi ya kufanya mambo mengi ya pekee !\n",
      "Mkataa Unaopatana na Akili\n",
      "Bado nina huzuni ya kufiwa , lakini mtazamo wangu umebadilika kwa sababu ya kumshukuru Yehova kwa kuwa nilikuwa na ndoa yenye furaha na pendeleo la kumtumikia pamoja na mtu aliyempenda sana . ”\n",
      "Tukifanya mapenzi yake , yeye hutubariki na hututegemeza katika uhai huu na hututolea lile tumaini hakika la uhai ulio mzuri hata zaidi utakaokuja .\n",
      "Tumeazimia kuendelea kutoa habari nyingi zenye kuelimisha na zinazovutia — zinazochapishwa na pia zinazopatikana kwenye Intaneti — ili kuwafaidi wasomaji wetu wengi wanaoiheshimu Biblia na wanaotaka kujua inafundisha nini hasa .\n",
      "Katika miaka kumi iliyopita , idadi ya Mashahidi watendaji iliongezeka kutoka 3,800,000 hivi hadi karibu 6,000,000 .\n",
      "Kwa kweli , kulingana na Biblia , hakuna sehemu yoyote ya kiroho ya mtu ambayo huendelea kuwa hai baada ya kufa kwa mwili .\n",
      "Huenda adui akaamua kushambulia kweli za Biblia ambazo ni muhimu kwa imani yako .\n",
      "Sauli aliporudi Damasko , “ Wayahudi walikata shauri pamoja kumwangamiza . ”\n",
      "Ili kusuluhisha hilo suala la Uungu , Eliya apendekeza shindano .\n",
      "==> dev.en <==\n",
      "There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "Lots of hugs and kisses .\n",
      "When Harald was arrested , his wife , Elsa , was still breast - feeding their ten - month - old baby girl .\n",
      "Still , from his painful experience , he learned some valuable lessons .\n",
      "How did Pharaoh show haughtiness , and with what result ?\n",
      "What seems to trigger such outbursts ?\n",
      "In a research project , Poole heard a sound that was different from the normal elephant calls .\n",
      "Misery will give way to happiness when God blesses you as he did Job .\n",
      "\n",
      "==> dev.sw <==\n",
      "Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "Kukumbatia kwingi na busu nyingi .\n",
      "Wakati Harald alipokamatwa , mke wake , Elsa , alikuwa bado anamnyonyesha binti yao mwenye umri wa miezi kumi .\n",
      "Ingawa hakupata utajiri , kisa chake chenye kuhuzunisha kilimfunza mambo muhimu .\n",
      "Farao alionyeshaje majivuno , na matokeo yalikuwa nini ?\n",
      "Ni nini huchochea milipuko hiyo ya hasira ?\n",
      "Poole alipokuwa akifanya uchunguzi fulani , alisikia mlio uliokuwa tofauti na milio ya kawaida ya tembo .\n",
      "Furaha itachukua mahali pa huzuni Mungu anapokubariki kama alivyombariki Ayubu .\n"
     ]
    }
   ],
   "source": [
    "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
    "# We use 1000 dev test and the given test set.\n",
    "import csv\n",
    "\n",
    "# Do the split between dev/train and create parallel corpora\n",
    "num_dev_patterns = 1000\n",
    "\n",
    "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
    "if lc:  # Julia: making lowercasing optional\n",
    "    df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
    "    df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
    "\n",
    "# Julia: test sets are already generated\n",
    "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n",
    "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n",
    "\n",
    "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
    "  for index, row in stripped.iterrows():\n",
    "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
    "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
    "    \n",
    "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
    "  for index, row in dev.iterrows():\n",
    "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
    "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
    "\n",
    "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
    "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
    "\n",
    "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
    "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
    "\n",
    "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
    "! head train.*\n",
    "! head dev.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "epeCydmCyS8X"
   },
   "source": [
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "## Installation of JoeyNMT\n",
    "\n",
    "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "iBRMm4kMxZ8L",
    "outputId": "97a36759-3f50-4498-f54e-9d36cb35c9ab"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'joeynmt'...\n",
      "remote: Enumerating objects: 2546, done.\u001b[K\n",
      "remote: Total 2546 (delta 0), reused 0 (delta 0), pack-reused 2546\u001b[K\n",
      "Receiving objects: 100% (2546/2546), 2.74 MiB | 30.50 MiB/s, done.\n",
      "Resolving deltas: 100% (1731/1731), done.\n",
      "Processing /content/joeynmt\n",
      "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.3) (0.16.0)\n",
      "Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.3) (7.0.0)\n",
      "Requirement already satisfied: numpy<1.19.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.3) (1.18.5)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.3) (50.3.0)\n",
      "Requirement already satisfied: torch>=1.1 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.3) (1.6.0+cu101)\n",
      "Requirement already satisfied: tensorboard>=1.15 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.3) (2.3.0)\n",
      "Requirement already satisfied: torchtext<0.8.0 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.3) (0.3.1)\n",
      "Collecting sacrebleu>=1.3.6\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a3/c4/8e948f601a4f9609e8b2b58f31966cb13cf17b940b82aa3e767f01c42c52/sacrebleu-1.4.14-py3-none-any.whl (64kB)\n",
      "\u001b[K     |████████████████████████████████| 71kB 3.5MB/s \n",
      "\u001b[?25hCollecting subword-nmt\n",
      "  Downloading https://files.pythonhosted.org/packages/74/60/6600a7bc09e7ab38bc53a48a20d8cae49b837f93f5842a41fe513a694912/subword_nmt-0.3.7-py2.py3-none-any.whl\n",
      "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.3) (3.2.2)\n",
      "Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.3) (0.11.0)\n",
      "Collecting pyyaml>=5.1\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz (269kB)\n",
      "\u001b[K     |████████████████████████████████| 276kB 15.2MB/s \n",
      "\u001b[?25hCollecting pylint\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/fb/13/519c1264a134beab2be4bac8dd3e64948980a5ca7833b31cf0255b21f20a/pylint-2.6.0-py3-none-any.whl (325kB)\n",
      "\u001b[K     |████████████████████████████████| 327kB 28.8MB/s \n",
      "\u001b[?25hCollecting six==1.12\n",
      "  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl\n",
      "Collecting wrapt==1.11.1\n",
      "  Downloading https://files.pythonhosted.org/packages/67/b2/0f71ca90b0ade7fad27e3d20327c996c6252a2ffe88f50a95bba7434eda9/wrapt-1.11.1.tar.gz\n",
      "Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.15->joeynmt==0.0.3) (3.12.4)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.15->joeynmt==0.0.3) (1.0.1)\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.15->joeynmt==0.0.3) (2.23.0)\n",
      "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.15->joeynmt==0.0.3) (0.10.0)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.15->joeynmt==0.0.3) (1.7.0)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.15->joeynmt==0.0.3) (0.4.1)\n",
      "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.15->joeynmt==0.0.3) (0.35.1)\n",
      "Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.15->joeynmt==0.0.3) (1.33.1)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.15->joeynmt==0.0.3) (3.3.2)\n",
      "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.15->joeynmt==0.0.3) (1.17.2)\n",
      "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext<0.8.0->joeynmt==0.0.3) (4.41.1)\n",
      "Collecting portalocker\n",
      "  Downloading https://files.pythonhosted.org/packages/89/a6/3814b7107e0788040870e8825eebf214d72166adf656ba7d4bf14759a06a/portalocker-2.0.0-py2.py3-none-any.whl\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.3) (1.2.0)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.3) (0.10.0)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.3) (2.8.1)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.3) (2.4.7)\n",
      "Requirement already satisfied: pandas>=0.23 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.3) (1.1.3)\n",
      "Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.3) (1.4.1)\n",
      "Collecting mccabe<0.7,>=0.6\n",
      "  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl\n",
      "Collecting isort<6,>=4.2.5\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ee/e3/75cacbe65a236934860880547fc612e8e3856b5cc3844a8beddae05e7b60/isort-5.6.4-py3-none-any.whl (98kB)\n",
      "\u001b[K     |████████████████████████████████| 102kB 11.0MB/s \n",
      "\u001b[?25hCollecting astroid<=2.5,>=2.4.0\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/24/a8/5133f51967fb21e46ee50831c3f5dda49e976b7f915408d670b1603d41d6/astroid-2.4.2-py3-none-any.whl (213kB)\n",
      "\u001b[K     |████████████████████████████████| 215kB 23.4MB/s \n",
      "\u001b[?25hRequirement already satisfied: toml>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from pylint->joeynmt==0.0.3) (0.10.1)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard>=1.15->joeynmt==0.0.3) (2.10)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard>=1.15->joeynmt==0.0.3) (1.24.3)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard>=1.15->joeynmt==0.0.3) (2020.6.20)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard>=1.15->joeynmt==0.0.3) (3.0.4)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=1.15->joeynmt==0.0.3) (1.3.0)\n",
      "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard>=1.15->joeynmt==0.0.3) (2.0.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.15->joeynmt==0.0.3) (4.6)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.15->joeynmt==0.0.3) (0.2.8)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.15->joeynmt==0.0.3) (4.1.1)\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.23->seaborn->joeynmt==0.0.3) (2018.9)\n",
      "Collecting typed-ast<1.5,>=1.4.0; implementation_name == \"cpython\" and python_version < \"3.8\"\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/90/ed/5459080d95eb87a02fe860d447197be63b6e2b5e9ff73c2b0a85622994f4/typed_ast-1.4.1-cp36-cp36m-manylinux1_x86_64.whl (737kB)\n",
      "\u001b[K     |████████████████████████████████| 747kB 29.0MB/s \n",
      "\u001b[?25hCollecting lazy-object-proxy==1.4.*\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0b/dd/b1e3407e9e6913cf178e506cd0dee818e58694d9a5cd1984e3f6a8b9a10f/lazy_object_proxy-1.4.3-cp36-cp36m-manylinux1_x86_64.whl (55kB)\n",
      "\u001b[K     |████████████████████████████████| 61kB 9.7MB/s \n",
      "\u001b[?25hRequirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=1.15->joeynmt==0.0.3) (3.1.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard>=1.15->joeynmt==0.0.3) (3.3.1)\n",
      "Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from rsa<5,>=3.1.4; python_version >= \"3\"->google-auth<2,>=1.6.3->tensorboard>=1.15->joeynmt==0.0.3) (0.4.8)\n",
      "Building wheels for collected packages: joeynmt, pyyaml, wrapt\n",
      "  Building wheel for joeynmt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for joeynmt: filename=joeynmt-0.0.3-cp36-none-any.whl size=77279 sha256=250f6b09f2bf7e40185d94579f80cf1963bde82939acd6255f57d60064c5bccb\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-6u2rlugr/wheels/db/01/db/751cc9f3e7f6faec127c43644ba250a3ea7ad200594aeda70a\n",
      "  Building wheel for pyyaml (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for pyyaml: filename=PyYAML-5.3.1-cp36-cp36m-linux_x86_64.whl size=44619 sha256=3387cdafd8840cac107045e97ae09e7253f4ddadcad6647793ad0976658819bb\n",
      "  Stored in directory: /root/.cache/pip/wheels/a7/c1/ea/cf5bd31012e735dc1dfea3131a2d5eae7978b251083d6247bd\n",
      "  Building wheel for wrapt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for wrapt: filename=wrapt-1.11.1-cp36-cp36m-linux_x86_64.whl size=67437 sha256=8d9620f4507ea63d070b630d32c8876e4965a603626d95f415d8071d51f26369\n",
      "  Stored in directory: /root/.cache/pip/wheels/89/67/41/63cbf0f6ac0a6156588b9587be4db5565f8c6d8ccef98202fc\n",
      "Successfully built joeynmt pyyaml wrapt\n",
      "\u001b[31mERROR: nbclient 0.5.1 has requirement jupyter-client>=6.1.5, but you'll have jupyter-client 5.3.5 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-colab 1.0.0 has requirement six~=1.15.0, but you'll have six 1.12.0 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.\u001b[0m\n",
      "Installing collected packages: portalocker, sacrebleu, subword-nmt, pyyaml, mccabe, isort, wrapt, six, typed-ast, lazy-object-proxy, astroid, pylint, joeynmt\n",
      "  Found existing installation: PyYAML 3.13\n",
      "    Uninstalling PyYAML-3.13:\n",
      "      Successfully uninstalled PyYAML-3.13\n",
      "  Found existing installation: wrapt 1.12.1\n",
      "    Uninstalling wrapt-1.12.1:\n",
      "      Successfully uninstalled wrapt-1.12.1\n",
      "  Found existing installation: six 1.15.0\n",
      "    Uninstalling six-1.15.0:\n",
      "      Successfully uninstalled six-1.15.0\n",
      "Successfully installed astroid-2.4.2 isort-5.6.4 joeynmt-0.0.3 lazy-object-proxy-1.4.3 mccabe-0.6.1 portalocker-2.0.0 pylint-2.6.0 pyyaml-5.3.1 sacrebleu-1.4.14 six-1.12.0 subword-nmt-0.3.7 typed-ast-1.4.1 wrapt-1.11.1\n"
     ]
    }
   ],
   "source": [
    "# Install JoeyNMT\n",
    "! git clone https://github.com/joeynmt/joeynmt.git\n",
    "! cd joeynmt; pip3 install .\n",
    "# Install Pytorch with GPU support v1.7.1.\n",
    "! pip install torch==1.8.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AaE77Tcppex9"
   },
   "source": [
    "# Preprocessing the Data into Subword BPE Tokens\n",
    "\n",
    "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
    "\n",
    "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
    "\n",
    "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 416
    },
    "id": "H-TyjtmXB1mL",
    "outputId": "9867ad89-e5ec-4007-b952-d73a158ff5cf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bpe.codes.4000\tdev.en\t     test.bpe.sw     test.en-any.en.1  train.bpe.sw\n",
      "dev.bpe.en\tdev.sw\t     test.en\t     test.sw\t       train.en\n",
      "dev.bpe.sw\ttest.bpe.en  test.en-any.en  train.bpe.en      train.sw\n",
      "bpe.codes.4000\tdev.en\t     test.bpe.sw     test.en-any.en.1  train.bpe.sw\n",
      "dev.bpe.en\tdev.sw\t     test.en\t     test.sw\t       train.en\n",
      "dev.bpe.sw\ttest.bpe.en  test.en-any.en  train.bpe.en      train.sw\n",
      "BPE Swahili Sentences\n",
      "N@@ g@@ ao kubwa ya imani ( T@@ azama f@@ ungu la 12 hadi 14 )\n",
      "K@@ of@@ ia ya chu@@ ma ya wo@@ ko@@ vu ( T@@ azama f@@ ungu la 15 hadi 18 )\n",
      "N@@ im@@ et@@ ambua kwamba watu hu@@ it@@ ikia vizuri wanapo@@ ona uki@@ zungum@@ zia habari za Biblia kwa sha@@ uku na unapo@@ fanya yote una@@ yo@@ weza kuwasaidia . ”\n",
      "Up@@ anga wa roho ( T@@ azama f@@ ungu la 19 na 20 )\n",
      "Kwa msaada wa Yehova tunaweza kus@@ im@@ ama im@@ ara na kum@@ p@@ inga !\n",
      "Combined BPE Vocab\n",
      "с@@\n",
      "ḥ\n",
      "т@@\n",
      "и\n",
      "х@@\n",
      "м\n",
      "і@@\n",
      "д@@\n",
      "і\n",
      "➊\n"
     ]
    }
   ],
   "source": [
    "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
    "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
    "\n",
    "# Do subword NMT\n",
    "from os import path\n",
    "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
    "os.environ[\"tgt\"] = target_language\n",
    "\n",
    "# Learn BPEs on the training data.\n",
    "os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\",target_language + source_language ) # Herman! \n",
    "! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 4000 -o bpe.codes.4000 --write-vocabulary vocab.$src vocab.$tgt\n",
    "\n",
    "# Apply BPE splits to the development and test data.\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n",
    "\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n",
    "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt\n",
    "\n",
    "# Create directory, move everyone we care about to the correct location\n",
    "! mkdir -p $data_path\n",
    "! cp train.* $data_path\n",
    "! cp test.* $data_path\n",
    "! cp dev.* $data_path\n",
    "! cp bpe.codes.4000 $data_path\n",
    "! ls $data_path\n",
    "\n",
    "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
    "! cp train.* \"$gdrive_path\"\n",
    "! cp test.* \"$gdrive_path\"\n",
    "! cp dev.* \"$gdrive_path\"\n",
    "! cp bpe.codes.4000 \"$gdrive_path\"\n",
    "! ls \"$gdrive_path\"\n",
    "\n",
    "# Create that vocab using build_vocab\n",
    "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
    "! joeynmt/scripts/build_vocab.py joeynmt/data/$tgt$src/train.bpe.$src joeynmt/data/$tgt$src/train.bpe.$tgt --output_path joeynmt/data/$tgt$src/vocab.txt\n",
    "\n",
    "# Some output\n",
    "! echo \"BPE Swahili Sentences\"\n",
    "! tail -n 5 test.bpe.$tgt\n",
    "! echo \"Combined BPE Vocab\"\n",
    "! tail -n 10 joeynmt/data/$tgt$src/vocab.txt  # Herman"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 69
    },
    "id": "IlMitUHR8Qy-",
    "outputId": "1656a274-cddb-47a2-bf71-6f64c5ef8a24"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bpe.codes.4000\tdev.en\t     test.bpe.sw     test.en-any.en.1  train.bpe.sw\n",
      "dev.bpe.en\tdev.sw\t     test.en\t     test.sw\t       train.en\n",
      "dev.bpe.sw\ttest.bpe.en  test.en-any.en  train.bpe.en      train.sw\n"
     ]
    }
   ],
   "source": [
    "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
    "! cp train.* \"$gdrive_path\"\n",
    "! cp test.* \"$gdrive_path\"\n",
    "! cp dev.* \"$gdrive_path\"\n",
    "! cp bpe.codes.4000 \"$gdrive_path\"\n",
    "! ls \"$gdrive_path\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ixmzi60WsUZ8"
   },
   "source": [
    "# Creating the JoeyNMT Config\n",
    "\n",
    "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
    "\n",
    "- We used Transformer architecture \n",
    "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
    "\n",
    "Things worth playing with:\n",
    "- The batch size (also recommended to change for low-resourced languages)\n",
    "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
    "- The decoder options (beam_size, alpha)\n",
    "- Evaluation metrics (BLEU versus Crhf4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "h8TMgv1p3L1z"
   },
   "outputs": [],
   "source": [
    "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
    "# (You can of course play with all the parameters if you'd like!)\n",
    "\n",
    "name = '%s%s' % (target_language, source_language)\n",
    "# gdrive_path = os.environ[\"gdrive_path\"]\n",
    "\n",
    "# Create the config\n",
    "config = \"\"\"\n",
    "name: \"{target_language}{source_language}_reverse_transformer\"\n",
    "\n",
    "data:\n",
    "    src: \"{target_language}\"\n",
    "    trg: \"{source_language}\"\n",
    "    train: \"data/{name}/train.bpe\"\n",
    "    dev:   \"data/{name}/dev.bpe\"\n",
    "    test:  \"data/{name}/test.bpe\"\n",
    "    level: \"bpe\"\n",
    "    lowercase: False\n",
    "    max_sent_length: 100\n",
    "    src_vocab: \"data/{name}/vocab.txt\"\n",
    "    trg_vocab: \"data/{name}/vocab.txt\"\n",
    "\n",
    "testing:\n",
    "    beam_size: 5\n",
    "    alpha: 1.0\n",
    "\n",
    "training:\n",
    "    #load_model: \"{gdrive_path}/models/{name}_transformer/1.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
    "    random_seed: 42\n",
    "    optimizer: \"adam\"\n",
    "    normalization: \"tokens\"\n",
    "    adam_betas: [0.9, 0.999] \n",
    "    scheduling: \"plateau\"           # TODO: try switching from plateau to Noam scheduling\n",
    "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
    "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
    "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
    "    decrease_factor: 0.7\n",
    "    loss: \"crossentropy\"\n",
    "    learning_rate: 0.0003\n",
    "    learning_rate_min: 0.00000001\n",
    "    weight_decay: 0.0\n",
    "    label_smoothing: 0.1\n",
    "    batch_size: 4096\n",
    "    batch_type: \"token\"\n",
    "    eval_batch_size: 3600\n",
    "    eval_batch_type: \"token\"\n",
    "    batch_multiplier: 1\n",
    "    early_stopping_metric: \"ppl\"\n",
    "    epochs: 5                  # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
    "    validation_freq: 1000          # TODO: Set to at least once per epoch.\n",
    "    logging_freq: 100\n",
    "    eval_metric: \"bleu\"\n",
    "    model_dir: \"models/{name}_reverse_transformer\"\n",
    "    overwrite: True              # TODO: Set to True if you want to overwrite possibly existing models. \n",
    "    shuffle: True\n",
    "    use_cuda: True\n",
    "    max_output_length: 100\n",
    "    print_valid_sents: [0, 1, 2, 3]\n",
    "    keep_last_ckpts: 3\n",
    "\n",
    "model:\n",
    "    initializer: \"xavier\"\n",
    "    bias_initializer: \"zeros\"\n",
    "    init_gain: 1.0\n",
    "    embed_initializer: \"xavier\"\n",
    "    embed_init_gain: 1.0\n",
    "    tied_embeddings: True\n",
    "    tied_softmax: True\n",
    "    encoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 4             # TODO: Increase to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 256   # TODO: Increase to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
    "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "    decoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 4              # TODO: Increase to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 256    # TODO: Increase to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
    "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "\"\"\".format(name=name, gdrive_path=os.environ[\"gdrive_path\"], source_language=source_language, target_language=target_language)\n",
    "with open(\"joeynmt/configs/transformer_reverse_{name}.yaml\".format(name=name),'w') as f:\n",
    "    f.write(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oEzoJtV2MIpt"
   },
   "source": [
    "# Train the Model\n",
    "\n",
    "This single line of joeynmt runs the training using the config we made above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "WzbNYNdjLgNb",
    "outputId": "cc18bde5-6226-4769-de84-5e5d27a6c5a8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-10-28 08:03:31,578 Hello! This is Joey-NMT.\n",
      "2020-10-28 08:03:31.709513: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "2020-10-28 08:03:32,785 Total params: 12219392\n",
      "2020-10-28 08:03:32,786 Trainable parameters: ['decoder.layer_norm.bias', 'decoder.layer_norm.weight', 'decoder.layers.0.dec_layer_norm.bias', 'decoder.layers.0.dec_layer_norm.weight', 'decoder.layers.0.feed_forward.layer_norm.bias', 'decoder.layers.0.feed_forward.layer_norm.weight', 'decoder.layers.0.feed_forward.pwff_layer.0.bias', 'decoder.layers.0.feed_forward.pwff_layer.0.weight', 'decoder.layers.0.feed_forward.pwff_layer.3.bias', 'decoder.layers.0.feed_forward.pwff_layer.3.weight', 'decoder.layers.0.src_trg_att.k_layer.bias', 'decoder.layers.0.src_trg_att.k_layer.weight', 'decoder.layers.0.src_trg_att.output_layer.bias', 'decoder.layers.0.src_trg_att.output_layer.weight', 'decoder.layers.0.src_trg_att.q_layer.bias', 'decoder.layers.0.src_trg_att.q_layer.weight', 'decoder.layers.0.src_trg_att.v_layer.bias', 'decoder.layers.0.src_trg_att.v_layer.weight', 'decoder.layers.0.trg_trg_att.k_layer.bias', 'decoder.layers.0.trg_trg_att.k_layer.weight', 'decoder.layers.0.trg_trg_att.output_layer.bias', 'decoder.layers.0.trg_trg_att.output_layer.weight', 'decoder.layers.0.trg_trg_att.q_layer.bias', 'decoder.layers.0.trg_trg_att.q_layer.weight', 'decoder.layers.0.trg_trg_att.v_layer.bias', 'decoder.layers.0.trg_trg_att.v_layer.weight', 'decoder.layers.0.x_layer_norm.bias', 'decoder.layers.0.x_layer_norm.weight', 'decoder.layers.1.dec_layer_norm.bias', 'decoder.layers.1.dec_layer_norm.weight', 'decoder.layers.1.feed_forward.layer_norm.bias', 'decoder.layers.1.feed_forward.layer_norm.weight', 'decoder.layers.1.feed_forward.pwff_layer.0.bias', 'decoder.layers.1.feed_forward.pwff_layer.0.weight', 'decoder.layers.1.feed_forward.pwff_layer.3.bias', 'decoder.layers.1.feed_forward.pwff_layer.3.weight', 'decoder.layers.1.src_trg_att.k_layer.bias', 'decoder.layers.1.src_trg_att.k_layer.weight', 'decoder.layers.1.src_trg_att.output_layer.bias', 'decoder.layers.1.src_trg_att.output_layer.weight', 'decoder.layers.1.src_trg_att.q_layer.bias', 'decoder.layers.1.src_trg_att.q_layer.weight', 'decoder.layers.1.src_trg_att.v_layer.bias', 'decoder.layers.1.src_trg_att.v_layer.weight', 'decoder.layers.1.trg_trg_att.k_layer.bias', 'decoder.layers.1.trg_trg_att.k_layer.weight', 'decoder.layers.1.trg_trg_att.output_layer.bias', 'decoder.layers.1.trg_trg_att.output_layer.weight', 'decoder.layers.1.trg_trg_att.q_layer.bias', 'decoder.layers.1.trg_trg_att.q_layer.weight', 'decoder.layers.1.trg_trg_att.v_layer.bias', 'decoder.layers.1.trg_trg_att.v_layer.weight', 'decoder.layers.1.x_layer_norm.bias', 'decoder.layers.1.x_layer_norm.weight', 'decoder.layers.2.dec_layer_norm.bias', 'decoder.layers.2.dec_layer_norm.weight', 'decoder.layers.2.feed_forward.layer_norm.bias', 'decoder.layers.2.feed_forward.layer_norm.weight', 'decoder.layers.2.feed_forward.pwff_layer.0.bias', 'decoder.layers.2.feed_forward.pwff_layer.0.weight', 'decoder.layers.2.feed_forward.pwff_layer.3.bias', 'decoder.layers.2.feed_forward.pwff_layer.3.weight', 'decoder.layers.2.src_trg_att.k_layer.bias', 'decoder.layers.2.src_trg_att.k_layer.weight', 'decoder.layers.2.src_trg_att.output_layer.bias', 'decoder.layers.2.src_trg_att.output_layer.weight', 'decoder.layers.2.src_trg_att.q_layer.bias', 'decoder.layers.2.src_trg_att.q_layer.weight', 'decoder.layers.2.src_trg_att.v_layer.bias', 'decoder.layers.2.src_trg_att.v_layer.weight', 'decoder.layers.2.trg_trg_att.k_layer.bias', 'decoder.layers.2.trg_trg_att.k_layer.weight', 'decoder.layers.2.trg_trg_att.output_layer.bias', 'decoder.layers.2.trg_trg_att.output_layer.weight', 'decoder.layers.2.trg_trg_att.q_layer.bias', 'decoder.layers.2.trg_trg_att.q_layer.weight', 'decoder.layers.2.trg_trg_att.v_layer.bias', 'decoder.layers.2.trg_trg_att.v_layer.weight', 'decoder.layers.2.x_layer_norm.bias', 'decoder.layers.2.x_layer_norm.weight', 'decoder.layers.3.dec_layer_norm.bias', 'decoder.layers.3.dec_layer_norm.weight', 'decoder.layers.3.feed_forward.layer_norm.bias', 'decoder.layers.3.feed_forward.layer_norm.weight', 'decoder.layers.3.feed_forward.pwff_layer.0.bias', 'decoder.layers.3.feed_forward.pwff_layer.0.weight', 'decoder.layers.3.feed_forward.pwff_layer.3.bias', 'decoder.layers.3.feed_forward.pwff_layer.3.weight', 'decoder.layers.3.src_trg_att.k_layer.bias', 'decoder.layers.3.src_trg_att.k_layer.weight', 'decoder.layers.3.src_trg_att.output_layer.bias', 'decoder.layers.3.src_trg_att.output_layer.weight', 'decoder.layers.3.src_trg_att.q_layer.bias', 'decoder.layers.3.src_trg_att.q_layer.weight', 'decoder.layers.3.src_trg_att.v_layer.bias', 'decoder.layers.3.src_trg_att.v_layer.weight', 'decoder.layers.3.trg_trg_att.k_layer.bias', 'decoder.layers.3.trg_trg_att.k_layer.weight', 'decoder.layers.3.trg_trg_att.output_layer.bias', 'decoder.layers.3.trg_trg_att.output_layer.weight', 'decoder.layers.3.trg_trg_att.q_layer.bias', 'decoder.layers.3.trg_trg_att.q_layer.weight', 'decoder.layers.3.trg_trg_att.v_layer.bias', 'decoder.layers.3.trg_trg_att.v_layer.weight', 'decoder.layers.3.x_layer_norm.bias', 'decoder.layers.3.x_layer_norm.weight', 'decoder.layers.4.dec_layer_norm.bias', 'decoder.layers.4.dec_layer_norm.weight', 'decoder.layers.4.feed_forward.layer_norm.bias', 'decoder.layers.4.feed_forward.layer_norm.weight', 'decoder.layers.4.feed_forward.pwff_layer.0.bias', 'decoder.layers.4.feed_forward.pwff_layer.0.weight', 'decoder.layers.4.feed_forward.pwff_layer.3.bias', 'decoder.layers.4.feed_forward.pwff_layer.3.weight', 'decoder.layers.4.src_trg_att.k_layer.bias', 'decoder.layers.4.src_trg_att.k_layer.weight', 'decoder.layers.4.src_trg_att.output_layer.bias', 'decoder.layers.4.src_trg_att.output_layer.weight', 'decoder.layers.4.src_trg_att.q_layer.bias', 'decoder.layers.4.src_trg_att.q_layer.weight', 'decoder.layers.4.src_trg_att.v_layer.bias', 'decoder.layers.4.src_trg_att.v_layer.weight', 'decoder.layers.4.trg_trg_att.k_layer.bias', 'decoder.layers.4.trg_trg_att.k_layer.weight', 'decoder.layers.4.trg_trg_att.output_layer.bias', 'decoder.layers.4.trg_trg_att.output_layer.weight', 'decoder.layers.4.trg_trg_att.q_layer.bias', 'decoder.layers.4.trg_trg_att.q_layer.weight', 'decoder.layers.4.trg_trg_att.v_layer.bias', 'decoder.layers.4.trg_trg_att.v_layer.weight', 'decoder.layers.4.x_layer_norm.bias', 'decoder.layers.4.x_layer_norm.weight', 'decoder.layers.5.dec_layer_norm.bias', 'decoder.layers.5.dec_layer_norm.weight', 'decoder.layers.5.feed_forward.layer_norm.bias', 'decoder.layers.5.feed_forward.layer_norm.weight', 'decoder.layers.5.feed_forward.pwff_layer.0.bias', 'decoder.layers.5.feed_forward.pwff_layer.0.weight', 'decoder.layers.5.feed_forward.pwff_layer.3.bias', 'decoder.layers.5.feed_forward.pwff_layer.3.weight', 'decoder.layers.5.src_trg_att.k_layer.bias', 'decoder.layers.5.src_trg_att.k_layer.weight', 'decoder.layers.5.src_trg_att.output_layer.bias', 'decoder.layers.5.src_trg_att.output_layer.weight', 'decoder.layers.5.src_trg_att.q_layer.bias', 'decoder.layers.5.src_trg_att.q_layer.weight', 'decoder.layers.5.src_trg_att.v_layer.bias', 'decoder.layers.5.src_trg_att.v_layer.weight', 'decoder.layers.5.trg_trg_att.k_layer.bias', 'decoder.layers.5.trg_trg_att.k_layer.weight', 'decoder.layers.5.trg_trg_att.output_layer.bias', 'decoder.layers.5.trg_trg_att.output_layer.weight', 'decoder.layers.5.trg_trg_att.q_layer.bias', 'decoder.layers.5.trg_trg_att.q_layer.weight', 'decoder.layers.5.trg_trg_att.v_layer.bias', 'decoder.layers.5.trg_trg_att.v_layer.weight', 'decoder.layers.5.x_layer_norm.bias', 'decoder.layers.5.x_layer_norm.weight', 'encoder.layer_norm.bias', 'encoder.layer_norm.weight', 'encoder.layers.0.feed_forward.layer_norm.bias', 'encoder.layers.0.feed_forward.layer_norm.weight', 'encoder.layers.0.feed_forward.pwff_layer.0.bias', 'encoder.layers.0.feed_forward.pwff_layer.0.weight', 'encoder.layers.0.feed_forward.pwff_layer.3.bias', 'encoder.layers.0.feed_forward.pwff_layer.3.weight', 'encoder.layers.0.layer_norm.bias', 'encoder.layers.0.layer_norm.weight', 'encoder.layers.0.src_src_att.k_layer.bias', 'encoder.layers.0.src_src_att.k_layer.weight', 'encoder.layers.0.src_src_att.output_layer.bias', 'encoder.layers.0.src_src_att.output_layer.weight', 'encoder.layers.0.src_src_att.q_layer.bias', 'encoder.layers.0.src_src_att.q_layer.weight', 'encoder.layers.0.src_src_att.v_layer.bias', 'encoder.layers.0.src_src_att.v_layer.weight', 'encoder.layers.1.feed_forward.layer_norm.bias', 'encoder.layers.1.feed_forward.layer_norm.weight', 'encoder.layers.1.feed_forward.pwff_layer.0.bias', 'encoder.layers.1.feed_forward.pwff_layer.0.weight', 'encoder.layers.1.feed_forward.pwff_layer.3.bias', 'encoder.layers.1.feed_forward.pwff_layer.3.weight', 'encoder.layers.1.layer_norm.bias', 'encoder.layers.1.layer_norm.weight', 'encoder.layers.1.src_src_att.k_layer.bias', 'encoder.layers.1.src_src_att.k_layer.weight', 'encoder.layers.1.src_src_att.output_layer.bias', 'encoder.layers.1.src_src_att.output_layer.weight', 'encoder.layers.1.src_src_att.q_layer.bias', 'encoder.layers.1.src_src_att.q_layer.weight', 'encoder.layers.1.src_src_att.v_layer.bias', 'encoder.layers.1.src_src_att.v_layer.weight', 'encoder.layers.2.feed_forward.layer_norm.bias', 'encoder.layers.2.feed_forward.layer_norm.weight', 'encoder.layers.2.feed_forward.pwff_layer.0.bias', 'encoder.layers.2.feed_forward.pwff_layer.0.weight', 'encoder.layers.2.feed_forward.pwff_layer.3.bias', 'encoder.layers.2.feed_forward.pwff_layer.3.weight', 'encoder.layers.2.layer_norm.bias', 'encoder.layers.2.layer_norm.weight', 'encoder.layers.2.src_src_att.k_layer.bias', 'encoder.layers.2.src_src_att.k_layer.weight', 'encoder.layers.2.src_src_att.output_layer.bias', 'encoder.layers.2.src_src_att.output_layer.weight', 'encoder.layers.2.src_src_att.q_layer.bias', 'encoder.layers.2.src_src_att.q_layer.weight', 'encoder.layers.2.src_src_att.v_layer.bias', 'encoder.layers.2.src_src_att.v_layer.weight', 'encoder.layers.3.feed_forward.layer_norm.bias', 'encoder.layers.3.feed_forward.layer_norm.weight', 'encoder.layers.3.feed_forward.pwff_layer.0.bias', 'encoder.layers.3.feed_forward.pwff_layer.0.weight', 'encoder.layers.3.feed_forward.pwff_layer.3.bias', 'encoder.layers.3.feed_forward.pwff_layer.3.weight', 'encoder.layers.3.layer_norm.bias', 'encoder.layers.3.layer_norm.weight', 'encoder.layers.3.src_src_att.k_layer.bias', 'encoder.layers.3.src_src_att.k_layer.weight', 'encoder.layers.3.src_src_att.output_layer.bias', 'encoder.layers.3.src_src_att.output_layer.weight', 'encoder.layers.3.src_src_att.q_layer.bias', 'encoder.layers.3.src_src_att.q_layer.weight', 'encoder.layers.3.src_src_att.v_layer.bias', 'encoder.layers.3.src_src_att.v_layer.weight', 'encoder.layers.4.feed_forward.layer_norm.bias', 'encoder.layers.4.feed_forward.layer_norm.weight', 'encoder.layers.4.feed_forward.pwff_layer.0.bias', 'encoder.layers.4.feed_forward.pwff_layer.0.weight', 'encoder.layers.4.feed_forward.pwff_layer.3.bias', 'encoder.layers.4.feed_forward.pwff_layer.3.weight', 'encoder.layers.4.layer_norm.bias', 'encoder.layers.4.layer_norm.weight', 'encoder.layers.4.src_src_att.k_layer.bias', 'encoder.layers.4.src_src_att.k_layer.weight', 'encoder.layers.4.src_src_att.output_layer.bias', 'encoder.layers.4.src_src_att.output_layer.weight', 'encoder.layers.4.src_src_att.q_layer.bias', 'encoder.layers.4.src_src_att.q_layer.weight', 'encoder.layers.4.src_src_att.v_layer.bias', 'encoder.layers.4.src_src_att.v_layer.weight', 'encoder.layers.5.feed_forward.layer_norm.bias', 'encoder.layers.5.feed_forward.layer_norm.weight', 'encoder.layers.5.feed_forward.pwff_layer.0.bias', 'encoder.layers.5.feed_forward.pwff_layer.0.weight', 'encoder.layers.5.feed_forward.pwff_layer.3.bias', 'encoder.layers.5.feed_forward.pwff_layer.3.weight', 'encoder.layers.5.layer_norm.bias', 'encoder.layers.5.layer_norm.weight', 'encoder.layers.5.src_src_att.k_layer.bias', 'encoder.layers.5.src_src_att.k_layer.weight', 'encoder.layers.5.src_src_att.output_layer.bias', 'encoder.layers.5.src_src_att.output_layer.weight', 'encoder.layers.5.src_src_att.q_layer.bias', 'encoder.layers.5.src_src_att.q_layer.weight', 'encoder.layers.5.src_src_att.v_layer.bias', 'encoder.layers.5.src_src_att.v_layer.weight', 'src_embed.lut.weight']\n",
      "2020-10-28 08:03:37,467 cfg.name                           : swen_reverse_transformer\n",
      "2020-10-28 08:03:37,467 cfg.data.src                       : sw\n",
      "2020-10-28 08:03:37,467 cfg.data.trg                       : en\n",
      "2020-10-28 08:03:37,468 cfg.data.train                     : data/ensw/train.bpe\n",
      "2020-10-28 08:03:37,468 cfg.data.dev                       : data/ensw/dev.bpe\n",
      "2020-10-28 08:03:37,468 cfg.data.test                      : data/ensw/test.bpe\n",
      "2020-10-28 08:03:37,468 cfg.data.level                     : bpe\n",
      "2020-10-28 08:03:37,468 cfg.data.lowercase                 : False\n",
      "2020-10-28 08:03:37,468 cfg.data.max_sent_length           : 100\n",
      "2020-10-28 08:03:37,468 cfg.data.src_vocab                 : data/ensw/vocab.txt\n",
      "2020-10-28 08:03:37,468 cfg.data.trg_vocab                 : data/ensw/vocab.txt\n",
      "2020-10-28 08:03:37,468 cfg.testing.beam_size              : 5\n",
      "2020-10-28 08:03:37,468 cfg.testing.alpha                  : 1.0\n",
      "2020-10-28 08:03:37,468 cfg.training.random_seed           : 42\n",
      "2020-10-28 08:03:37,468 cfg.training.optimizer             : adam\n",
      "2020-10-28 08:03:37,468 cfg.training.normalization         : tokens\n",
      "2020-10-28 08:03:37,468 cfg.training.adam_betas            : [0.9, 0.999]\n",
      "2020-10-28 08:03:37,468 cfg.training.scheduling            : plateau\n",
      "2020-10-28 08:03:37,468 cfg.training.patience              : 5\n",
      "2020-10-28 08:03:37,468 cfg.training.learning_rate_factor  : 0.5\n",
      "2020-10-28 08:03:37,468 cfg.training.learning_rate_warmup  : 1000\n",
      "2020-10-28 08:03:37,469 cfg.training.decrease_factor       : 0.7\n",
      "2020-10-28 08:03:37,469 cfg.training.loss                  : crossentropy\n",
      "2020-10-28 08:03:37,469 cfg.training.learning_rate         : 0.0003\n",
      "2020-10-28 08:03:37,469 cfg.training.learning_rate_min     : 1e-08\n",
      "2020-10-28 08:03:37,469 cfg.training.weight_decay          : 0.0\n",
      "2020-10-28 08:03:37,469 cfg.training.label_smoothing       : 0.1\n",
      "2020-10-28 08:03:37,469 cfg.training.batch_size            : 4096\n",
      "2020-10-28 08:03:37,469 cfg.training.batch_type            : token\n",
      "2020-10-28 08:03:37,469 cfg.training.eval_batch_size       : 3600\n",
      "2020-10-28 08:03:37,469 cfg.training.eval_batch_type       : token\n",
      "2020-10-28 08:03:37,469 cfg.training.batch_multiplier      : 1\n",
      "2020-10-28 08:03:37,469 cfg.training.early_stopping_metric : ppl\n",
      "2020-10-28 08:03:37,469 cfg.training.epochs                : 5\n",
      "2020-10-28 08:03:37,469 cfg.training.validation_freq       : 1000\n",
      "2020-10-28 08:03:37,469 cfg.training.logging_freq          : 100\n",
      "2020-10-28 08:03:37,469 cfg.training.eval_metric           : bleu\n",
      "2020-10-28 08:03:37,469 cfg.training.model_dir             : models/ensw_reverse_transformer\n",
      "2020-10-28 08:03:37,470 cfg.training.overwrite             : True\n",
      "2020-10-28 08:03:37,470 cfg.training.shuffle               : True\n",
      "2020-10-28 08:03:37,470 cfg.training.use_cuda              : True\n",
      "2020-10-28 08:03:37,470 cfg.training.max_output_length     : 100\n",
      "2020-10-28 08:03:37,470 cfg.training.print_valid_sents     : [0, 1, 2, 3]\n",
      "2020-10-28 08:03:37,470 cfg.training.keep_last_ckpts       : 3\n",
      "2020-10-28 08:03:37,470 cfg.model.initializer              : xavier\n",
      "2020-10-28 08:03:37,470 cfg.model.bias_initializer         : zeros\n",
      "2020-10-28 08:03:37,470 cfg.model.init_gain                : 1.0\n",
      "2020-10-28 08:03:37,470 cfg.model.embed_initializer        : xavier\n",
      "2020-10-28 08:03:37,470 cfg.model.embed_init_gain          : 1.0\n",
      "2020-10-28 08:03:37,470 cfg.model.tied_embeddings          : True\n",
      "2020-10-28 08:03:37,470 cfg.model.tied_softmax             : True\n",
      "2020-10-28 08:03:37,470 cfg.model.encoder.type             : transformer\n",
      "2020-10-28 08:03:37,470 cfg.model.encoder.num_layers       : 6\n",
      "2020-10-28 08:03:37,470 cfg.model.encoder.num_heads        : 4\n",
      "2020-10-28 08:03:37,470 cfg.model.encoder.embeddings.embedding_dim : 256\n",
      "2020-10-28 08:03:37,471 cfg.model.encoder.embeddings.scale : True\n",
      "2020-10-28 08:03:37,471 cfg.model.encoder.embeddings.dropout : 0.2\n",
      "2020-10-28 08:03:37,471 cfg.model.encoder.hidden_size      : 256\n",
      "2020-10-28 08:03:37,471 cfg.model.encoder.ff_size          : 1024\n",
      "2020-10-28 08:03:37,471 cfg.model.encoder.dropout          : 0.3\n",
      "2020-10-28 08:03:37,471 cfg.model.decoder.type             : transformer\n",
      "2020-10-28 08:03:37,471 cfg.model.decoder.num_layers       : 6\n",
      "2020-10-28 08:03:37,471 cfg.model.decoder.num_heads        : 4\n",
      "2020-10-28 08:03:37,471 cfg.model.decoder.embeddings.embedding_dim : 256\n",
      "2020-10-28 08:03:37,471 cfg.model.decoder.embeddings.scale : True\n",
      "2020-10-28 08:03:37,471 cfg.model.decoder.embeddings.dropout : 0.2\n",
      "2020-10-28 08:03:37,471 cfg.model.decoder.hidden_size      : 256\n",
      "2020-10-28 08:03:37,471 cfg.model.decoder.ff_size          : 1024\n",
      "2020-10-28 08:03:37,471 cfg.model.decoder.dropout          : 0.3\n",
      "2020-10-28 08:03:37,471 Data set sizes: \n",
      "\ttrain 873221,\n",
      "\tvalid 1000,\n",
      "\ttest 2721\n",
      "2020-10-28 08:03:37,471 First training example:\n",
      "\t[SRC] Is@@ ito@@ she , kwa kuwa si@@ j@@ a@@ anzisha ur@@ afiki kama huo n@@ ina nafasi ya kufanya mambo mengi ya pekee !\n",
      "\t[TRG] B@@ es@@ id@@ es , being sing@@ le gives one the opportun@@ ity to do so many great things !\n",
      "2020-10-28 08:03:37,472 First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) , (5) . (6) the (7) na (8) ya (9) to\n",
      "2020-10-28 08:03:37,472 First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) , (5) . (6) the (7) na (8) ya (9) to\n",
      "2020-10-28 08:03:37,472 Number of Src words (types): 4528\n",
      "2020-10-28 08:03:37,472 Number of Trg words (types): 4528\n",
      "2020-10-28 08:03:37,472 Model(\n",
      "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
      "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
      "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=4528),\n",
      "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=4528))\n",
      "2020-10-28 08:03:37,476 EPOCH 1\n",
      "2020-10-28 08:03:53,235 Epoch   1 Step:      100 Batch Loss:     5.857098 Tokens per Sec:    15635, Lr: 0.000300\n",
      "2020-10-28 08:04:07,950 Epoch   1 Step:      200 Batch Loss:     5.479127 Tokens per Sec:    17226, Lr: 0.000300\n",
      "2020-10-28 08:04:23,188 Epoch   1 Step:      300 Batch Loss:     5.434530 Tokens per Sec:    16949, Lr: 0.000300\n",
      "2020-10-28 08:04:38,190 Epoch   1 Step:      400 Batch Loss:     5.162995 Tokens per Sec:    16283, Lr: 0.000300\n",
      "2020-10-28 08:04:53,332 Epoch   1 Step:      500 Batch Loss:     5.056055 Tokens per Sec:    16569, Lr: 0.000300\n",
      "2020-10-28 08:05:08,178 Epoch   1 Step:      600 Batch Loss:     5.164367 Tokens per Sec:    16593, Lr: 0.000300\n",
      "2020-10-28 08:05:23,268 Epoch   1 Step:      700 Batch Loss:     4.644957 Tokens per Sec:    16859, Lr: 0.000300\n",
      "2020-10-28 08:05:38,370 Epoch   1 Step:      800 Batch Loss:     4.659764 Tokens per Sec:    16063, Lr: 0.000300\n",
      "2020-10-28 08:05:53,575 Epoch   1 Step:      900 Batch Loss:     4.767367 Tokens per Sec:    16447, Lr: 0.000300\n",
      "2020-10-28 08:06:08,473 Epoch   1 Step:     1000 Batch Loss:     4.590811 Tokens per Sec:    16664, Lr: 0.000300\n",
      "2020-10-28 08:06:53,106 Hooray! New best validation result [ppl]!\n",
      "2020-10-28 08:06:53,106 Saving new checkpoint.\n",
      "2020-10-28 08:06:53,657 Example #0\n",
      "2020-10-28 08:06:53,658 \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2020-10-28 08:06:53,658 \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2020-10-28 08:06:53,658 \tHypothesis: The Bible years of the Bible of the Bible .\n",
      "2020-10-28 08:06:53,658 Example #1\n",
      "2020-10-28 08:06:53,658 \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2020-10-28 08:06:53,658 \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2020-10-28 08:06:53,658 \tHypothesis: The Bible of the Bible of the Bible of the Bible of the Bible of the Bible .\n",
      "2020-10-28 08:06:53,658 Example #2\n",
      "2020-10-28 08:06:53,658 \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2020-10-28 08:06:53,659 \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2020-10-28 08:06:53,659 \tHypothesis: The Bible of the Bible of the Sry , the S.00 , the S.00 , the S.00 , the S.00 .\n",
      "2020-10-28 08:06:53,659 Example #3\n",
      "2020-10-28 08:06:53,659 \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2020-10-28 08:06:53,659 \tReference:  Lots of hugs and kisses .\n",
      "2020-10-28 08:06:53,659 \tHypothesis: The Bible of the Bible of the Bible .\n",
      "2020-10-28 08:06:53,659 Validation result (greedy) at epoch   1, step     1000: bleu:   0.65, loss: 130064.1875, ppl:  93.1953, duration: 45.1859s\n",
      "2020-10-28 08:07:08,904 Epoch   1 Step:     1100 Batch Loss:     4.363812 Tokens per Sec:    16774, Lr: 0.000300\n",
      "2020-10-28 08:07:24,001 Epoch   1 Step:     1200 Batch Loss:     4.571873 Tokens per Sec:    16974, Lr: 0.000300\n",
      "2020-10-28 08:07:38,952 Epoch   1 Step:     1300 Batch Loss:     4.495600 Tokens per Sec:    16592, Lr: 0.000300\n",
      "2020-10-28 08:07:54,146 Epoch   1 Step:     1400 Batch Loss:     4.358923 Tokens per Sec:    16642, Lr: 0.000300\n",
      "2020-10-28 08:08:09,250 Epoch   1 Step:     1500 Batch Loss:     4.187090 Tokens per Sec:    16597, Lr: 0.000300\n",
      "2020-10-28 08:08:24,085 Epoch   1 Step:     1600 Batch Loss:     4.179198 Tokens per Sec:    16518, Lr: 0.000300\n",
      "2020-10-28 08:08:38,896 Epoch   1 Step:     1700 Batch Loss:     4.276514 Tokens per Sec:    16347, Lr: 0.000300\n",
      "2020-10-28 08:08:53,814 Epoch   1 Step:     1800 Batch Loss:     4.157403 Tokens per Sec:    16771, Lr: 0.000300\n",
      "2020-10-28 08:09:08,793 Epoch   1 Step:     1900 Batch Loss:     4.006355 Tokens per Sec:    16619, Lr: 0.000300\n",
      "2020-10-28 08:09:23,678 Epoch   1 Step:     2000 Batch Loss:     4.083451 Tokens per Sec:    16927, Lr: 0.000300\n",
      "2020-10-28 08:10:08,214 Hooray! New best validation result [ppl]!\n",
      "2020-10-28 08:10:08,214 Saving new checkpoint.\n",
      "2020-10-28 08:10:08,728 Example #0\n",
      "2020-10-28 08:10:08,728 \tSource:     Katika mwaka wa 1997 wasichana 90,000 walipata mimba huko Uingereza .\n",
      "2020-10-28 08:10:08,728 \tReference:  There were almost 90,000 conceptions to teenagers in England in 1997 .\n",
      "2020-10-28 08:10:08,728 \tHypothesis: In 198 - year century century of the United States .\n",
      "2020-10-28 08:10:08,728 Example #1\n",
      "2020-10-28 08:10:08,728 \tSource:     Lazima ujikakamue kujua kile isemacho Biblia kusudi usadikishwe na kutegemeka kwake .\n",
      "2020-10-28 08:10:08,728 \tReference:  You must exert yourself to find out what the Bible says so as to be convinced of its reliability .\n",
      "2020-10-28 08:10:08,728 \tHypothesis: The Bible who have been been a Bible who had to be a Bible and that he was a good news of his Kingdom .\n",
      "2020-10-28 08:10:08,728 Example #2\n",
      "2020-10-28 08:10:08,729 \tSource:     Kwenye kituo kikuu katika jiji la Strasbourg , ambalo ni makao ya Mahakama ya Ulaya ya Haki za Kibinadamu , wasafiri walipanga foleni wakisubiri kupata nakala yao .\n",
      "2020-10-28 08:10:08,729 \tReference:  In Strasbourg , home of the European Court of Human Rights , travelers at the central station line up patiently to receive their copy .\n",
      "2020-10-28 08:10:08,729 \tHypothesis: The first - century of the book of the Shurch , the book of the United States , the book of the book of the United States , the Bible - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -\n",
      "2020-10-28 08:10:08,729 Example #3\n",
      "2020-10-28 08:10:08,729 \tSource:     Kukumbatia kwingi na busu nyingi .\n",
      "2020-10-28 08:10:08,729 \tReference:  Lots of hugs and kisses .\n",
      "2020-10-28 08:10:08,729 \tHypothesis: The source of the source of the sast of the source of the sast .\n",
      "2020-10-28 08:10:08,729 Validation result (greedy) at epoch   1, step     2000: bleu:   1.71, loss: 111684.9609, ppl:  49.1022, duration: 45.0513s\n",
      "2020-10-28 08:10:23,790 Epoch   1 Step:     2100 Batch Loss:     4.100314 Tokens per Sec:    16667, Lr: 0.000300\n",
      "2020-10-28 08:10:38,977 Epoch   1 Step:     2200 Batch Loss:     3.512985 Tokens per Sec:    16564, Lr: 0.000300\n",
      "2020-10-28 08:10:53,762 Epoch   1 Step:     2300 Batch Loss:     3.825576 Tokens per Sec:    16690, Lr: 0.000300\n",
      "2020-10-28 08:11:09,007 Epoch   1 Step:     2400 Batch Loss:     3.734296 Tokens per Sec:    16873, Lr: 0.000300\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
    "!cd joeynmt; python3 -m joeynmt train configs/transformer_reverse_$tgt$src.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MBoDS09JM807"
   },
   "outputs": [],
   "source": [
    "# Copy the created models from the notebook storage to google drive for persistant storage \n",
    "!cp -r joeynmt/models/${tgt}${src}_reverse_transformer/* \"$gdrive_path/models/${src}${tgt}_reverse_transformer/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "n94wlrCjVc17"
   },
   "outputs": [],
   "source": [
    "# Output our validation accuracy\n",
    "! cat \"$gdrive_path/models/${tgt}${src}_reverse_transformer/validations.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "66WhRE9lIhoD"
   },
   "outputs": [],
   "source": [
    "# Test our model\n",
    "! cd joeynmt; python3 -m joeynmt test \"$gdrive_path/models/${tgt}${src}_reverse_transformer/config.yaml\""
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "starter_notebook_reverse_training.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}