tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Item2Vec\n",
    "Item2Vec: Neural Item Embedding for Collaborative Filtering\n",
    "***"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "- Paper: https://arxiv.org/vc/arxiv/papers/1603/1603.04259v2.pdf\n",
    "- Dataset: https://grouplens.org/datasets/movielens/\n",
    "- Source Code:\n",
    "https://github.com/bwange/Item2vec_Tutorial_with_Recommender_System_Application/blob/master/Making_Your_Own_Recommender_System_with_Item2Vec.ipynb\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Preparing the Data\n",
    "- MovieLens 데이터셋을 사용하여 데이터를 읽습니다.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4903</th>\n",
       "      <td>4999</td>\n",
       "      <td>Dodsworth (1936)</td>\n",
       "      <td>Drama|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22418</th>\n",
       "      <td>107630</td>\n",
       "      <td>High School (2010)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10535</th>\n",
       "      <td>39703</td>\n",
       "      <td>Little Fish (2005)</td>\n",
       "      <td>Crime|Drama|Romance|Thriller</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21653</th>\n",
       "      <td>104865</td>\n",
       "      <td>Ping Pong (2012)</td>\n",
       "      <td>Documentary</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8137</th>\n",
       "      <td>8820</td>\n",
       "      <td>Spinout (1966)</td>\n",
       "      <td>Comedy|Musical</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       movieId               title                        genres\n",
       "4903      4999    Dodsworth (1936)                 Drama|Romance\n",
       "22418   107630  High School (2010)                        Comedy\n",
       "10535    39703  Little Fish (2005)  Crime|Drama|Romance|Thriller\n",
       "21653   104865    Ping Pong (2012)                   Documentary\n",
       "8137      8820      Spinout (1966)                Comedy|Musical"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Randomly displaying 5 of the total 27278 data points\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11538517</th>\n",
       "      <td>79641</td>\n",
       "      <td>537</td>\n",
       "      <td>5.0</td>\n",
       "      <td>837388707</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18436672</th>\n",
       "      <td>127615</td>\n",
       "      <td>2324</td>\n",
       "      <td>3.0</td>\n",
       "      <td>949789468</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83744</th>\n",
       "      <td>593</td>\n",
       "      <td>195</td>\n",
       "      <td>3.0</td>\n",
       "      <td>862658718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17262126</th>\n",
       "      <td>119384</td>\n",
       "      <td>1200</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1079582071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10203944</th>\n",
       "      <td>70604</td>\n",
       "      <td>919</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1107415170</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          userId  movieId  rating   timestamp\n",
       "11538517   79641      537     5.0   837388707\n",
       "18436672  127615     2324     3.0   949789468\n",
       "83744        593      195     3.0   862658718\n",
       "17262126  119384     1200     3.5  1079582071\n",
       "10203944   70604      919     4.0  1107415170"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Randomly displaying 5 of the total 20000263 data points\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# 데이터 읽기\n",
    "df_movies = pd.read_csv('../dataset/ml-20m/movies.csv') # 영화 데이터\n",
    "df_ratings = pd.read_csv('../dataset/ml-20m/ratings.csv') # user의 rating 데이터\n",
    "\n",
    "# 인코딩\n",
    "movieId_to_name = pd.Series(df_movies.title.values, index = df_movies.movieId.values)\n",
    "name_to_movieId = pd.Series(df_movies.movieId.values, index = df_movies.title).to_dict()\n",
    "\n",
    "# Randomly display 5 records in the dataframe\n",
    "for df in list((df_movies, df_ratings)):\n",
    "    rand_idx = np.random.choice(len(df), 5, replace=False)\n",
    "    display(df.iloc[rand_idx,:])\n",
    "    print(\"Randomly displaying 5 of the total \"+str(len(df))+\" data points\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rating Histogram\n",
    "- User가 rating한 영화의 개수를 히스토그램으로 시각화합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhsAAAGMCAYAAABzrGDbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de5xfVX3v/9cbIiCEKGBEoApiRRR/gJhW2+MlKipesECsUhBEK2g9HKpotefITdBTQWktKgoKcrdUuRXxYFWMFGvVIDdTEAXDnRpoDIRAUPz8/th7ZPPNTGYGZs+Qyev5eHwfM3utvdZe3z3fZN6z9trfb6oKSZKkvqw11QOQJEnTm2FDkiT1yrAhSZJ6ZdiQJEm9MmxIkqReGTYkSVKvDBta4yXZL0l1HvclWZTkvCRvSbLWwP5btfvtN45jzE1yxGBfYxzXVp2yRUnOGGsfj3Zcj+Y5TqYkayX5dJI7kvwuyfmr2HdR+1zOGqF+flt/WY/jXZTklAnqa+7A6/W3SW5OcnySjR5Dn6vd60CrjxlTPQDpceTPgVuBdYFnAG8AvgIckGTXqrq/3e8O4E+AG8bR91zgcOBjwO/G2Oai9jh3jOM44zWX4cf1aJ7jZHoz8NfAB4AfAHePsv+9wG5JNqyqe4cKk2wJvKyt79PuwD0T3OdBwI+B9YFXAR8Gng7s+ij6msvq+TrQasKwIT3syqr6RWf79CRfBb4KHAP8L4CqWgH8R1+DSPIE4LdVtRhY3NdxVqXv5zgBntt+/XRVjSW8fQvYGZgHnNIp3wdYBNwCrD2B43uEqrqih26vraqhn9ElSZ4KvCvJ06rqzok4wGrwOtBqwsso0ipU1TnABcD+SdaH4aeWk/xRkm8luTvJ8iQ3Jjm+rTuC5q9GgN8MTX8P9PXeJMckuR1YATx5uMsonePtn+QXSR5I8pMkrxion59k/jDtfj+dP8Zx7TfQ/m1JrmqPe1eS05NsNswxzkiyZ5Jr28tSC5K8ZLTz3bbfJckPktyfZGmS85M8p9s/cES7+dAYp/nvB86hCRdd+wCnAyu9lXKSzZKc1j7PFUmuTvK2Tv0ft8deaSYhyeeTLG6D47CXUZI8M8mZ7X4rklyZZPdRnseq/KT9+ozOMdZL8g9JfppkWZI7k1yYZNvOPkcwjtdBklOS3JrkBUn+rX29/zzJe4Y5DzsnuaJ9vfwiybva9os6+8xIclSSGzqvq8vG+nrR6sGwIY3uGzSXVuYMV5lkJvBN4CFgP+D1wJE8PHP4JeCk9vuX0ExL/8lANx8BtgEOoJlyf2AV43k5cHDbZk+acPL/ur+Qx2gs4/q9JAfQ/GK+FtgD+FvgtcD32nPQ9VKaSxyHAm+lmTX4epInr2pASXahuXy0rG33V8DzgcuSbNHutjsPz04MjfmiVT9VAE4D5ib5g/ZYL6Y556cPM44NgO8BrwP+D7AbcA3NbNcBAFX1I+BnDASYJOsAbwH+qap+M8LzfDrwQ2AH4P3Am2jCwjlJ3jSG5zKcrWheg4s6ZesCG9JcHnkDzflcD/iPJE9r9xnX66A1CzgLOAP4M5rLOZ/vht4kz+Phn+WeNOfxr4FXDvT1YZpzcBzN6+kdwHeAjUd9xlp9VJUPH2v0gyYgFPCHI9S/tq1/a7u9Vbu9X7s9p93efhXHOKLdZ8ZA+VBfPwEywri26pQtAh4EntEp2xD4b+D0Ttl8YP4w41gEnDKOcQ09x7WB/wK+O7DfS9r9Dho4xhJgo07Z0Dnaa5SfxQLg593xAM8EfgP8fafsY81/X2P6+S6i+aWY9vu/bcuPB77fOV+Xddoc2I537kBf3wZ+Bazdbn+EZtbkSZ19dmvb/vEqzvtJNJfINhno/1s0l/NW9Xzmtv2/hibQbtge8x7gU6O0XZtmjce9wPvH+zpoy05py17RKVsXuAs4sVN2Vvsc1++UbUYTpBd1yr4OnDtR/559PD4fzmxIo0v7daRPLfw58GvghPYyw9MfxTHOr/Z/3jH4j6q6eWijmgWPQ4tJ+/Ic4KnAmd3CqroMuIlmtqXrB1W1pLN9Tfv1GYygnU3YCTi7qn7bOcYvge8Pc4xxac/vGcA+7ezDW2lmO4bzMuC2qpo/UH4GMBt4Xmd7XZrFxUP2AX5WzczHSHahmTFb2l5GmJFkBs0M2Q5JZo3hKX2TJoTdA5wHXAr8zeBOae6o+mGSXwO/Be4DZtL8TB+t5VX13aGNatZ2/JxH/nxfDHyjqpZ39rsD+PeBvn4MvD7Jx5O8pP3ZaJoxbEijGwoPw94VUlVLgVcAt9P8tXxze4183jiOMZ47Tv5rhLIthimfKENT2sON805WnvL+7+5G+8sImin8kWxEE+zGeoxH4zSaoHA4sAFw9gj7bbyKcQzVU1U30fyS3wegvUz0Boa5NDPgqcC+NGGh+/hkW7/J6E+F/wn8Ec3C17Pb4x7a3aFdT3I2zaWvvYAXtW0Ws+qfxWiWDFO2YqDPzWhmgQYNvn7/L83P403AvwF3J/lykqc8hvHpcca7UaTRvYFm6vfykXaoqiuBee1fp3OA/w38c5IdquqnYzjGWGc1ADYdoey2zvYDNNfVBz3aX9hD4eFpw9Q9jebyx2O1hOY8jHSM0W5vHVVVXZ/khzTrTc6tql+PsOt/M/xf/kNj647ldOCLaW6jfS2wDgMzQMO4m+YX69Ej1N8+SnuA66tqAUCSS2heA/8nyZer6pZ2nz2BX1TVfkON2kWrk7Ee4g6aUDXoEa/fata1HA0c3a4jeSPw9zSXe97a9yA1OZzZkFYhyR40f3F9oTsdPJKq+m01tyMeSvPva+gWzaG/7J84AcN6cfdSTZINaQLRDzr73ARs052STvIymuv7XWMd189o/iLds1uY5E+BLWkWUz4mVXUfTaD78yS/vw21/SX+pxNxjNYxwIXAZ1exz/eAP0jyPwbK96L5a/3aTtlXacLd3jQzHJdW1aJRxnAxsD2wsKoWDPNYMUr7R2gvEb2PJuj8badqfZpLJ137sPJtvhP5+hzyHzSXR9YfKmjvXBo8p79XVXdW1Zdo1sY8fwLHoinmzIb0sB3bqdt1aK49v5HmWvy3aGYqhpXkjTR3kZwP/JJmev4gmkV4QwHgP9uvH0jy/4CHhv4qfRT+C/jX9pbFFTSr+TcAjurs80/tmE5ub7l8Js0dLEsH+hrTuKrqoSSH0axLOYNmrcIWwMdprtV/+VE+l0GH0qw/+XqaW4dnAh9tx33sRBygqs4Fzh1lt1No7pw4N8lHaN7sbW/g1cC7q+qhTn/3JPkXmssamwH7j2EYhwE/Ai5N8lmaBaQb0fyC3bqq3jme59SO46ok5wB/meTjVXU7TajZLck/0CzEfCHNa3NwRmciX59DPkbz5mvfTPIpmrUth9K8fn//3ihJLgCuolkkvQR4Ac2alhMe4/H1eDLVK1R9+JjqBw/f9TH0uJ9mZuA8mrAxeJfIVjzyTo3n0FwX/yXNX7iLaRb/vajTZm3gczR/Ff+O3/8x+vu+3rWKcW3VKVtE84v+XTTv6rgCuAJ45TDt300TBO6nWZT3Qla+K2K0ce030OfbaH4xrKC5FHA6sNnAPouAM4YZTwFHjOHnsQtNSLufJmRcADxnYJ9x340yyj7z6dyN0pZt1j6/u9rnezXwthHav6Hz2nnSCGM4ZaDsD2huO72N5g6jO2iC7bDH6LSb2x5r52Hqnktz++s/tttrtefqdmA5zYzNCx7L64AmiN06wjmcP1D2auDK9vzd2L4mzwOu6OzzAZpZkLvb8/czmrtjnjBZ/wf46P+R9octSVKv2vdj+QVwUVX95VSPR5PHyyiSpF4k+QzNrNrtwOY0l6Y2Av5xKselyWfYkCT1ZT2aO002pblU9COayz9XT+moNOm8jCJJknrlra+SJKlXhg1JktQr12z0ZJdddqmLL754qochSdJkyUgVzmz05K677prqIUiS9Lhg2JAkSb0ybEiSpF4ZNiRJUq8MG5IkqVeGDUmS1CvDhiRJ6pVhQ5Ik9cqwIUmSemXYkCRJvTJsSJKkXhk2JElSrwwbkiSpV4YNSZLUKz9iXpL0CHP3PWyqhzCq+acdOdVD0Dg4syFJknpl2JAkSb0ybEiSpF4ZNiRJUq8MG5IkqVeGDUmS1CvDhiRJ6pVhQ5Ik9cqwIUmSemXYkCRJvTJsSJKkXhk2JElSrwwbkiSpV4YNSZLUK8OGJEnqlWFDkiT1yrAhSZJ6ZdiQJEm9MmxIkqReGTYkSVKvDBuSJKlXhg1JktQrw4YkSeqVYUOSJPXKsCFJknpl2JAkSb0ybEiSpF5NathIMj/JA0mWtY+fdepeleS6JMuTfDfJlp26JDk6yd3t45gk6dRv1bZZ3vax88Bx90pyU5L7kpyfZONO3bpJTk5yT5I7kxw80HbHJJe3fV+eZMd+zo4kSdPTVMxsHFhVM9vHcwCSPAU4FzgU2BhYAJzdaXMAsBuwA7A98Ebg3Z36rwBXAJsAHwG+lmR22/d2wAnAPsCmwHLg+E7bI4BnA1sCrwA+lGSXtu06wAXAGcBGwKnABW25JEkag8fLZZQ9gIVV9dWqeoAmAOyQZNu2/u3AsVV1a1XdBhwL7AeQZBtgJ+Dwqrq/qs4BrgHmtW33Bi6sqkurahlNoNkjyYZt/b7AUVW1pKquBb441DcwF5gBfLqqVlTVcUCAV/ZxEiRJmo6mImz8XZK7knw/ydy2bDvgqqEdquo+4Ia2fKX69vtu3Y1Vde8q6rt93wA8CGyTZCNg81H6vrqqqlN/dadekiSNYrLDxoeBrYEtgBOBC5M8C5gJLB3YdykwNPswWL8UmNmu2xhv2279zM72eNuuJMkBSRYkWbB48eLhdpEkaY0zqWGjqn5YVfe2lyROBb4PvB5YBswa2H0WMDRbMVg/C1jWzjiMt223fllne7xth3t+J1bVnKqaM3v27OF2kSRpjTPVazaKZg3EQprFnwAk2QB4VlvOYH37fbdu684ajOHqu31vDawLXF9VS4A7Rul7++6dLzQLVBciSZLGZNLCRpInJ3ltkvWSzEiyN/Ay4JvAecDzk8xLsh5wGM1aieva5qcBByfZIsnmwAeAUwCq6nrgSuDwtu/daQLBOW3bM4Fdk7y0DTFHAud21nicBhySZKN2Qer+Q30D84GHgIPaW2QPbMsvmeDTI0nStDVjEo/1BOBjwLY0v8CvA3arqp8BJJkHfJbmNtMfAnt22p5As9bjmnb7S23ZkD1pAsIS4GbgzVW1GKCqFiZ5D03o2AT4NvCOTtvDgc8DNwH3A0dX1cVt2weT7NYe7xPAte2YH3yM50KSpDVGHnmjhSbKnDlzasGCBVM9DEkat7n7HjbVQxjV/NOOnOohaGUZqWKq12xIkqRpzrAhSZJ6ZdiQJEm9MmxIkqReGTYkSVKvDBuSJKlXhg1JktQrw4YkSeqVYUOSJPXKsCFJknpl2JAkSb0ybEiSpF4ZNiRJUq8MG5IkqVeGDUmS1CvDhiRJ6pVhQ5Ik9cqwIUmSemXYkCRJvTJsSJKkXhk2JElSrwwbkiSpV4YNSZLUK8OGJEnqlWFDkiT1yrAhSZJ6ZdiQJEm9MmxIkqReGTYkSVKvDBuSJKlXhg1JktQrw4YkSeqVYUOSJPVqxlQPQJIm0tx9D5vqIazS/NOOnOohSJPOmQ1JktQrw4YkSeqVYUOSJPXKsCFJknpl2JAkSb0ybEiSpF4ZNiRJUq8MG5IkqVeGDUmS1CvDhiRJ6pVhQ5Ik9cqwIUmSejUlYSPJs5M8kOSMTtmrklyXZHmS7ybZslOXJEcnubt9HJMknfqt2jbL2z52HjjeXkluSnJfkvOTbNypWzfJyUnuSXJnkoMH2u6Y5PK278uT7NjPWZEkaXqaqpmNzwE/HtpI8hTgXOBQYGNgAXB2Z/8DgN2AHYDtgTcC7+7UfwW4AtgE+AjwtSSz2763A04A9gE2BZYDx3faHgE8G9gSeAXwoSS7tG3XAS4AzgA2Ak4FLmjLJUnSGEx62EiyJ/Br4Dud4j2AhVX11ap6gCYA7JBk27b+7cCxVXVrVd0GHAvs1/a3DbATcHhV3V9V5wDXAPPatnsDF1bVpVW1jCbQ7JFkw7Z+X+CoqlpSVdcCXxzqG5gLzAA+XVUrquo4IMArJ+yESJI0zU1q2EgyCzgS+MBA1XbAVUMbVXUfcENbvlJ9+3237saquncV9d2+bwAeBLZJshGw+Sh9X11V1am/ulM/+PwOSLIgyYLFixcPt4skSWucyZ7ZOAo4qapuGSifCSwdKFsKbDhC/VJgZrtuY7xtu/UzO9vjbbuSqjqxquZU1ZzZs2cPt4skSWucGZN1oHZh5c7AC4apXgbMGiibBdw7Qv0sYFlVVZLxtu3WL+tsPzDOtpIkaQwmc2ZjLrAVcHOSO4EPAvOS/ARYSLP4E4AkGwDPassZrG+/79Zt3VmDMVx9t++tgXWB66tqCXDHKH1v373zhWaB6kIkSdKYTGbYOJEmQOzYPr4AXAS8FjgPeH6SeUnWAw6jWStxXdv2NODgJFsk2ZxmzccpAFV1PXAlcHiS9ZLsThMIzmnbngnsmuSlbYg5Eji3s8bjNOCQJBu1C1L3H+obmA88BBzU3iJ7YFt+yQSeF0mSprVJu4xSVctpbjsFoL388UBVLW635wGfpbnN9IfAnp3mJwBb09xlAvCltmzInjQBYQlwM/DmoX6ramGS99CEjk2AbwPv6LQ9HPg8cBNwP3B0VV3ctn0wyW7t8T4BXAvsVlUPPpZzIUnSmmTSwsagqjpiYPvbwLYj7FvAh9rHcPWLaC7TjHSss4CzRqhbAbyzfQxXfwXwwpH6liRJq+bblUuSpF4ZNiRJUq8MG5IkqVeGDUmS1CvDhiRJ6pVhQ5Ik9cqwIUmSemXYkCRJvTJsSJKkXhk2JElSrwwbkiSpV4YNSZLUK8OGJEnqlWFDkiT1yrAhSZJ6ZdiQJEm9MmxIkqReGTYkSVKvDBuSJKlXhg1JktQrw4YkSeqVYUOSJPXKsCFJknpl2JAkSb0ybEiSpF4ZNiRJUq8MG5IkqVeGDUmS1CvDhiRJ6pVhQ5Ik9cqwIUmSemXYkCRJvTJsSJKkXhk2JElSrwwbkiSpV2MOG0lelmTGMOUzkrxsYoclSZKmi/HMbHwX2HiY8ie1dZIkSSsZT9gIUMOUbwLcNzHDkSRJ081Kl0UGJfmX9tsCzkiyolO9NvB84N97GJskSZoGRg0bwN3t1wBLgPs7dQ8ClwFfnOBxSZKkaWLUsFFV7wBIsgj4VFV5yUSSJI3ZWGY2AKiqj/Y5EEmSND2NOWwk2Rj4OPAq4KkMLC6tqlkTOzRJkjQdjDlsACcBLwBOBG5n+DtTJEmSHmE8YeNVwKur6od9DUaSJE0/43mfjV8By/oaiCRJmp7GEzY+AhyZZOajPViSM5LckeSeJNcneVen7lVJrkuyPMl3k2zZqUuSo5Pc3T6OSZJO/VZtm+VtHzsPHHevJDcluS/J+e36k6G6dZOc3I7pziQHD7TdMcnlbd+XJ9nx0T5/SZLWROMJG4cArwF+leTaJFd3H2Ps4++ArdrFpG8CPpbkhUmeApwLHErzlugLgLM77Q4AdgN2ALYH3gi8u1P/FeAKmncz/QjwtSSzAZJsB5wA7ANsCiwHju+0PQJ4NrAl8ArgQ0l2aduuA1wAnAFsBJwKXNCWS5KkMRjPmo2vPdaDVdXC7mb7eBbwQmBhVX0VIMkRwF1Jtq2q64C3A8dW1a1t/bHA/sAXkmwD7AS8pqruB85J8j5gHvAFYG/gwqq6tG17KHBtkg2r6l5gX+AdVbUEWJLki8B+wMXAXJpz9OmqKuC4JB8EXtnWS5KkUUz6+2wkOZ7ml/kTaWYjvkFzS+1VnWPdl+QGYDvguvbrVZ1urmrLaL/e2AaHkep//3bqVXVDkgeBbZLcCGw+TN+7ddpe3QaNIVe35SuFjSQH0MzC8IxnPGNVp0GSpDXGeC6jTIiqei+wIfBSmksnK4CZwNKBXZe2+zFM/VJgZrtuY7xtu/UzO9vjbTvcczuxquZU1ZzZs2cPt4skSWuc8byp172s4r01xvOmXlX1EHBZkrcBf0Vzl8tg+1nA0GzFYP0sYFlVVZLxtu3WL+tsPzDOtpIkaQzGs2bjwIHtJ9C8ydc8mssgj/b4zwIW0qzLACDJBp1y2q87AD9qt3cYqNu6swZjqP6sgbZDfW8NrAtcX1X3Jrmjrf/WCH1/IEk6l1K2Bz73KJ+vJElrnPGs2Th1uPIkP6F5w6/PrKp9kqfSLKz8Os0nx+4M/AWwF82aik8mmQdcBBxGs1biurb5acDBSb5BM7vygaHjVdX1Sa4EDk9yCPA6mkAwr217JvCDJC8FfgIcCZzbCSanAYckWUBzt8r+wDvauvnAQ8BBSb7Q1gFcsqrnKkmSHjYRaza+C+w6hv2K5pLJrTQfVf8p4H1VdUFVLebhGZIlwIuAPTttTwAuBK4BfkoTSE7o1O8JzGnbfgJ4c9vn0B0w76EJHb+iWW/x3k7bw4EbgJuA7wGfrKqL27YP0iwW3Rf4NfBOYLe2XJIkjcF4LqOMZE/grtF2an/5v3wV9d8Gth2hroAPtY/h6hfR3KY6Ut9n8fBllcG6FTQh4p0j1F9Bc2uuJEljMnffw6Z6CKOaf9qRk3as8SwQvYZHLhANzWWHjWlmLCRJklbyWN7U63fAYmB+Z22FJEnSI0z6m3pJkqQ1y7jXbCR5JfA8mksqC6tq/kQPSpIkTR/jWbOxBXAezWLJ29vizdtbRnevqttHbCxJktZY47n19Tia95z4w6p6elU9nebTUh9q6yRJklYynssorwbmVtUvhwqq6sYkBwHfmfCRSZKkaWEi3tTrdxPQhyRJmqbGEza+AxyX5OlDBUmeAfwjzmxIkqQRjCdsHASsD9yY5KYki2je5nv9tk6SJGkl43mfjVuAnZK8muZtxQP8Z/s245IkScMadWYjyeuSLEryJICq+lZVfaaqjgN+3Na9pveRSpKk1dJYLqMcSPNJqEsHK9qyo4G/nuiBSZKk6WEsYWN7YFWXSi4BdpiY4UiSpOlmLGFjNqu+vbWATSZmOJIkaboZS9i4lWZ2YyTbA7dNzHAkSdJ0M5awcRFwVJInDlYkWR84st1HkiRpJWO59fXjwJuBnyf5DHBdW/5cmsWjAf5vP8OTJEmru1HDRlX9KsmfAp+nCRUZqgK+Cby3qv6rvyFKkqTV2Zje1KuqbgJen2Qj4A9pAsfPq2pJn4OTJEmrv/F86ittuPhxT2ORJEnT0ER86qskSdKIDBuSJKlXhg1JktQrw4YkSeqVYUOSJPXKsCFJknpl2JAkSb0ybEiSpF4ZNiRJUq8MG5IkqVeGDUmS1CvDhiRJ6pVhQ5Ik9cqwIUmSemXYkCRJvTJsSJKkXhk2JElSr2ZM9QAkaU0yd9/DpnoI0qRzZkOSJPXKsCFJknpl2JAkSb0ybEiSpF4ZNiRJUq8MG5IkqVeGDUmS1KtJCxtJ1k1yUpKbktyb5Iokr+vUvyrJdUmWJ/luki07dUlydJK728cxSdKp36pts7ztY+eBY+/VHve+JOcn2XhgXCcnuSfJnUkOHmi7Y5LL274vT7JjP2dIkqTpaTJnNmYAtwAvB54EHAr8cxsUngKc25ZtDCwAzu60PQDYDdgB2B54I/DuTv1XgCuATYCPAF9LMhsgyXbACcA+wKbAcuD4TtsjgGcDWwKvAD6UZJe27TrABcAZwEbAqcAFbbkkSRqDSQsbVXVfVR1RVYuq6ndV9XXgl8ALgT2AhVX11ap6gCYA7JBk27b524Fjq+rWqroNOBbYDyDJNsBOwOFVdX9VnQNcA8xr2+4NXFhVl1bVMppAs0eSDdv6fYGjqmpJVV0LfHGob2AuTUj6dFWtqKrjgACvnPATJEnSNDVlazaSbApsAywEtgOuGqqrqvuAG9pyBuvb77t1N1bVvauo7/Z9A/AgsE2SjYDNR+n76qqqTv3VnfrB53RAkgVJFixevHjkJy9J0hpkSsJGkicAZwKnVtV1wExg6cBuS4Gh2YfB+qXAzHbdxnjbdutndrbH23YlVXViVc2pqjmzZ88ebhdJktY4kx42kqwFnE4zu3BgW7wMmDWw6yzg3hHqZwHL2hmH8bbt1i/rbI+3rSRJGoNJDRvtTMRJNAs151XVb9qqhTSLP4f22wB4Vlu+Un37fbdu684ajOHqu31vDawLXF9VS4A7Rul7++6dLzQLVBciSZLGZLJnNj4PPBfYtaru75SfBzw/ybwk6wGH0ayVuK6tPw04OMkWSTYHPgCcAlBV1wNXAocnWS/J7jSB4Jy27ZnArkle2oaYI4FzO2s8TgMOSbJRuyB1/6G+gfnAQ8BB7S2yQzMxl0zQ+ZAkadqbzPfZ2JLmdtUdgTuTLGsfe1fVYpq7Rz4OLAFeBOzZaX4CcCHNXSY/BS5qy4bsCcxp234CeHPbJ1W1EHgPTej4Fc16i/d22h5Osxj1JuB7wCer6uK27YM0t9zuC/waeCewW1suSZLGYMZkHaiqbqK5bXSk+m8D245QV8CH2sdw9YtoblMdqe+zgLNGqFtBEyLeOUL9FTS350qSpEfBtyuXJEm9MmxIkqReGTYkSVKvDBuSJKlXhg1JktQrw4YkSeqVYUOSJPXKsCFJknpl2JAkSb0ybEiSpF4ZNiRJUq8MG5IkqVeGDUmS1CvDhiRJ6pVhQ5Ik9cqwIUmSemXYkCRJvTJsSJKkXhk2JElSr2ZM9QAkSRqvufseNtVD0Dg4syFJknpl2JAkSb0ybEiSpF4ZNiRJUq8MG5IkqVfejSI9jjzeV9jPP+3IqR6CpNWQMxuSJKlXhg1JktQrw4YkSeqVYUOSJPXKsCFJknpl2JAkSb0ybEiSpF4ZNiRJUq8MG5IkqVeGDUmS1CvDhiRJ6pVhQ5Ik9cqwIUmSemXYkCRJvTJsSJKkXhk2JElSrwwbkiSpV4YNSZLUK8OGJEnqlWFDkiT1alLDRpIDkyxIsiLJKQN1r0pyXZLlSb6bZMtOXZIcneTu9nFMknTqt2rbLG/72Hmg772S3JTkviTnJ9m4U7dukpOT3JPkziQHD7TdMcnlbd+XJ9lxwgRwP8YAAAxQSURBVE+MJEnT2GTPbNwOfAw4uVuY5CnAucChwMbAAuDszi4HALsBOwDbA28E3t2p/wpwBbAJ8BHga0lmt31vB5wA7ANsCiwHju+0PQJ4NrAl8ArgQ0l2aduuA1wAnAFsBJwKXNCWS5KkMZjUsFFV51bV+cDdA1V7AAur6qtV9QBNANghybZt/duBY6vq1qq6DTgW2A8gyTbATsDhVXV/VZ0DXAPMa9vuDVxYVZdW1TKaQLNHkg3b+n2Bo6pqSVVdC3xxqG9gLjAD+HRVraiq44AAr5yYMyJJ0vT3eFmzsR1w1dBGVd0H3NCWr1Tfft+tu7Gq7l1FfbfvG4AHgW2SbARsPkrfV1dVdeqv7tQ/QpID2stECxYvXrzKJyxJ0pri8RI2ZgJLB8qWAhuOUL8UmNmu2xhv2279zM72eNuupKpOrKo5VTVn9uzZw+0iSdIa5/ESNpYBswbKZgH3jlA/C1jWzjiMt223fllne7xtJUnSGDxewsZCmsWfACTZAHhWW75Sfft9t27rzhqM4eq7fW8NrAtcX1VLgDtG6Xv77p0vNAtUFyJJksZksm99nZFkPWBtYO0k6yWZAZwHPD/JvLb+MJq1Ete1TU8DDk6yRZLNgQ8ApwBU1fXAlcDhbX+70wSCc9q2ZwK7JnlpG2KOBM7trPE4DTgkyUbtgtT9h/oG5gMPAQe1t8ge2JZfMsGnRpKkaWuyZzYOAe4H/hZ4W/v9IVW1mObukY8DS4AXAXt22p0AXEhzl8lPgYvasiF7AnPatp8A3tz2SVUtBN5DEzp+RbPe4r2dtofTLEa9Cfge8Mmqurht+yDNLbf7Ar8G3gns1pZLkqQxyCNvtNBEmTNnTi1YsGCqh6HVzNx9D5vqIUhaQ8w/7ciJ7jIjVTxe1mxIkqRpyrAhSZJ6ZdiQJEm9MmxIkqReGTYkSVKvDBuSJKlXhg1JktQrw4YkSeqVYUOSJPXKsCFJknpl2JAkSb2aMdUD0PSwOnymRw+fAyBJGgNnNiRJUq8MG5IkqVeGDUmS1CvDhiRJ6pVhQ5Ik9cqwIUmSemXYkCRJvTJsSJKkXhk2JElSrwwbkiSpV4YNSZLUK8OGJEnqlWFDkiT1yrAhSZJ6ZdiQJEm9MmxIkqReGTYkSVKvDBuSJKlXhg1JktQrw4YkSeqVYUOSJPVqxlQPQJosc/c9bKqHIElrJGc2JElSr5zZWE34V7kkaXXlzIYkSeqVYUOSJPXKsCFJknpl2JAkSb0ybEiSpF4ZNiRJUq8MG5IkqVeGDUmS1CvDhiRJ6pVhYwySbJzkvCT3JbkpyV5TPSZJklYXvl352HwOeBDYFNgRuCjJVVW1cGqHJUnS458zG6NIsgEwDzi0qpZV1WXAvwD7TO3IJElaPRg2RrcN8FBVXd8puwrYborGI0nSasXLKKObCSwdKFsKbDi4Y5IDgAPazWVJftbz2B7vngLcNdWDWAN4nieH53nyeK4nQU4/aqLP88VVtctwFYaN0S0DZg2UzQLuHdyxqk4ETpyMQa0OkiyoqjlTPY7pzvM8OTzPk8dzPTkm8zx7GWV01wMzkjy7U7YD4OJQSZLGwLAxiqq6DzgXODLJBkn+B/BnwOlTOzJJklYPho2xeS/wROBXwFeAv/K21zHxktLk8DxPDs/z5PFcT45JO8+pqsk6liRJWgM5syFJknpl2JAkSb0ybGhCJTkwyYIkK5KcMtXjma6SrJvkpPazeu5NckWS1031uKajJGckuSPJPUmuT/KuqR7TdJbk2UkeSHLGVI9lukoyvz3Hy9pH7+8JZdjQRLsd+Bhw8lQPZJqbAdwCvBx4EnAo8M9JtprCMU1XfwdsVVWzgDcBH0vywike03T2OeDHUz2INcCBVTWzfTyn74MZNjShqurcqjofuHuqxzKdVdV9VXVEVS2qqt9V1deBXwL+EpxgVbWwqlYMbbaPZ03hkKatJHsCvwa+M9Vj0cQybEjTQJJNaT7Hx1uye5Dk+CTLgeuAO4BvTPGQpp0ks4AjgQ9M9VjWEH+X5K4k308yt++DGTak1VySJwBnAqdW1XVTPZ7pqKreS/N5SC+leZO/FatuoUfhKOCkqrplqgeyBvgwsDWwBc17bVyYpNfZOsOGtBpLshbNu9k+CBw4xcOZ1qrqoaq6DPgD4K+mejzTSZIdgZ2Bf5jqsawJquqHVXVvVa2oqlOB7wOv7/OYfhCbtJpKEuAkYFPg9VX1myke0ppiBq7ZmGhzga2Am5uXNTOBtZM8r6p2msJxrSkKSJ8HcGZDEyrJjCTrAWvT/GexXhJDbT8+DzwX2LWq7p/qwUxHSZ6aZM8kM5OsneS1wF8Al0z12KaZE2kC3I7t4wvARcBrp3JQ01GSJyd57dD/zUn2Bl4GfLPP4/pLQBPtEODwzvbbgI8CR0zJaKapJFsC76ZZO3Bn+9cgwLur6swpG9j0UzSXTL5A88fZTcD7quqCKR3VNFNVy4HlQ9tJlgEPVNXiqRvVtPUEmrcn2BZ4iGbR825V1et7bfjZKJIkqVdeRpEkSb0ybEiSpF4ZNiRJUq8MG5IkqVeGDUmS1CvDhiRJ6pVhQ9LjSpL5ST471eNYldVhjNLjiWFD0oiSnJKkknxpmLpj2rqvT/Bh9wD+92PpoDPuSvLbJDcn+XySjcbZz37tG0xN+BilNYlhQ9JobgHemmSDoYL2Lej3AW6e6INV1X9X1b0T0NW3gc1oPnPjXcCuwPET0O9EjlFaIxg2JI3mauDnwFs6ZW8AHgDmd3dMslaSQ5PckmRFkmuS/Fmn/gdJjh1oMyvJ/Ul2b7cfcYkiyTpJjk5ya5L7kvy4/YyS0ayoqjur6taq+lfgbOA1A8c+OMnVbb+3JflSkie3dXOBLwMbdGZJjhhhjIuSHJLkhCT3tGP9m4FjbZPke0keSPKzJK9PsizJfmN4LtJqzbAhaSxOAt7Z2X4nzS/iwc87+Gvgb4APA/8fcB5wbvsR4gBnAHsm6f7fMw+4n+aDt4bzZeDlwF5tn6cCFybZYayDT7I1sAsw+Mm4vwPeB2zX9v/HwGfaun9v65bTzJBsBnxqFYd5P3ANsBNwNHBMkj9pj78Wzbn4LfBiYD+azxBad6zPQVqdGTYkjcVZwJwkz07yNJpf3KcMs98HgU9V1VlVdX1VHQb8W1sO8E/AbOAVnTZ7A1+tqgcHO0vyLJpPWX1LVV1aVTdW1WeBb9B8EN2q7NLOHNwP3AA8jyYE/F5VfbqqLqmqRVX1PeBDwFuSrNWOZ2mzW93ZPoZbvzHkX6vqs1X1i6r6DPAL4FVt3auB5wD7VtWVVfUDmnDih2FqjeALXdKoqmpJkvNoZjR+Dcyvqps7nzZLklnA5sD3B5pfBry+7efuJN+kCRjfSbIZTfD46AiH3gkI8J/dY9HMCIz2Me+XAgcATwT2p/kI8+O6OyR5Jc1Cz+cCTwLWBtYBngbcPkr/g64e2L4deGr7/bbA7VV1W6f+xzQzK9K058yGpLE6GdiXJnCcvIr9hvso6W7ZGcC8JOvRzFrcQhNIhrNW2/aPgB07j+fyyMs6w1nezjJcU1UHAesDhw5VJtmS5tLNtcCfAy/s9LnOKH0PZ/ASTfHw/7Fh+PMirREMG5LG6jvAg8BTgPMHK6vqHpq/5l8yUPUS4D872xe0X99IM8NxZlWN9Iv4Cppf1E9rg0P3cdsIbUbyUeDDSTZvt+fQhIr3V9UPqup6mpmZrgdpZjseq2uBLTrHHjq+/wdrjeALXdKYtIFge+CZVbVihN0+CXwwyV+0d18cCbwUOLbTzwPAucAhNJdJzljFMa8HzgROSfLmJFsnmZPkg0n2GOf45wML2+NCc4fNWsD7kjwzyV/QLAjtWgSsl+TVSZ6SZP3xHLPjW8DPgFOT7JDkxcDf0ywYdcZD055hQ9KYVdW97QzGSI6jCRzHAD8FdgfmVdWVA/udDuwA/KSqrh3lsO+guSPlGOA64OvAy4Cbxv8M+HvgL5NsWVVX09w9czDNzMu7eHghKwBV9e/AF4CvAItpFpCOW1X9juZcrAv8iOaOmo/TBI0HHk2f0uokI89eSpL60t66eyUwp6oun+rxSH0ybEjSJGjftOw+mss3W9HMsgR4wSrWrEjTgre+StLk2JDmfT6eDiyheffV9xs0tCZwZkOSJPXKBaKSJKlXhg1JktQrw4YkSeqVYUOSJPXKsCFJknpl2JAkSb36/wGBOU1M19eVnwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import chart_studio.plotly as py\n",
    "%matplotlib inline\n",
    "\n",
    "plt.figure(figsize=(8, 6))\n",
    "ax = plt.subplot(111)\n",
    "ax.set_title(\"Distribution of Movie Ratings\", fontsize=16)\n",
    "ax.spines[\"top\"].set_visible(False)  \n",
    "ax.spines[\"right\"].set_visible(False)  \n",
    "  \n",
    "plt.xticks(fontsize=12)  \n",
    "plt.yticks(fontsize=12)  \n",
    "  \n",
    "plt.xlabel(\"Movie Rating\", fontsize=14)  \n",
    "plt.ylabel(\"Count\", fontsize=14)  \n",
    "  \n",
    "plt.hist(df_ratings['rating'], color=\"#3F5D7D\")  \n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  train set, test set 만들기\n",
    "- ratings 데이터의 학습데이터와 테스트데이터를 분리합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "df_ratings_train, df_ratings_test= train_test_split(df_ratings,\n",
    "                                                    stratify=df_ratings['userId'],\n",
    "                                                    random_state = 15688,\n",
    "                                                    test_size=0.30)\n",
    "\n",
    "\n",
    "print(\"Number of training data: \"+str(len(df_ratings_train)))\n",
    "print(\"Number of test data: \"+str(len(df_ratings_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "user의 평가 데이터를 기준으로 데이터를 구분하여 영화정보를 추출합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rating_splitter(df):\n",
    "    \n",
    "    df['liked'] = np.where(df['rating']>=4, 1, 0) # df['rating']이 4이상이면 1 아니면 0\n",
    "    df['movieId'] = df['movieId'].astype('str')\n",
    "    gp_user_like = df.groupby(['liked', 'userId']) # liked와 user_id로 그룹화\n",
    "    return ([gp_user_like.get_group(gp)['movieId'].tolist() for gp in gp_user_like.groups]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "즉, user 정보는 무시하고 movieId만 반환합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.mode.chained_assignment = None\n",
    "splitted_movies = rating_splitter(df_ratings_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## 모델 생성 및 학습"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(action='ignore', category=UserWarning, module='gensim')\n",
    "\n",
    "import gensim\n",
    "assert gensim.models.word2vec.FAST_VERSION > -1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "276252"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(splitted_movies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_splitted_movies = splitted_movies[:1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shuufle train data\n",
    "- 영화의 순서는 의미있는 정보가 아니므로 학습데이터를 셔플링합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "for movie_list in new_splitted_movies:\n",
    "    random.shuffle(movie_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "모델을 생성합니다.\n",
    "- Skip-gram을 사용한 Word2Vec 모델을 생성합니다.\n",
    "- 각 파라미터를 다르게 설정하여 2가지 모델을 만들었습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time passed: 0:00:15.709966\n"
     ]
    }
   ],
   "source": [
    "\n",
    "from gensim.models import Word2Vec\n",
    "import datetime\n",
    "start = datetime.datetime.now()\n",
    "\n",
    "model = Word2Vec(sentences = new_splitted_movies, # We will supply the pre-processed list of moive lists to this parameter\n",
    "                 iter = 5, # epoch\n",
    "                 min_count = 10, # a movie has to appear more than 10 times to be keeped\n",
    "                 size = 200, # size of the hidden layer\n",
    "                 workers = 4, # specify the number of threads to be used for training\n",
    "                 sg = 1, # Defines the training algorithm. We will use skip-gram so 1 is chosen.\n",
    "                 hs = 0, # Set to 0, as we are applying negative sampling.\n",
    "                 negative = 5, # If > 0, negative sampling will be used. We will use a value of 5.\n",
    "                 window = 9999999)\n",
    "\n",
    "print(\"Time passed: \" + str(datetime.datetime.now()-start))\n",
    "model.save('item2vec_20200908')\n",
    "del model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import Word2Vec\n",
    "import datetime\n",
    "start = datetime.datetime.now()\n",
    "\n",
    "model_w2v_sg = Word2Vec(sentences = splitted_movies,\n",
    "                        iter = 10, # epoch\n",
    "                        min_count = 5, # a movie has to appear more than 5 times to be keeped\n",
    "                        size = 300, # size of the hidden layer\n",
    "                        workers = 4, # specify the number of threads to be used for training\n",
    "                        sg = 1,\n",
    "                        hs = 0,\n",
    "                        negative = 5,\n",
    "                        window = 9999999)\n",
    "\n",
    "print(\"Time passed: \" + str(datetime.datetime.now()-start))\n",
    "model_w2v_sg.save('item2vec_word2vecSg_20200908')\n",
    "del model_w2v_sg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "생성한 모델을 load합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(action='ignore', category=UserWarning, module='gensim')\n",
    "\n",
    "from gensim.models import Word2Vec\n",
    "model = Word2Vec.load('item2vec_20200908')\n",
    "word_vectors = model.wv\n",
    "# del model # uncomment this line will delete the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "## Let's Make Some Recommendations!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import re\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "def refine_search(search_term):\n",
    "    \"\"\"\n",
    "    Refine the movie name to be recognized by the recommender\n",
    "    Args:\n",
    "        search_term (string): Search Term\n",
    "\n",
    "    Returns:\n",
    "        refined_term (string): a name that can be search in the dataset\n",
    "    \"\"\"\n",
    "    target_url = \"http://www.imdb.com/find?ref_=nv_sr_fn&q=\"+\"+\".join(search_term.split())+\"&s=tt\"\n",
    "    \n",
    "    # 크롤링\n",
    "    html = requests.get(target_url).content\n",
    "    parsed_html = BeautifulSoup(html, 'html.parser')\n",
    "    for tag in parsed_html.find_all('td', class_=\"result_text\"):\n",
    "        print(tag)\n",
    "        title = tag.find('a').get_text()\n",
    "        year = re.findall('</a>(.*)</td>', str(tag))\n",
    "\n",
    "        return title + ' ' + year[0].strip()\n",
    "        print(search_result)\n",
    "        refined_name = \"\"\n",
    "        if search_result:\n",
    "            if search_result[0][0].split()[0]==\"The\":\n",
    "                str_frac = \" \".join(title.split()[1:])+\", \"+title.split()[0]\n",
    "                refined_name = title + ' ' + year[0].strip()\n",
    "            else:\n",
    "                refined_name = title + ' ' + year[0].strip()\n",
    "        return refined_name\n",
    "\n",
    "def produce_list_of_movieId(list_of_movieName, useRefineSearch=False):\n",
    "    \"\"\"\n",
    "    Turn a list of movie name into a list of movie ids. The movie names has to be exactly the same as they are in the dataset.\n",
    "    Ambiguous movie names can be supplied if useRefineSearch is set to True\n",
    "    \n",
    "    Args:\n",
    "        list_of_movieName (List): A list of movie names.\n",
    "        useRefineSearch (boolean): Ambiguous movie names can be supplied if useRefineSearch is set to True\n",
    "\n",
    "    Returns:\n",
    "        list_of_movie_id (List of strings): A list of movie ids.\n",
    "    \"\"\"\n",
    "    list_of_movie_id = []\n",
    "    for movieName in list_of_movieName:\n",
    "        if useRefineSearch:\n",
    "            movieName = refine_search(movieName)\n",
    "            print(movieName)\n",
    "            print(\"Refined Name: \"+movieName)\n",
    "        if movieName in name_to_movieId.keys():\n",
    "            list_of_movie_id.append(str(name_to_movieId[movieName]))\n",
    "    return list_of_movie_id\n",
    "\n",
    "def recommender(positive_list=None, negative_list=None, useRefineSearch=False, topn=20):\n",
    "    recommend_movie_ls = []\n",
    "    if positive_list:\n",
    "        positive_list = produce_list_of_movieId(positive_list, useRefineSearch)\n",
    "    if negative_list:\n",
    "        negative_list = produce_list_of_movieId(negative_list, useRefineSearch)\n",
    "    for movieId, prob in model.wv.most_similar_cosmul(positive=positive_list, negative=negative_list, topn=topn):\n",
    "        recommend_movie_ls.append(movieId)\n",
    "    return recommend_movie_ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "recommender\n",
    "- user로부터 'UP'이 검색되면 해당 검색어를 크롤링을 통해 dataset의 title값을 받아옵니다.\n",
    "- 이후 해당 movie title 리스트를 앞에서 인코딩한 name_to_movieId 사전을 통해 movieId로 바꿔줍니다.\n",
    "- 그리고 앞에서 정의한 모델을 통해 top-n 개의 similar한 데이터를 반환합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<td class=\"result_text\"> <a href=\"/title/tt1049413/\">Up</a> (2009) </td>\n",
      "Up (2009)\n",
      "Refined Name: Up (2009)\n",
      "Recommendation Result based on \"Up (2009)\":\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4794</th>\n",
       "      <td>4890</td>\n",
       "      <td>Shallow Hal (2001)</td>\n",
       "      <td>Comedy|Fantasy|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5574</th>\n",
       "      <td>5673</td>\n",
       "      <td>Punch-Drunk Love (2002)</td>\n",
       "      <td>Comedy|Drama|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7318</th>\n",
       "      <td>7444</td>\n",
       "      <td>13 Going on 30 (2004)</td>\n",
       "      <td>Comedy|Fantasy|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11352</th>\n",
       "      <td>48385</td>\n",
       "      <td>Borat: Cultural Learnings of America for Make ...</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12089</th>\n",
       "      <td>54997</td>\n",
       "      <td>3:10 to Yuma (2007)</td>\n",
       "      <td>Action|Crime|Drama|Western</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       movieId                                              title  \\\n",
       "4794      4890                                 Shallow Hal (2001)   \n",
       "5574      5673                            Punch-Drunk Love (2002)   \n",
       "7318      7444                              13 Going on 30 (2004)   \n",
       "11352    48385  Borat: Cultural Learnings of America for Make ...   \n",
       "12089    54997                                3:10 to Yuma (2007)   \n",
       "\n",
       "                           genres  \n",
       "4794       Comedy|Fantasy|Romance  \n",
       "5574         Comedy|Drama|Romance  \n",
       "7318       Comedy|Fantasy|Romance  \n",
       "11352                      Comedy  \n",
       "12089  Action|Crime|Drama|Western  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ls = recommender(positive_list=[\"UP\"], useRefineSearch=True, topn=5)\n",
    "print('Recommendation Result based on \"Up (2009)\":')\n",
    "display(df_movies[df_movies['movieId'].isin(ls)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## Evaluating the Model Performance\n",
    "- 모델의 성능을 평가합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def user_liked_movies_builder(model, df, for_prediction=False):\n",
    "    df['liked'] = np.where(df['rating']>=4, 1, 0)\n",
    "    df['movieId'] = df['movieId'].astype('str')\n",
    "    df_liked = df[df['liked']==1]\n",
    "    if for_prediction:\n",
    "        df_liked = df[df['movieId'].isin(model.wv.vocab.keys())]\n",
    "        \n",
    "    user_liked_movies = df_liked.groupby('userId').agg({'movieId': lambda x: x.tolist()})['movieId'].to_dict()\n",
    "    \n",
    "    return user_liked_movies\n",
    "\n",
    "def scores_at_m (model, user_liked_movies_test, user_liked_movies_training, topn=10):\n",
    "    sum_liked = 0\n",
    "    sum_correct = 0\n",
    "    sum_total = 0\n",
    "    common_users = set(user_liked_movies_test.keys()).intersection(set(user_liked_movies_training.keys()))\n",
    "\n",
    "    for userid in common_users:\n",
    "        current_test_set = set(user_liked_movies_test[userid])\n",
    "        pred = [pred_result[0] for pred_result in model.wv.most_similar_cosmul(positive = user_liked_movies_training[userid], topn=topn)]\n",
    "        sum_correct += len(set(pred).intersection(current_test_set))\n",
    "        sum_liked += len(current_test_set)\n",
    "    precision_at_m = sum_correct/(topn*len(common_users))\n",
    "    recall_at_m = sum_correct/sum_liked\n",
    "    f1 = 2/((1/precision_at_m)+(1/recall_at_m))\n",
    "    return [precision_at_m, recall_at_m, f1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.mode.chained_assignment = None\n",
    "user_liked_movies_train = user_liked_movies_builder(model, df_ratings_train, for_prediction=True)\n",
    "user_liked_movies_test = user_liked_movies_builder(model, df_ratings_test)\n",
    "\n",
    "model = Word2Vec.load('item2vec_20200908')\n",
    "model_score_sg1 = scores_at_m(model, user_liked_movies_test, user_liked_movies_train)\n",
    "del model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Respectively, the [precision, recall, F-1 score] at 10 for our model are:\n",
      "[0.08375905229567676, 0.03835838932533997, 0.05261922121575363]\n"
     ]
    }
   ],
   "source": [
    "print(\"Respectively, the [precision, recall, F-1 score] at 10 for our model are:\")\n",
    "print(model_score_sg1)  ## congrats!! haha bylu! yating"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}