Initial commit

.gitignore

@@ -0,0 +1,230 @@
+
+# Created by https://www.gitignore.io/api/c,node,python
+
+### C ###
+# Prerequisites
+*.d
+
+# Object files
+*.o
+*.ko
+*.obj
+*.elf
+
+# Linker output
+*.ilk
+*.map
+*.exp
+
+# Precompiled Headers
+*.gch
+*.pch
+
+# Libraries
+*.lib
+*.a
+*.la
+*.lo
+
+# Shared objects (inc. Windows DLLs)
+# *.dll
+# *.so
+# *.so.*
+# *.dylib
+
+# Executables
+*.exe
+*.out
+# *.app
+*.i*86
+*.x86_64
+*.hex
+
+# Debug files
+*.dSYM/
+*.su
+*.idb
+*.pdb
+
+# Kernel Module Compile Results
+*.mod*
+*.cmd
+.tmp_versions/
+modules.order
+Module.symvers
+Mkfile.old
+dkms.conf
+
+### Node ###
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# Runtime data
+pids
+*.pid
+*.seed
+*.pid.lock
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+lib-cov
+
+# Coverage directory used by tools like istanbul
+coverage
+
+# nyc test coverage
+.nyc_output
+
+# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
+.grunt
+
+# Bower dependency directory (https://bower.io/)
+bower_components
+
+# node-waf configuration
+.lock-wscript
+
+# Compiled binary addons (http://nodejs.org/api/addons.html)
+build/Release
+
+# Dependency directories
+node_modules/
+jspm_packages/
+
+# Typescript v1 declaration files
+typings/
+
+# Optional npm cache directory
+.npm
+
+# Optional eslint cache
+.eslintcache
+
+# Optional REPL history
+.node_repl_history
+
+# Output of 'npm pack'
+*.tgz
+
+# Yarn Integrity file
+.yarn-integrity
+
+# dotenv environment variables file
+.env
+
+
+### Python ###
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+
+# Distribution / packaging
+.Python
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+# lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Application-specific files
+aim_backend/configs/*.json
+aim_backend/data/results/*.png
+aim_backend/data/screenshots/*.png
+aim_backend/inputs/*.txt
+aim_backend/screenshots/*.png
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+.pytest_cache/
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+
+# Translations
+*.mo
+*.pot
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule.*
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+
+
+# End of https://www.gitignore.io/api/c,node,python
+
+# Mac OS
+.DS_Store
+
+# IDEs
+.idea/

CONTRIBUTING.md

@@ -0,0 +1,32 @@
+# AIM Contributing Guide
+
+We are really excited that you are interested in contributing to AIM! Before submitting your contribution though, please make sure to take a moment and read through the following instructions and guidelines.
+
+
+## Extending AIM with New Metrics
+
+AIM has been designed from the ground-up with extensibility in mind. As a result, new metrics can be added with relatively small effort, but there are a few steps that need to be followed.
+
+First, the new metric should be defined in a separate Python file and placed under a proper subdirectory within the [aim_metrics](./aim_metrics/aim_metrics/) directory. The entry point to the metric must be a function called `execute`, and it must take as a parameter a base64 encoded image (PNG or JPG) representation of the web page. Segmentation-based metrics must take as an additional parameter to this a list of page elements, generated by the segmentation script. The `execute` function must return a list of results, which must have at least one entry. The result entries must be *int*, *float*, or a *base64* encoded image representation of the web page.Metrics are executed on the backend by the [metric_executor.py](./aim_backend/uimetrics_backend/metric_executor.py) file. The metric must be imported here, and it must to be registered to the list of executable metrics. Each list entry has the metric ID as key and a dictionary of options as value. The dictionary specifies the type of the metric (`png` for pixel-based, `jpg` for pixel-based, or `seg` for segmentation-based) and the imported metric module itself.In addition to the backend, the metric must be registered on the frontend too to show up on the list of metrics. This is done in the [metrics.js](./aim_frontend/src/config/metrics.js) file. Here, a new entry must be added in the JSON under the `metrics` key. Each entry’s key is its metric ID.   **Table 1.** Description of `metrics` entry in `metrics.js`
+| Key               | Description |
+|:------------------|:------------|
+| id                | Metric ID   |
+| name              | Metric name. Will be displayed in list of metrics on the frontend |
+| category          | Metric category ID |
+| description       | Description of the metric |
+| evidence          | 1-5 `int`; rating of evidence for this metric || relevance         | 1-5 `int`; rating of relevance for this metric |
+| speed             | 0-2 `int`; 0=slow, 1=medium, 2=fast, will be displayed as ‘Computation time’ |
+| visualizationType | `table` or `b64`, table=numerical results, b64=image results || references        | List of `references` entries specifying references, cf. description below || results           | List of `results` entries specifying results, cf. description below |   **Table 2.** Description of `references` entry in `metrics.js`
+
+| Key      | Description |
+|:---------|:------------|| title    | Reference title, will be shown in a tooltip when hovering over the reference number || fileName | Name of the PDF file of the reference paper |
+
+   **Table 3.** Description of `results` entry in `metrics.js`
+| Key         | Description ||:------------|:------------|
+| id          | Result ID, format is metric ID + underscore + index in the array returned by the metric || index       | Index in metric result array || type        | `int`, `float`, or `b64` || name        | Result name || description | Optional description of the result. `false` if there is no description |Every metric also must be added under a proper `category` at the start of the same file. A new category can also be created.
+
+   **Table 4.** Description of `categories` entry in `metrics.js`
+
+| Key     | Description |
+|:--------|:------------|| name    | Category name || id      | Category ID || color   | Name of the CSS class in `AIMForm.vue` specifying the category color || metrics | List of metric IDs belonging to this category |
+After adding a new metric, the frontend must be restarted with `npm run dev` for development or recompiled with `npm run build` for production (executed in the [aim_frontend](./aim_frontend/) directory). For the backend, the AIM metrics package must be reinstalled with `pip install ../aim_metrics` and the server must be relaunched with `python uimetrics_backend/main.py` (executed in the [aim_backend](./aim_backend/) directory).

LICENSE.txt

@@ -0,0 +1,10 @@
+MIT License
+
+
+Copyright © 2018 User Interfaces group, Aalto University, Finland
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

README.md

@@ -1,2 +1,101 @@
-# aim
-Code repository for the Aalto Interface Metrics (AIM) service. https://interfacemetrics.aalto.fi/
+![interfacemetrics.aalto.fi](./aim_frontend/src/assets/workflow.png)
+
+# AIM – Aalto Interface Metrics
+
+AIM is an online service and an open code repository for *computational evaluation* of graphical user interface (GUI) designs. AIM pools several previously published metrics and models, which have been empirically shown to be predictive of how users perceive, search, and aesthetically experience a  design. These metrics range from design heuristics like symmetry to metrics and full-fledged models, such as saliency and visual clutter.
+
+More information on AIM can be found in the [paper](./aim_frontend/static/publications/oulasvirta_et_al_2018.pdf) published in [UIST'18](http://uist.acm.org/uist2018/).
+
+
+## Architecture and Technologies
+
+AIM's codebase is divided into four distinct parts:
+
+* [Frontend - web application](./aim_frontend/)
+* [Backend - web application](./aim_backend/)
+* [Metrics library](./aim_metrics/)
+* [Segmentation script](./aim_segmentation/)
+
+The web application's frontend is built with [Vue.js](https://vuejs.org/), whereas the backend is based on [Tornado](http://www.tornadoweb.org/). The frontend and the backend communicate with each other via [WebSocket](https://tools.ietf.org/html/rfc6455).
+
+### Depencencies
+
+The metrics library and the segmentation script are both dependencies for the web application's backend. Other dependencies include [Node.js](https://nodejs.org/) + [npm](https://www.npmjs.com/) (frontend), [Python 2.7](https://www.python.org/) + [pip](https://pypi.org/project/pip/) (backend), and [MongoDB](https://www.mongodb.com/) (database). In addition, the backend depends on [Headless Chrome](https://www.google.com/chrome/) and [layout-learning](./aim_backend/bin/layout-learning). The former is used to capture web page screenshots and the latter offers an implementation for the visual search performance metric (needs to be compiled under the target platform, only Linux is available for now).
+
+
+## Installation
+
+### Frontend
+
+Start by configuring frontend environment variables for [development](./aim_frontend/config/dev.env.js), [test](./aim_frontend/config/test.env.js), and [production](./aim_frontend/config/prod.env.js). Then, go to the [aim_frontend](./aim_frontend/) directory and run the following command:
+
+```bash
+# Install required packages
+npm install
+```
+
+### Backend
+
+Configure backend environment variables for [development](./aim_backend/configs/development.conf), [test](./aim_backend/configs/test.conf), and [production](./aim_backend/configs/productions.conf). Next, go to the [aim_backend](./aim_backend/) directory, and create and activate a new Python virtual environment (if needed). Run the following commands:
+
+```bash
+# Install required packages
+pip install ../aim_metricspip install ../aim_segmentation
+pip install -r requirements.txtpip install opencv-python
+```
+
+### Database
+
+Create `aim` database in MongoDB with the following two collections under it: `screenshots` and `results`.
+
+
+## Usage
+
+### Frontend
+
+Go to the [aim_frontend](./aim_frontend/) directory first, and then run the following command:
+
+```bash
+# Serve with hot reload at localhost:8080
+npm run dev
+```
+To build the frontend for production:
+
+```bash
+# Build for production with minification
+npm run build
+```
+
+After the build is complete, the files (for production) can be found under the *dist* directory. These files are meant to be served over an HTTP server, such as [Apache HTTP Server](https://httpd.apache.org/).
+
+### Backend
+
+To run the backend server, go the [aim_backend](./aim_backend/) directory and execute:
+
+```bash
+# Start the serverpython uimetrics_backend/main.py
+```
+
+It is highly recommended to use a load balancer in a production environment, as certain metrics are extremely CPU intensive. This means that the backend needs to be launched with multiple instances, each listening to a different port. A process manager (e.g., [pm2](http://pm2.keymetrics.io/)) will come in handy at that point. Also, define the AIM_ENV environment variable on the production server and set its value to "production" (defaults to "development").
+
+
+## Contributing
+
+Please read [CONTRIBUTING.md](./CONTRIBUTING.md) for details on contributing to AIM.
+
+
+## Changelog
+
+Detailed changes for each release are documented in the [release notes](https://github.com/aalto-ui/aim/releases).
+
+
+## Contact
+
+For questions and further information, please contact us via email at <[email protected]>.
+
+
+## License
+
+This project is licensed under the [MIT License](https://opensource.org/licenses/MIT), see the [LICENSE.txt](./LICENSE.txt) file for details.
+
+Copyright © 2018 [User Interfaces group](https://userinterfaces.aalto.fi/), [Aalto University](https://www.aalto.fi/), Finland

aim_backend/configs/development.conf

@@ -0,0 +1,7 @@
+port=8888
+name="aim-dev"
+chrome_path="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
+layout_learning_path="bin/layout-learning"
+database_uri="mongodb://<USERNAME>:<PASSWORD>@localhost:27017/aim"
+results_data_dir="data/results"
+screenshots_data_dir="data/screenshots"