Skip to content

stanford-crfm/halie

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

An Interactive Evaluation Framework for Human-Language Model Interaction

Overview

This repository contains the code for HALIE (Human-AI Language-based Interaction Evaluation), a new framework for evaluating human-LM interaction. It is designed to be flexible and extensible to support a variety of interactive tasks beyond the five tasks we considered in our paper: social dialogue, question answering, crossword puzzles, text summarization, and metaphor generation. All the code for the five tasks are included in this repository.

If you have any questions, please reach out to Mina Lee at [email protected]. For scenario-specific questions or feedback (e.g. task design, data issues), please include the following authors:


Contents


Data

See README for a more comprehensieve overview of data in HALIE.

Standardized data. If you simply want to take a look at our data or use it to perform your own analysis, you can find the standardized data for the five tasks in ./data/std.

Raw data. On the other hand, if you want to standardize data from raw data we collected (./data/raw), you can follow the steps below to convert the raw data.

First, install the required packages:

pip3 install -r requirements.txt

Then, run the following command to standardize the logs (e.g., for question answering):

python3 ./src/run_question.py

The above command reads the raw data at ./data/raw/question and saves the standardized data at ./data/std/question. For the other four tasks, replace question with the name of the task you want to standardize in the path as well as command.

Visualizations. Static and dynamic visualizations of our data, which allows for an easier way for looking at raw interaction traces, is included at ./data/visualizations.

Your data. If you are interested in extending HALIE to support a new task, please create a PR or contact Mina Lee at [email protected].


Analysis

We provide Jupyter Notebook files for analyzing the data collected for HALIE. The files are stored in ./notebook.


Interfaces

We currently provide code for the interfaces used to collect interaction traces for both the crossword puzzles and question answering scenarios in HALIE, located in ./interfaces.

We unfortunately do not currently release code for the text summarization, metaphor generation, and social dialogue scenarios. Those scenarios were built using code from CoAuthor and Chirpy Cardinal, and please feel free to create a PR for any information.

About

No description, website, or topics provided.

Resources

License

Apache-2.0, Apache-2.0 licenses found

Licenses found

Apache-2.0
LICENSE
Apache-2.0
LICENSE.md

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published