Data API: Need and Design

[rufuspollock]

Main page: https://datahub.io/notes/design-data-api

• What needs to people have
• What features do people want?
• What tool should we for this?

Job Stories

My rough use case is: i want to push a dataset and get an API i can share with others for them to use.

Bonus: i can meter it and charge for it!

Tools to use

• datasette run in a worker
• RoAPI
• SeaFowl

[rufuspollock]

RoAPI looks really cool

🔗 https://github.com/roapi/roapi

ROAPI automatically spins up read-only APIs for static datasets without requiring you to write a single line of code. It builds on top of Apache Arrow and Datafusion. The core of its design can be boiled down to the following.

Comments from @davidgasquez

This is probably the most straightforward way. Point RoAPI to your slowly changing dataset and have a bunch of API ready. Could be served via Cloud Run/Lambda to reduce costs.

[davidgasquez]

Deploying ROAPI shouldn't be hard (e.g: Docker image on Cloud Run).

Regarding deployment, I can think of two modes. Have 1 instance with all the desired datasets or 1 instance per dataset (similar to what Seafowl advocates for).

Configuration wise, what are you looking to tweak?

[rufuspollock]

@davidgasquez and how would that work out pricing wise? I think the charge is per request so it doesn't matter how many deployments you have (IIRC) ...

Would be worth doing a prototype of this i think!

[davidgasquez]

I think the charge is per request so it doesn't matter how many deployments you have (IIRC) ...

Depending where the deployment is made, having multiple ones might be harder to manage and more expensive. Would also have to pay the cold start times for each instance.

Would be worth doing a prototype of this i think!

https://github.com/datopian/datahub-next/issues/96

[davidgasquez]

Managed to deploy a data API using ROAPI via HuggingFace Spaces (free docker setup that scales to 0).

https://huggingface.co/spaces/datonic/API/tree/main

The repo is simple and works with the same hf datasets!

Some examples:

❯ curl -s "https://datonic-api.hf.space/api/schema"  | jq keys
[
  "spain_aemet_historical_weather",
  "spain_energy_demand",
  "spain_ipc",
  "spain_water_reservoirs_data",
  "threatened_animal_species",
  "wikidata_asteroids"
]

Run SQL

❯ curl -s -X POST -d "select * from spain_energy_demand order by datetime desc limit 10" https://datonic-api.hf.space/api/sql | jq .
[
  {
    "datetime": "2024-06-19T20:20:00",
    "percentage": 0.3315928078459862,
    "value": 29212.0
  },
  {
    "datetime": "2024-06-19T20:15:00",
    "percentage": 0.3309668464286527,
    "value": 29090.0
  },
.
.
.

Also URL encoded!

https://datonic-api.hf.space/api/tables/wikidata_asteroids?limit=2

---

The initial list of datasets could be parametrized or derived at runtime with a previous script that scrapes an organization or repo for datapackages and publishes them.

[rufuspollock]

@davidgasquez Super cool. Definite micro-post material.

[rufuspollock]

SeaFowl

🔗 https://seafowl.io/

With Seafowl, datasets could be packaged and deployed on the edge (CF Workers, Fly.io, Serverless platform, ...). They've figured out things like caching and CDNs. They have a great end to end example showcasing some interesting features.

[davidgasquez]

Spotted Radiant Earth's new Source Cooperative initiative.

Source Cooperative is a neutral, non-profit data-sharing utility that allows trusted organizations to share data without purchasing a data portal SaaS subscription or managing infrastructure. Source allows organizations to share data using standard HTTP methods rather than requiring proprietary APIs or SaaS interfaces. It is currently in private beta.

Their demo is worth watching too!

I think this is an awesome approach to distributing datasets. Paired with things like WASM DuckDB, you get beautiful and useful websites all served from static assets (the browser does the HTTP range requests in real time!).

I've been applying the same ideas on a recent project and wanted to surface this approach (Parquet files on S3) as an easy step between no API/ CSV and REST/GraphQL API.

[rufuspollock]

We did experiments with range request on csv back in 2018 or so with datahub.io. I really like the idea.