Skip to content

Commit

Permalink
docs: πŸ“ update docs
Browse files Browse the repository at this point in the history
  • Loading branch information
davidgasquez committed Jul 6, 2023
1 parent f805b8b commit 03f872d
Show file tree
Hide file tree
Showing 8 changed files with 112 additions and 26,305 deletions.
35 changes: 12 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,46 +11,35 @@

<br>

Datadex links together tools and frameworks with the goal to allow everyone collaborate on Open Data like people collaborate on Open Source using the principles from the Open Data Stack.

With Datadex and the help of tools like [dbt](https://docs.getdbt.com/docs/introduction) and [DuckDB](https://duckdb.org/) you can start modeling data by writing simple `select` statements!
Datadex is a platform where people collaborate on Open Data using modern and open source tools and frameworks.

### πŸ’‘ Principles

- **Open**: Play well with the rest of the ecosystem.
- **Open**: Play well with the ecosystem. Use open standards and share data in open formats.
- **Modular**: Each component can be replaced, extended, or removed. Works well in many environments (your laptop, in a cluster, or from the browser), and with multiple tools (thanks to the Arrow ecosystem).
- **Permissionless**. Don't ask, fork it and improve the models or scripts!
- **Data as Code**. Declarative stateless transformations tracked in `git`. Version your data as code! Publish and share your reusable models for others to build on top. Data, reproducible and accessible!
- **Modern**: Supports types, tests, materialized views, and more. Datadex isn't a new tool, it is a bridge between existing ones.

## πŸ’» Usage

This is an example of how you can use Datadex to model data, which is already configured with some sample datasets. Get things working end to end with the following steps:

1. Setup dependencies with `make deps`.
1. Build your dbt models and save them to Parquet files with `make run`.
1. Explore the data with `make rill`.

![](https://user-images.githubusercontent.com/1682202/195888267-ab119222-9269-4e00-98a9-8cf3a6405252.png)
- **Permissionless**. Don't ask, fork it and improve the models, add a new source or update any script.
- **Data as Code**. Declarative stateless transformations tracked in `git`. Version your data as code! Publish and share your reusable models for others to build on top.
- **Modern**: Supports types, tests, materialized views, and more.

## πŸš€ What can you do with Datadex?
Datadex isn't a new tool, it is an opinionated bridge between existing ones.

- Model local and remote datasets with `dbt`.
- Use any of the other awesome `dbt` features like `tests` and `docs`. [Docs are automatically generated and published on GitHub Pages](https://davidgasquez.github.io/datadex/docs).
### πŸš€ What can you do with Datadex?

![](https://user-images.githubusercontent.com/1682202/195890290-a27498dd-1d7b-4613-ba9a-4848fb3001be.png)
- Model [existing datasets using `dbt` and `SQL` like you would do in your company](dbt/models/climate/sources.yml). You can use and abuse any of the other awesome `dbt` features like `tests` and `docs` ([automatically generated and published on GitHub Pages](https://davidgasquez.github.io/datadex/docs)).
- Add new sources. Use Dagster, dbt Python models, or dbt Plugins to bring data locally and work with it.
- Explore your data with Rill or Jupyter Notebooks. [Share your findings with others](https://davidgasquez.github.io/datadex/notebooks/quarto.html) by publishing your notebooks (Quarto) or using Evidence.

## βš™οΈ Setup

The fastest way to start using Datadex is via [VSCode Remote Containers](https://code.visualstudio.com/docs/remote/containers). Once inside the develpment environment, you'll only need to run `make deps`.

[![](https://github.com/codespaces/badge.svg)](https://codespaces.new/davidgasquez/datadex)

PS: The development environment can also run in your browser thanks to GitHub Codespaces.
PS: The development environment can also run in your browser thanks to GitHub Codespaces!

## 🎯 Motivation

This small project was created after [thinking how an Open Data Protocol could look like](https://publish.obsidian.md/davidgasquez/Open+Data)! I just wanted to stitch together a few open source technologies and see what could they do.
This small project was created after [thinking how an Open Data Protocol could look like](https://publish.obsidian.md/davidgasquez/Open+Data)!

## πŸ‘ Acknowledgements

Expand Down
2 changes: 1 addition & 1 deletion dbt/packages.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
packages:
- package: dbt-labs/codegen
version: 0.9.0
version: 0.10.0
Loading

0 comments on commit 03f872d

Please sign in to comment.