You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
local airflow with load code for local csv converting to json. Part of Epic #1
Acceptance
Parameterized task on examples/folder; receives a csv file and outputs a pretty printed json file
Tasks
Define airflow operator (PythonOperator for now)
Libraries - converting with pandas
Specify in/out directories in kwargs
Json Pretty Printing
wrap up DAG code
Analysis
Questions
How to deal with invalid CSV files? This may deserve another issue on the tracker
~rufus definitely deserves another item. Here you assume the CSV's are "tidy/good"
tests? (unsure how to do them for this example case)
~rufus - what are you testing? there's the core lib for processing and airflow itself. For airflow itself it would be more UAT like e.g. calling the API and checking there's some output on disk
Formatting scripts/libraries? Ideally set them up with tests (i.e. run make test and also get feedback from formatting)
~rufus: can you clarify what you mean by formatting scripts / lib.
The text was updated successfully, but these errors were encountered:
local airflow with load code for local csv converting to json. Part of Epic #1
Acceptance
Tasks
Json Pretty PrintingAnalysis
Questions
The text was updated successfully, but these errors were encountered: