Skip to content
This repository has been archived by the owner on Sep 20, 2024. It is now read-only.

Implement collections/sources for P5 - P17 (with the scraper transformers) #181

Open
7 tasks
osahon-okungbowa opened this issue Jun 15, 2020 · 0 comments
Open
7 tasks

Comments

@osahon-okungbowa
Copy link
Contributor

Description

datasets belong to Collections, while Collections belong to Sources. We have implemented Collections and Sources for that data portal using collections transformer and sources transformer respectively.

We have implemented & tested collections & sources for P1-P4. Now, we need to do the same for P5-P17.

Tasks

  • run the scraped output from P5-P17 through the collections transformer. Ensure the scraped output can be processed by the transformer without generating errors/breaking things
  • run the scraped output from P5-P17 through the sources transformer. Ensure the scraped output can be processed by the transformer without generating errors/breaking things
  • run the datajson transformer for P5-P17. Ensure a datajson file can be generated from the collections/sources json output
  • if there are any errors, identify them and fix without breaking the transformer pipeline

Acceptance Criteria

  • the collections/sources transformers work well and generate output for P5-P17
  • the output provided by the collections/sources transformers is compatible with the datajson transformer
  • the output structure of the datajson transformer still remains valid i.e. it conforms to the agreed schema
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant