(New Office) Office of Inspector General: create scraper for this office (Phase 1)

[higorspinto]

Description

The Office of Inspector General is among the list of new offices whose datasets need to be ingested into the data portal. For this to happen, we need to create a new scraper to crawl/parse the available webpages of the office.

https://www2.ed.gov/about/offices/list/oig/index.html

Acceptance Criteria

• We have a functional crawler that crawls through the webpages of the offices
• We have a functional parser that understands the page structures and generates structured data
• Datasets are produced when the scraper is run

Tasks

• Identify the possible page structures in the target site
• Write one or multiple parsers that cover as many cases as possible
• Test if it runs well within the pipeline

Jira Card