Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADD] an module for scapers #1

Open
ruohoruotsi opened this issue Mar 21, 2019 · 2 comments
Open

[ADD] an module for scapers #1

ruohoruotsi opened this issue Mar 21, 2019 · 2 comments
Assignees
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed

Comments

@ruohoruotsi
Copy link
Member

Add a module that does Yorùbá TS website scraping. Right now candidates are:

@ruohoruotsi ruohoruotsi added the enhancement New feature or request label Mar 21, 2019
@ruohoruotsi ruohoruotsi self-assigned this Mar 21, 2019
@ruohoruotsi ruohoruotsi added good first issue Good for newcomers help wanted Extra attention is needed labels Jun 19, 2019
@ruohoruotsi ruohoruotsi assigned Olamyy and unassigned ruohoruotsi Jul 20, 2019
@Olamyy
Copy link
Contributor

Olamyy commented Jul 21, 2019

While thinking about the best way to implement this, I came up with a design and would like to get opinions about it.

Since we want the crawler to be reusable, I'm proposing a minimal https://docs.scrapy.org app that contains:

  1. A spider : BaseCrawler that inherits from scrappy.Spider.
  2. A settings for handling scrapy configs
  3. An item item file for handling ordered saves.

The BaseCrawler from 1 above can implement a yaml/json loader for any other person to subclass to run create their own config/logic.

I imagine this is a bit confusing right now but I can come up with a simple demo of my thoughts and share it for opinions here when I'm done.

@realsama
Copy link

realsama commented Oct 13, 2019

@ruohoruotsi so i see there are plans for a scraper already, what is the progress so far cause am thinking of one that scrapes bbc yoruba although it can be sub classed and tuned for other blogs and websites generally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

3 participants