Skip to content

A NodeJS program created to scrape URLs from dynamically rendered web apps without the need of spawning headless browsers.

Notifications You must be signed in to change notification settings

haydencarlson/site-scraper-node

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Welcome to site-scraper-node 👋

Version License: ISC Twitter: itsheydoon

A NodeJS program created to scrape URLs from dynamically rendered web apps without the need of spawning headless browsers. It uses much less resources then headless browser alternatives and im currently using this to generate sitemaps for a production web app.

Install

npm install

Configuration

Move config.example.js to config.js
Change baseUrl to the websites base url
Change startUrl to the websites url you want to start scraping on

Usage

npm start

Author

👤 Hayden

Show your support

Give a ⭐️ if this project helped you!

About

A NodeJS program created to scrape URLs from dynamically rendered web apps without the need of spawning headless browsers.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published