crawler
| Date | Project Name | 🎉 | ? | Tags |
|---|---|---|---|---|
| 11/05 |
CrawlerDetect v1.2.9
* [CrawlerDetect v1.2.9](https://github.com/loadkpi/crawler_detect) – Ruby gem for detecting bots, crawlers, and spiders based on user agent and HTTP headers.
Ruby gem for detecting bots, crawlers, and spiders based on user agent and HTTP headers.
|
5
|
Ruby 140 ⭐2657 days old |
ruby crawler spider bots crawler-detection |
| 11/02 |
Browsertrix Crawler v1.9.0-beta.1
* [Browsertrix Crawler v1.9.0-beta.1](https://github.com/webrecorder/browsertrix-crawler) – Standalone browser-based high-fidelity crawling system running customizable crawls in a single Docker container.
Standalone browser-based high-fidelity crawling system running customizable crawls in a single Docker container.
|
4
|
TypeScript 907 ⭐1831 days old |
typescript crawler crawling wacz warc web-archiving |
| 10/30 |
Firecrawl v2.5.0
* [Firecrawl v2.5.0](https://github.com/firecrawl/firecrawl) – API service that crawls URLs and converts web content into clean markdown or structured data without requiring sitemaps.
API service that crawls URLs and converts web content into clean markdown or structured data without requiring sitemaps.
|
9
|
TypeScript 66441 ⭐569 days old |
typescript data ai markdown scraper crawler html-to-markdown |