Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
Key features include:
- Simple: Scrapy was designed with simplicity in mind, by providing the features you need without getting in your way.
- Productive- Just write the rules to extract the data from web pages and let Scrapy crawl the entire web site for you.
- Fast: Scrapy is used in production crawlers to completely scrape more than 500 retailer sites daily, all in one server.
- Extensible: Scrapy was designed with extensibility in mind and so it provides several mechanisms to plug new code without having to touch the framework core.
- Portable: Scrapy runs on Linux, Windows, Mac and BSD
Open Source and 100% Python.
- Scrapy is completely written in Python, which makes it very easy to hack.
- Well-tested: Scrapy has an extensive test suite with very good code coverage.
For a full overview of Scrapy, see the ‘Scrapy at a glance‘ wiki page.
Open Source Living is a dynamic archive of Open Source software (OSS) spanning all major platforms, inclusive of small to large scale projects. It aims to introduce and inform new users about viable OSS alternatives to corporate, closed source software.
OS Living adheres to the Open Source Initiative's definition of OSS. Each software item included in the archive endeavours to conform to OSI guidelines on standards and licensing. Find out more »
Working on a top OSS project? Found something that could benefit others? Send us all the details via our user-friendly submission form and we'll consider it for inclusion in the Archive. Submit »
If you are looking to trade binary options with SpotOption software, make sure to check out this broker review first.
If you find the Open Source Living project a valuable resource and would like to help towards maintaining the site, we welcome donations through Paypal™. Donate »