Scrapy

  • Version

    0.10.4
  • Date Added
    (to site)

    December 9th, 2010
  • Author

    Insophia
  • License

    BSD (?)
  • Description

    Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

    Key features include:

    - Simple: Scrapy was designed with simplicity in mind, by providing the features you need without getting in your way.

    - Productive- Just write the rules to extract the data from web pages and let Scrapy crawl the entire web site for you.

    - Fast: Scrapy is used in production crawlers to completely scrape more than 500 retailer sites daily, all in one server.

    - Extensible: Scrapy was designed with extensibility in mind and so it provides several mechanisms to plug new code without having to touch the framework core.

    - Portable: Scrapy runs on Linux, Windows, Mac and BSD
    Open Source and 100% Python.

    - Scrapy is completely written in Python, which makes it very easy to hack.

    - Well-tested: Scrapy has an extensive test suite with very good code coverage.

    For a full overview of Scrapy, see the ‘Scrapy at a glance‘ wiki page.

  • Platforms Supported

    WindowsMacLinux
  • Website

Scrapy

Rating
Give your rating »

  • Overall

    3.5

Installation ★★★☆☆
Features ★★★★★
Usability ★★★★★

Reviews
Write a review »

  • 1 positive
  • 0 negative

Reviews 1 reviews | Write a review »

Marcos
Dec 9th, 2010, 11:30 am | #

Review

I work at a scraping company which is migrating from Web Harvest to Scrapy, and I recommend it wholeheartedly. Web Harvest XML syntax was too limiting for US. With Scrapy, we can write any kind of scrapers (as we have full HTTP request control), and it provides a great environment for debugging and testing your scrapers. I still need to get used to referring them as “spiders” though :)

Rating

Installation ★★★☆☆
Features ★★★★★
Usability ★★★★★

Write a review

Review Guidelines:
You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Off-topic or inappropriate reviews will be edited or deleted.

Email addresses will never be published.

Ratings:
5 stars = Excellent | 4 stars = Very Good | 3 stars = Average | 2 stars = Below Average | 1 star = Poor

Rate

About

Open Source Living is a dynamic archive of Open Source software (OSS) spanning all major platforms, inclusive of small to large scale projects. It aims to introduce and inform new users about viable OSS alternatives to corporate, closed source software.

Through its Community Facebook Page and exciting multi-authored publication, Sourced, OS Living houses informed discussion on issues of import in the Open Source field.

OS Living adheres to the Open Source Initiative's definition of OSS. Each software item included in the archive endeavours to conform to OSI guidelines on standards and licensing. Find out more »

Submit

Working on a top OSS project? Found something that could benefit others? Send us all the details via our user-friendly submission form and we'll consider it for inclusion in the Archive. Submit »

If you are looking to trade binary options with SpotOption software, make sure to check out this broker review first.

Donate

If you find the Open Source Living project a valuable resource and would like to help towards maintaining the site, we welcome donations through Paypal™. Donate »