PyPy support

[joehillen]

This is a fix for Issue #188.

It is caused by a disagreement about methods' __call__ attribute being im_func in CPython vs PyPy.

Here is the test case

[pablohoffman]

Nice patch!

Could we add a quick test to Scrapy test suite? (maybe just the one referred in the test suite)

[joehillen]

Just wanted to let you know I haven't forgotten this pull request.

I found some segfaults when running tests in PyPy, and I want to explore those more since it's a more serious issue than the one I originally found.

You can follow the thread here:

http://mail.python.org/pipermail/pypy-dev/2012-December/010741.html

[pablohoffman]

Thanks for the update @joehillen, I hereby pronounce you the official Scrapy PyPy ambassador! :)

[dangra]

We need test cases to merge this pull request, nobody wants to break your hard work without notice in later changes.

[joehillen]

I have been waiting on a fix for the segfaulting issue. Here is the PyPy ticket, though I think I can/should make a test case anyway. I'll be able to get to it later this week.

[dangra]

thanks for the update. I will leave it open :)

[kmike]

I don't think this PR needs tests because most of scrapy test suite already fails under PyPy 2.0.2 without this patch and passes with it (I'm checking this using 'tox -e pypy' after applying #341).

The test suite runs super-slow under PyPy though (about 1 test/minute, with idle CPU).

[dangra]

fair point about tests, and great to have an easy way to run tests with pypy

[lopuhin]

Checked current status of PyPy support:

• pydispatch fix is gone as of 776064a (upsteam pydispatcher does not have this fix, and has a bug report about it http://sourceforge.net/p/pydispatcher/bugs/3/)
• lxml-cffi at cffi branch with this PR applied added explicit relaxng.h includes to fix building on Debian-based OSs amauryfa/lxml#10 seems to work fine

Modulo these two issues, most tests pass (15 failures out of 1200+ tests, most look easy to fix), and scrapy bench is 2x faster after about 1 minute warmup (6800-7200 pages/s for PyPy 4.0.1 vs 2800-3200 for CPython 2.7.9)

[jschilling1]

pypy would be so great. i wouldn't have guessed a 2x speed up given lxml does most of the work. scrapy crawl currently fails silently after printing the enabled middlewares list; scrapy shell print this error:

2016-02-06 11:20:24 [flowershop] ERROR: Error caught on signal handler: <bound method CoreStats.spider_opened of <scrapy.extensions.corestats.CoreStats object at 0x0000000004819948>>
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/pypy-4.0.1/envs/master2pypy/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/home/user/.pyenv/versions/pypy-4.0.1/envs/master2pypy/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
TypeError: spider_opened() got 2 unexpected keyword arguments

but starts up and i can confirm lxml works, i.e.

from spider.spiders import MySpider
MySpider.parse_item(response)

i used the following lxml branch https://github.com/lxml/lxml/tree/pypy4

[lopuhin]

@jschilling1 To fix the first issue, you need to first install pydispatcher with PyPy support (sorry, I don't know a simple way to do it offhand).
I am too excited about PyPy support :)

[lopuhin]

Btw, lxml-cffi is on pypi now (and on github too: https://github.com/lxml-cffi/lxml-cffi), although you need a very recent setuptools, and the package name is changed to lxml-cffi, so some lxml-cffi-compat module (like this https://github.com/dstufft/psycopg2cffi-compat) is required so that scrapy dependencies that depend on lxml will not try to install lxml alongside lxml-cffi.

[jschilling1]

very good, the older cffi branch wouldn’t even compile for me in the past; wow pydispatcher hasn't seen a commit in almost a decade ((

[lopuhin]

pydispatcher was updated just a year ago: https://pypi.python.org/pypi/PyDispatcher. But still I think the short-term plan is to put pydispatcher with pypy support to pypi, and the long-term plan is to refactor signals and get rid of it (see #8)

[kmike]

From http://morepypy.blogspot.ru/2016/02/c-api-support-update.html:

A side effect of this work is that now PyPy actually supports the upstream lxml package---which is is one of the most popular packages on PyPI. (Specifically, you need version 3.5.0 with this pull request to remove old PyPy-specific hacks that were not really working. See details.) At this point, we no longer recommend using the cffi-lxml alternative: although it may still be faster, it might be incomplete and old.