Scrapy freeze on connection timeut

I wrote a scrapy crawler that uses an Internet connection that is pretty unreliable. This is something that is a given. I cannot easily or cheaply change it - once in a while the Internet connection will be lost and after a few seconds or so it will be restored.

I observe behaviour where a Scrapy 18.4 crawler would freeze indefinitely without printing any error messages. It stops reacting to Ctrl+C, which makes me think this happens somewhere pretty deep in the reactor stack, though I cannot be sure.

There are absolutely no error messages which makes things rather hopeless to debug.

Question: Would anyone have any clues as to how to debug this problem? I don't really have any meaningful logs to attach for the reasons laid out above.

from Scrapy freeze on connection timeut