I'v created a script in scrapy to parse the author name of different posts from it's landing page and then pass it to the parse_page method using meta keyword in order to print the post content along with the author name at the same time.
I've used download_slot within meta keyword which allegedly maskes the script run faster. Although it is not necessary to comply with the logic I tried to apply here, I would like to stick to it only to understand how download_slot works within any script and why. I searched a lot to know more about download_slot but I end up some links like this one.
An example usage of download_slot (I'm not quite sure about it though):
from scrapy.crawler import CrawlerProcess
from scrapy import Request
import scrapy
class ConventionSpider(scrapy.Spider):
name = 'stackoverflow'
start_urls = ['https://stackoverflow.com/questions/tagged/web-scraping']
def parse(self,response):
for link in response.css('.summary'):
name = link.css('.user-details a::text').extract_first()
url = link.css('.question-hyperlink::attr(href)').extract_first()
nurl = response.urljoin(url)
yield Request(nurl,callback=self.parse_page,meta={'item':name,"download_slot":name})
def parse_page(self,response):
elem = response.meta.get("item")
post = ' '.join([item for item in response.css("#question .post-text p::text").extract()])
yield {'Name':elem,'Main_Content':post}
if __name__ == "__main__":
process = CrawlerProcess({
'USER_AGENT': 'Mozilla/5.0',
})
process.crawl(ConventionSpider)
process.start()
The above script runs flawlessly.
My question: how
download_slotworks within scrapy?
from How "download_slot" works within scrapy
No comments:
Post a Comment