Send Email Alert Using Scrapy After Multiple Spiders Have Finished Crawling

Just wondering what is the best way to implement this. I have 2 spiders and I want to send an email alert depending on what is scraped after the 2 spiders have finished crawling. I

Solution 1:

You can use this

from scrapy import signals
from scrapy.xlib.pydispatch import dispatcher

classMySpider(CrawlSpider):
    def__init__(self):
        dispatcher.connect(self.spider_closed, signals.spider_closed)

    defspider_closed(self, spider):
      # second param is the instance of the spider about to be closed.#  Write the mail sending part here

If you want to include the scraped data with the mail, write the script in the pipelines.py file.

classMyPipeline(Pipeline):
    spider = Nonedefprocess_item(self, item, spider):
        if spider.name == 'Name of the spider':
            # Use the data and send the mail from herereturn item