Xpath Returns Null

I need to scrape the price of this page: https://www.asos.com/monki/monki-lisa-cropped-vest-top-with-ruched-side-in-black/prd/23590636?colourwayid=60495910&cid=2623 However it

Solution 1:

Your problem is not the xpath, it's that the price is being retrieved with XHR.

If you use scrapy sheel and type view(response) you can see that the price is not being generated:

Look at the source of the original webpage and search for the price:

Then use this url the scrape the price:

defparse(self, response):
        import re
        price_url = 'https://www.asos.com' + re.search(r'window.asos.pdp.config.stockPriceApiUrl = \'(.+)\'', response.text).group(1)
        yield scrapy.Request(url=price_url,
                             method='GET',
                             callback=self.parse_price,
                             headers=self.headers)

    defparse_price(self, response):
        import json
        jsonresponse = json.loads(response.text)
        ...............
        ...............
        ...............

I couldn't get around 403 error with the headers I provided, but maybe you'll have more luck.

Edit:

In order to get the price from the json file there's actually no need for json.loads

defparse_price(self, response):
        jsonresponse = response.json()[0]
        price = jsonresponse['productPrice']['current']['text']
        # You can also use jsonresponse.get() if you preferprint(price)

Output:

£10.00

Solution 2:

This code that you've tried :

price' :response.xpath('//*[contains(@class, "current-price")]').get()

looks like you have your own written methods from your framework, but in native Selenium-Python Binding I would do this :-

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import Byfrom selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 20)
print(wait.until(EC.visibility_of_element_located((By.XPATH, "//*[contains(@class, "current-price")]/span"))).text)

Solution 3:

XPath you have used is returning 2 different elements. Try the following xpath to get the price of the item

driver.find_element_by_xpath("//span[@data-id='current-price']").text

Update:

price :response.xpath('//span[@data-id='current-price']').get()