Saturday, 4 December 2021

how to scrape product data from website pages

I previously used the code below to scrape the search result for a word search, for example book, on https://www.walmart.com/. They have currently changed their request and response parameters and this code does not get any response again.


        params = {
            'query': 'book',
            'cat_id': 0,
            'ps': 24,
            'offset': 0,

            'prg': 'desktop',
            'stores': re.search(r'store/(\d+)', url).group(1)
        }

        try:
            data1 = requests.get(api_url, params=params).json()

        except Exception as e:
            print("Sleeping for 10 seconds", e)
            time.sleep(10)

            try:
                data1 = requests.get(api_url, params=params).json()
            except Exception as e:
                print("sleeping for 60 seconds", e)
                time.sleep(60)

                try:
                    data1 = requests.get(api_url, params=params).json()
                except Exception as e:
                    print("sleeping for 360 seconds")
                    time.sleep(360)

                    data1 = requests.get(api_url, params=params).json()

I want to get the json response for a product page for example the product the in this url

https://www.walmart.com/ip/SKIPPY-Natural-Creamy-Peanut-Butter-Spread-15-oz/37447671

How could i rewrite the code with their current parameters to get the json response?



from how to scrape product data from website pages

No comments:

Post a Comment