Thursday, 22 June 2023

Web-scrapping ASPX page where loop over page number from Network>Payload

I am trying to scrap the table with column office, cadre, designation, name, and asset_details:

Here P2 ranges from 1 to 38.

In the Network > Payload > Form Data has the page number (__EVENTARGUMENT: Page$x), which can't be input and has to be ascertained.

http://bpsm.bihar.gov.in/Assets2020/AssetDetails.aspx?P1=2&P2=7&P3=0&P4=0

Here is my attempt at code:

import json
import requests
import pandas as pd


api_url = (
    "http://bpsm.bihar.gov.in/Assets2019/AssetDetails.aspx?P1=2&P2=33&P3=0&P4=0"
)
payload = {"P1": "2", "P2": "33", "P3": "0", "P4": "0"}


all_data = []
for P2 in range(1, 39):  # <-- increase from 1 to 200
    print(P2)
    payload['P2'] = P2
    data = requests.post(api_url, json=payload).json()
        data = json.loads(data['d'])
        if not data:
            break
        for name, count in data[0].items():
            all_data.append({
            })


from Web-scrapping ASPX page where loop over page number from Network>Payload

No comments:

Post a Comment