Thursday, 29 July 2021

Proper HTTP POST request to mimic submitting an online form (using HTML, cURL or python)

Problem description: I am having trouble figuring out what is missing from my HTTP request to get to the next page after this form: When I do this from my code, I land on a "welcome" page instead of landing on the expected one from Step 2 below:

  1. Step 1: Filling it out with the following and clicking "Search" in a browser:

  2. Step 2: I get to this next page:

I have looked up all form data being posted and did a simplified pre-filled form here, but clicking on Search in this case does not work:

What I have tried

  • I have read this thread which advises against simplifying the form, but I believe I am doing this correctly: all the fields I am aware of are present, I simply change their HTML visualization (which has no bearing on what is POSTed).
  • The jspSynchronizerToken field could be checked on the server side, therefore I already tried filling it out manually with a valid value retrieved in the same browser. This alone does not solve the problem.
  • I have tried using the following cURL command, but this does not work either:
curl -A "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" \
    -F 'pageName=com/sed/jsp/reports/ItdReportInput.jsp' \
    -F 'menukey=15.03.00' \
    -F 'locale=en_CA' \
    -F 'jspSynchronizerToken=BzDQe5n2dIhV4LHKyqV_u341627003039920' \
    -F 'PUBLIC_USERS=0' \
    -F 'FORMAT=TXT' \
    -F 'SORTED_BY=0' \
    -F 'SELECT_TYPE=8' \
    -F 'SELECT_TYPE_VALUE=Test' \
    -F 'SELECT_TYPE_VALUE_SEARCH_TYPE=0' \
    -F 'DATE_RANGE_TYPE=0' \
    -F 'MONTH_FROM_PUBLIC=0' \
    -F 'DAY_FROM_PUBLIC=1' \
    -F 'YEAR_FROM_PUBLIC=2020' \
    -F 'MONTH_TO_PUBLIC=11' \
    -F 'DAY_TO_PUBLIC=31' \
    -F 'YEAR_TO_PUBLIC=2020' \
    -F 'Search=Search' \
    https://www.sed.com/sed/SVTItdController?locale=en_CA \
> ~/Downloads/sed.html
  • I have also coded a more in-depth python version in which I provide the referer as well as the retrieved token from the Step 1, but with no success. (This code is available but it is simply a less readable version of the above simplified code so I don't see the point in posting, let me know if I am wrong).
  • I have considered cookies and javascript might be the problem, yet a browser with no javascript or cookies allowed correctly submits the form and lands on the correct result.
  • I am currently looking at the requests / responses using Charles Proxy, but I am not getting any new insights so far...

I realize this question is a bit specific to the website I am showing, nevertheless it seems to me that there is a generic understanding of how websites process HTTP requests and how to validly mimic those which could be gained by a proper answer here. I have had other experience akin to this one in the past and am hoping to understand this properly once and for all :)



from Proper HTTP POST request to mimic submitting an online form (using HTML, cURL or python)

No comments:

Post a Comment