I am reading chunks of data that is an API response using the following code:
d = zlib.decompressobj(zlib.MAX_WBITS|16) # for gzip
for i in range(0, len(data), 4096):
chunk = data[i:i+4096]
# print(chunk)
str_chunk = d.decompress(chunk)
str_chunk = str_chunk.decode()
# print(str_chunk)
if '"@odata.nextLink"' in str_chunk:
ab = '{' + str_chunk[str_chunk.index('"@odata.nextLink"'):len(str_chunk)+1]
ab = ast.literal_eval(ab)
url = ab['@odata.nextLink']
return url
An example of this working is: "@odata.nextLink":"someurl?$count=true
It works in most cases but sometimes this key value pair gets cut off and it appears something like this: "@odata.nextLink":"someurl?$coun
I can play around with the number of bits in this line for i in range(0, len(data), 4096)
but that doesn't ensure that in some cases the data doesn't cut off as the page sizes (data size) can be different for each page size.
How can I ensure that this key value pair is never cut off. Also, note that this key value pair is the last line/ last key-value pair of the API response.
P.S.: I can't play around with API request parameters.
Even tried reading it backwards but this gives a header incorrect issue:
for i in range(len(data), 0, -4096):
chunk = data[i -4096: i]
str_chunk = d.decompress(chunk)
str_chunk = str_chunk.decode()
if '"@odata.nextLink"' in str_chunk:
ab = '{' + str_chunk[str_chunk.index('"@odata.nextLink"'):len(str_chunk)+1]
ab = ast.literal_eval(ab)
url = ab['@odata.nextLink']
#print(url)
return url
The above produces the following error which is really strange:
str_chunk = d.decompress(chunk)
zlib.error: Error -3 while decompressing data: incorrect header check
from Chunking API response cuts off required data
No comments:
Post a Comment