My Problem
I have been using the python library Thread, and it seems to have some issues returning the correct results. When I run the same function e.g. ten times in a row, eight times the results are correct and two times they are wrong.
When the results are wrong, it is because some of the results-dictionarys from the individual calls have been seemingly randomly merged together.
My Code:
This function makes a session that retries rest calls for certain status codes:
# Makes retry sessions
def requests_retry_session(retries=3,backoff_factor=0.3,status_forcelist=(500, 502, 504),session=None):
"""
Description:
Creates a session which uses retries
Input:
retries (int): Max number of retries
backoff_factor (int): Time between retries
status_forcelist (tuble): Status for which to retry
Returns:
session: Requests session which handles different status and connection errors
"""
session = session or requests.Session()
retry = Retry(
total=retries,
read=retries,
connect=retries,
redirect=retries,
backoff_factor=backoff_factor,
status_forcelist=status_forcelist,
)
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)
return session
This function makes rest calls for multiple urls:
def make_rest_calls(urls, header, store=None):
"""
Description:
Processes list of urls
Input:
urls (list): List of urls for rest call
header (dictionary): Dictionary containing credentials
store (dictionary): Dictionary for collecting results
Returns:
store (dictionary): Dictionary with results
"""
if store is None:
store = {}
for url in urls:
store[url] = requests_retry_session().get(url, headers=header, timeout=5)
return store
This function runs rest calls multithreaded
def run_multi_threaded(nthreads, list_of_urls, header):
"""
Description:
Runs multiple threads
Input:
nthreads (int): Number for threads to run
list_of_urls(list): List of rest urls
header (dictionary): Dictionary containing credentials
Returns:
store (dictionary): Dictionary with results
"""
store = {}
threads = []
# create the threads
for i in range(nthreads):
small_list_of_urls = list_of_urls[i::nthreads]
t = Thread(target=make_rest_calls, args=(small_list_of_urls))
threads.append(t)
# start the threads
[t.start() for t in threads ]
# wait for the threads to finish
[ t.join() for t in threads ]
return store
Questions
Is this a weakness of the package? Should I use multiple processes instead? Or am I doing something wrong that results in this side effect only some times?
I need to make MANY calls, so it needs to be done multithreaded. The obviously also has to be correct.
from Python Threading at random returns wrong results
No comments:
Post a Comment