Tuesday 27 June 2023

Can't spawn background processes after the app was running for some time

In our Python application we spawn some background processes when the program starts up. Then in some cases we need to shutdown one of them and some time later we want to re-start (= spawn) another background process again.

However, it seems that sometimes, when the application was running for some days already (it's a server application), then it cannot spawn another process. In this case we get the following error log at the call of new_proc.start():

Jun 23 14:23:46 srv gunicorn[2717986]: Traceback (most recent call last):
Jun 23 14:23:46 srv gunicorn[2717986]:   File "<string>", line 1, in <module>
Jun 23 14:23:46 srv gunicorn[2717986]:   File "/usr/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
Jun 23 14:23:46 srv gunicorn[2717986]:     exitcode = _main(fd, parent_sentinel)
Jun 23 14:23:46 srv gunicorn[2717986]:   File "/usr/lib/python3.8/multiprocessing/spawn.py", line 126, in _main
Jun 23 14:23:46 srv gunicorn[2717986]:     self = reduction.pickle.load(from_parent)
Jun 23 14:23:46 srv gunicorn[2717986]:   File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 110, in __setstate__
Jun 23 14:23:46 srv gunicorn[2717986]:     self._semlock = _multiprocessing.SemLock._rebuild(*state)
Jun 23 14:23:46 srv gunicorn[2717986]: FileNotFoundError: [Errno 2] No such file or directory

So apparently the Python multiprocessing module cannot create the file required for a semaphore (or something similar). When I then shutdown the whole application and restart it, then again all subprocesses can be spawned again. So it really seems to be related to the fact that the app was running for some days already ...

Any ideas what might be the issue here?



from Can't spawn background processes after the app was running for some time

No comments:

Post a Comment