Monday 12 July 2021

Python ProcessPoolExecutor process inside gRPC server dies without throwing any errors

I have a gRPC server which uses ProcessPoolExecutor with max_workers=1 to run a script. Here is why i had to use a ProcessPoolExecutor inside run_brain_application with max_workers=1. Below is the simplified version of the server script.

I've put two print statements to print the begining and ending of the process.

The issue is that sometimes, for some requests, the process starts but never finishes. I mean it prints START, but I never see the STOP. The same request works the next time, hence the issue is not with the request. Also, if there is an exception inside the run_application, it's thrown without any issues.

I've checked all the related questions on Stackoverflow, but most of them (1, 2) are about their processes not throwing exceptions, which is not my case.

Any thoughts greatly appreciated!

Thanks

class BrainServer(brain_pb2_grpc.BrainServiceServicer):

    def run_brain_application(self, request, context):
            request_dict = MessageToDict(
                request, 
                preserving_proto_field_name=True,
                keep_null=True
            )

            print("START")
            with futures.ProcessPoolExecutor(max_workers=1) as executor:
                result = executor.submit(run_application, request_dict).result()
            res = result.get('data')
            print("STOP")
            report = dict_to_protobuf(result_pb2.Result, {res_type: res}, strict=False)

            return report

        except Exception as e:
            _log_fatal(
                msg=str(e)
            )
            raise e


def serve():
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=16))
    brain_pb2_grpc.add_BrainServiceServicer_to_server(BrainServer(), server)

    server.add_insecure_port(f"[::]:{GRPC_PORT}")
    server.start()
    server.wait_for_termination()

if __name__ == '__main__':
    serve()


from Python ProcessPoolExecutor process inside gRPC server dies without throwing any errors

No comments:

Post a Comment