Monday, 21 September 2020

Enhanced Python Multiprocessing Data Pipeline Wrapper

Objective

This is a piece of a big project I'm working on. This is an important part that will massively simplify report transmission in my program. The program tests a function against millions of inputs and uses multiprocessing to speed thing up. Source code on Pastebin.

Goals and Benefit

Put simply, multiprocessing.Pipe() is inadequate. It should be able to handle massive strings and switch process execution between a sender and receiver. I wrote this to implement:

  • Automatic error handling
  • Transmission error categorization
  • Data transmission chunking and reassembly
  • Unlimited data transmission size
  • Process synchronization
  • Simple abstraction to enhance usability

Problem

It has a weird bug I can't find. Days and plenty of documentation later, it's not fixed. I've left in a good many debug lines. Try entering "hi": you don't see "Receiver.Test: Output: hi" but should. Try a second time, it just hangs: Sample output.

Tests

The GPE works. Both of these first two tests work. For test 1, this source code outputs these results correctly and consistently. For test 2, this source code outputs something like these results correctly. For test 3, this source code outputs something like these results correctly.

Plea!

It's time to ask for help. It is part of a larger project. To be fair, there are a good many lines of code. This should be part of the multiprocessing module. I'm humbled. Can someone tell me what up? Thank you all in advance!



from Enhanced Python Multiprocessing Data Pipeline Wrapper

No comments:

Post a Comment