Friday 27 November 2020

Handling two incoming data streams and combining them in python?

I have been researching various options in python of threading, multiprocessing async etc as ways of handling two incoming streams and combining them. There is a lot of information about, but the examples are often convoluted and complicated, and more commonly are to split up a single task into multiple threads or processes to speed up the end result of the task.

I have a data stream coming in over a socket (currently using UDP as its another application running locally on my PC, but may consider switching to TCP in the future if the application needs to be run on a separate PC), and a serial stream coming in via an RS232 adaptor, and I need to combine the streams. This new stream is then retransmitted on another socket.

The issue is that they come in at different rates (serial data is coming in at 125hz, socket data at 60-120hz), so I want to add the latest serial data to the socket data.

My question is essentially what is the best way to handle this, based on other peoples previous experience. Since this is essentially an I/O task, it lends more towards threading (which I know is limited to concurrency by the GIL), but due to the high input rate, I am wondering if Multi-processing is the way to go? If using threading, I guess the best way to access each shared resource is using a Lock to write the serial data to an object, and in a separate thread whenever there is new socket data then acquiring the lock, accessing the latest serial data in the object, processing it then sending it out on the other socket. However, the main thread has a lot of work to in between each new incoming socket message. With Multi-processing I could use a pipe to request and receive the latest serial data from the other process, but that only offloads the serial data handling, and still leaves a lot for the main process.

Has anyone had any experience and could recommend an approach? Any links or examples would also be hugely appreciated. Thanks!



from Handling two incoming data streams and combining them in python?

No comments:

Post a Comment