Sunday, 11 August 2019

Read output output of a script with spark streaming

I have a script similar to this one

import json 
def line_generator():
    d = dict({1:1, 2:2, 3:3})
    while True:
        yield json.dumps(d)

it = line_generator()
for l in it:
    print(l)

which output values to stdout. I would like to "catch" those values with the spark Streaming api to store them in parquet files, and apply some inference code which is written is HiveQL. I am not a Scala person :/ so, if possible, I would prefer to have a solution in PySpark but am happy with any piece of advice.

I know it is possible to read stream of data coming from a Kafka stream for example, is there a similar way to read data sent to stdout or data that are continuously writtten to a file ?
Thanks in advance for your help



from Read output output of a script with spark streaming

No comments:

Post a Comment