I have a script similar to this one
import json
def line_generator():
d = dict({1:1, 2:2, 3:3})
while True:
yield json.dumps(d)
it = line_generator()
for l in it:
print(l)
which output values to stdout. I would like to "catch" those values with the spark Streaming api to store them in parquet files, and apply some inference code which is written is HiveQL. I am not a Scala person :/ so, if possible, I would prefer to have a solution in PySpark but am happy with any piece of advice.
I know it is possible to read stream of data coming from a Kafka stream for example, is there a similar way to read data sent to stdout or data that are continuously writtten to a file ?
Thanks in advance for your help
from Read output output of a script with spark streaming
No comments:
Post a Comment