Friday, 25 January 2019

Multiple pyspark "window()" calls shows error when doing a "groupBy()"

This question is a follow up of this answer. Spark is displaying an error when the following situation arises:

# Group results in 12 second windows of "foo", then by integer buckets of 2 for "bar"
fooWindow = window(col("foo"), "12 seconds"))

# A sub bucket that contains values in [0,2), [2,4), [4,6]...
barWindow = window(col("bar").cast("timestamp"), "2 seconds").cast("struct<start:bigint,end:bigint>")

results = df.groupBy(fooWindow, barWindow).count()

The error is:

"Multiple time window expressions would result in a cartesian product of rows, therefore they are currently not supported."

Is there some way to achieve the desired behavior?



from Multiple pyspark "window()" calls shows error when doing a "groupBy()"

No comments:

Post a Comment