I have a pyspark dataframe like this,
+----------+--------+----------+----------+
|id_ | p |d1 | d2 |
+----------+--------+----------+----------+
| 1 | A |2018-09-26|2018-10-26|
| 2 | B |2018-06-21|2018-07-19|
| 2 | C |2018-07-13|2018-10-07|
| 2 | B |2018-12-31|2019-02-27|
| 2 | A |2019-01-28|2019-06-25|
-------------------------------------------
From this dataframe I have to make a dataframe like this,
+----------+--------+----------+----------+
|id_ | q |d1 | d2 |
+----------+--------+----------+----------+
| 1 | A |2018-09-26|2018-10-26|
| 2 | B |2018-06-21|2018-07-12|
| 2 | B C |2018-07-13|2018-07-19|
| 2 | C |2018-07-20|2019-10-07|
| 2 | B |2018-12-31|2019-01-27|
| 2 | B A |2019-01-28|2019-02-27|
| 2 | A |2019-02-28|2019-06-25|
-------------------------------------------
It is something like, finding which values of p are present in the data for a particular id_ from when to when. If there are multiple p in a same day then both should be present in the data, seperated by a space.
I tried to do this is by creating each and every dates in the range min(d1) and max(d2) and filling them accordingly. From that dataframe, after some melting and grouping I can get the desired result.
But the process takes very long time and is very inefficient.
I am looking for an efficient method for performing this task.
I can also have more complex cases of overlap, ie overlap among more than two p-values.
See a sample data below,
+----------+--------+----------+----------+
|id_ | p |d1 | d2 |
+----------+--------+----------+----------+
| 1 | A |2018-09-26|2018-10-26|
| 2 | B |2018-06-21|2018-07-19|
| 2 | C |2018-06-27|2018-07-07|
| 2 | A |2018-07-02|2019-02-27|
| 2 | A |2019-03-28|2019-06-25|
-------------------------------------------
This must be converted to,
+----------+--------+----------+----------+
|id_ | q |d1 | d2 |
+----------+--------+----------+----------+
| 1 | A |2018-09-26|2018-10-26|
| 2 | B |2018-06-21|2018-06-26|
| 2 | B C |2018-06-27|2018-07-01|
| 2 | B C A |2018-07-02|2018-07-07|
| 2 | A B |2018-07-08|2018-07-19|
| 2 | A |2018-07-20|2019-02-27|
| 2 | A |2019-03-28|2019-06-25|
-------------------------------------------
Order of individual items in the q doesn't matter. ie either if A, B and C are in overlap. It can either be shown as A B C, or B C A or A C B so on.
from Find min and max range with a combination of column values in PySpark
No comments:
Post a Comment