Sunday, 8 July 2018

Best way to process a click stream to create features in Pandas

I am processing a dataframe with a click-stream and I'm extracting features for each user in the click-stream to be used in a Machine Learning project.

The dataframe is something like this:

data = pd.DataFrame({'id':['A01','B01','A01','C01','A01','B01'],
                     'event':['search','search','buy','home','cancel','home'],
                     'date':['2018-01-01','2018-01-01','2018-01-02','2018-01-03','2018-01-04','2018-01-04'],
                     'product':['tablet','dvd','tablet','tablet','tablet','book']})

Since I have to create features for each user I'm using a groupby/apply with a custom function like:

data.groupby('id').apply(create_user_features)

Create user features will take a chunk of the dataframe and create many (hundreds) of features. The whole process is just too slow so I'm looking for a recommendation to do this more effciently.

One potential problem is that each feature potentially scans the whole chunk so if I have 100 features I scan the chunk 100 times instead of just one.

For example a feature can be the number of "tablet" events the user has, other can be the number of "home" events, other can be the average time difference between "search" events, then average time difference between "search" events for "tablets", etc etc. Each feature can be coded as a function that takes a chunk (df) and creates the feature but when we have 100s of features each is scanning the whole chunk when a single linear scan would suffice. The problem is the code would get ugly if I do a manual for loop over each record in the chunk and code all the features in the loop.

Questions:

  1. If I have to process a dataframe hundreds of times, is there a way to abstract this in a single scan that will create all the needed features?

  2. Is there a speed improvement over the groupby/apply approach I'm currently using?



from Best way to process a click stream to create features in Pandas

No comments:

Post a Comment