Sunday, 17 November 2019

How to get a specific number of group after applying groupby?

I have more than 1000 groups with different id and I only need to select a specific number of groups and read the nth number of every group. Here an example of what I need:

import pandas as pd
import numpy as np

df = pd.DataFrame({
  'index':[0, 1, 2, 2, 3, 3, 4, 4, 5, 5, 5, 6, 6, 6, 7, 7, 7, 8, 8, 8, 8, 9, 9, 9, 9, 10, 10, 10, 10, 11, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884],
'id'    :   ['veh0',    'veh0', 'veh0', 'veh1', 'veh0', 'veh1', 'veh0', 'veh1', 'veh0', 'veh1', 'veh2', 'veh0', 'veh1', 'veh2', 'veh0', 'veh1', 'veh2', 'veh0', 'veh1', 'veh2', 'veh3', 'veh0', 'veh1', 'veh2', 'veh3', 'veh0', 'veh1', 'veh2', 'veh3', 'veh0', 'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192',  'veh1192'],
'veh_x' :[0, 1, 2, 2, 3, 3, 4, 4, 5, 5, 5, 6, 6, 6, 7, 7, 7, 8, 8, 8, 8, 9, 9, 9, 9, 10, 10, 10, 10, 11, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884],
'veh_y':[0, 1, 2, 2, 3, 3, 4, 4, 5, 5, 5, 6, 6, 6, 7, 7, 7, 8, 8, 8, 8, 9, 9, 9, 9, 10, 10, 10, 10, 11, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884]

}
)

print(df.groupby(['id']).head(1))
#first  part
for i in range(1,20):
   g=df.groupby(['id']).head(i).drop_duplicates(['id'], keep='last')
   for idx, row in g.iterrows():
     print("code1 group",i,"=",row['id'])



#second part
#This is what I need but it is running slowly when I add it to my whole dataset
data=['veh0',   'veh1', 'veh2', 'veh3']  
for i in range(1,20):
 for x in data: #these are the selected groups
            g = df[df['id'].isin([x])].groupby(['id']).nth([i]).reset_index()
            for x, row in g.iterrows():
               print("code2 group",i,"=",row['id'])

The first part of the code read all the groups and return the nth number of every group but I need only five or six or more. The problem is that I don't know any information about the group. I can use a counter after that I can use break but the code is running so slowly because I need to load more than 30000 records every iteration. Here I added data=['veh0', 'veh1', 'veh2', 'veh3'] as an example but it can be chosen randomly.

The second part is what I want but the code still running slowly. What is the best way of making it better?

I appreciate any help?



from How to get a specific number of group after applying groupby?

1 comment:

  1. Hi admin
    The first part of the code examine all of the agencies and go back the nth number of each group however I want only 5 or six or greater. The trouble is that I do not know any facts about the group. I can use a counter after that I can use wreck but the code is running so slowly because I want to load more than 30000 statistics every new release.
    Thankskomal

    ReplyDelete