I have more than 1000 groups with different id and I only need to select a specific number of groups and read the nth number of every group. Here an example of what I need:
import pandas as pd
import numpy as np
df = pd.DataFrame({
'index':[0, 1, 2, 2, 3, 3, 4, 4, 5, 5, 5, 6, 6, 6, 7, 7, 7, 8, 8, 8, 8, 9, 9, 9, 9, 10, 10, 10, 10, 11, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884],
'id' : ['veh0', 'veh0', 'veh0', 'veh1', 'veh0', 'veh1', 'veh0', 'veh1', 'veh0', 'veh1', 'veh2', 'veh0', 'veh1', 'veh2', 'veh0', 'veh1', 'veh2', 'veh0', 'veh1', 'veh2', 'veh3', 'veh0', 'veh1', 'veh2', 'veh3', 'veh0', 'veh1', 'veh2', 'veh3', 'veh0', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192', 'veh1192'],
'veh_x' :[0, 1, 2, 2, 3, 3, 4, 4, 5, 5, 5, 6, 6, 6, 7, 7, 7, 8, 8, 8, 8, 9, 9, 9, 9, 10, 10, 10, 10, 11, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884],
'veh_y':[0, 1, 2, 2, 3, 3, 4, 4, 5, 5, 5, 6, 6, 6, 7, 7, 7, 8, 8, 8, 8, 9, 9, 9, 9, 10, 10, 10, 10, 11, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884]
}
)
print(df.groupby(['id']).head(1))
#first part
for i in range(1,20):
g=df.groupby(['id']).head(i).drop_duplicates(['id'], keep='last')
for idx, row in g.iterrows():
print("code1 group",i,"=",row['id'])
#second part
#This is what I need but it is running slowly when I add it to my whole dataset
data=['veh0', 'veh1', 'veh2', 'veh3']
for i in range(1,20):
for x in data: #these are the selected groups
g = df[df['id'].isin([x])].groupby(['id']).nth([i]).reset_index()
for x, row in g.iterrows():
print("code2 group",i,"=",row['id'])
The first part of the code read all the groups and return the nth number of every group but I need only five or six or more. The problem is that I don't know any information about the group. I can use a counter after that I can use break but the code is running so slowly because I need to load more than 30000 records every iteration. Here I added data=['veh0', 'veh1', 'veh2', 'veh3'] as an example but it can be chosen randomly.
The second part is what I want but the code still running slowly. What is the best way of making it better?
I appreciate any help?
from How to get a specific number of group after applying groupby?
Hi admin
ReplyDeleteThe first part of the code examine all of the agencies and go back the nth number of each group however I want only 5 or six or greater. The trouble is that I do not know any facts about the group. I can use a counter after that I can use wreck but the code is running so slowly because I want to load more than 30000 statistics every new release.
Thankskomal