Thursday 3 December 2020

Finding relationships between values based on their name in Python with Panda

I want to make relationship between values by their Name based on below rules:

1- I have a CSV file (with more than 100000 rows) that consists of lots of values, I shared some examples as below:

    Name:
A02-father
A03-father
A04-father
A05-father
A07-father
A08-father
A09-father
A17-father
A18-father
A20-father
A02-SA-A03-SA
A02-SA-A04-SA
A03-SA-A02-SA
A03-SA-A05-SA
A03-SA-A17-SA
A04-SA-A02-SA
A04-SA-A09-SA
A05-SA-A03-SA
A09-SA-A04-SA
A09-SA-A20-SA
A17-SA-A03-SA
A17-SA-A18-SA
A18-SA-A17-SA
A20-SA-A09-SA
A05-NA
B02-Father
B04-Father
B06-Father
B02-SA-B04-SA
B04-SA-BO2-SA
B04-SA-B06-SA
B06-SA-B04-SA
B06-NA

2- Now I have another CSV file which let me know from which value I should start? in this case the value is A03-father & B02-father & ... which dont have any influence on each other and they all have seperate path to go, so for each path we will start from mentioned start point. father.csv A03-father B02-father ....

3- Based on the naming I want to make the relationships, As A03-Father has been determined as Father I should check for any value which has been started with A03.(All of them are A0's babies.) Also as B02 is father, we will check for any value which starts with B02. (B02-SA-B04-SA)

4- Now If I find out A03-SA-A02-SA , this is A03's baby. I find out A03-SA-A05-SA , this is A03's baby. I find out A03-SA-A17-SA , this is A03's baby.

and after that I must check any node which starts with A02 & A05 & A17: As you see A02-Father exists so it is Father and now we will search for any string which starts with A02 and doesn't have A03 which has been detected as Father(It must be ignored)

This must be checked till end of values which exist in the CSV file. As you see I should check the path based on name (REGEX) and should go forward till end of path.

The expected result:

    Father      Baby
A03-father   A03-SA-A02-SA
A03-father   A03-SA-A05-SA
A03-father   A03-SA-A17-SA
A02-father   A02-SA-A04-SA
A05-father   A05-NA
A17-father   A17-SA-A18-SA
A04-father   A04-SA-A09-SA
A02-father   A02-SA-A04-SA
A09-father   A09-SA-A20-SA
B02-father   B02-SA-B04-SA
B04-father   B04-SA-B06-SA
B06-father   B06-NA

I have coded it as below with pandas:

import pandas as pd
import numpy as np
import re

#Read the file which consists of all Values
df = pd.read_csv("C:\\total.csv")


#Read the file which let me know who is father
Fa = pd.read_csv("C:\\Father.csv")

#Get the first part of Father which is A0
Fa['sub'] = Fa['Name'].str.extract(r'(\w+\s*)', expand=False)
r2 = []

#check in all the csv file and find anything which starts with A0 and is not Father
for f in Fa['sub']:
     baby=(df[df['Name'].str.startswith(f) & ~df['Name'].str.contains('Father')])
     baby['sub'] = bay['Name'].str.extract(r'(\w+\s*)', expand=False)
     r1= pd.merge(Fa, baby, left_on='sub', right_on='sub',suffixes=('_f', '_c'))
     r2.append(result1)
out_df = pd.concat(result2)
out_df= out_df.replace(np.nan, '', regex=True)
#find A0-N-A2-M and A0-N-A4-M
out_df.to_csv('C:\\child1.csv')



#check in all the csv file and find anything which starts with the second part of child1 which is A2 and A4
out_df["baby2"] = out_df['Name_baby'].str.extract(r'^(?:[^-]*-){2}\s*([^-]+)', expand=False)
baby3= out_df["baby2"]
r4 = []
for f in out_df["baby2"]:
    #I want to exclude A0 which has been detected.
     l = ['A0']  
     regstr = '|'.join(l)
     baby1=(df[df['Name'].str.startswith(f) & ~df['Name'].str.contains(regstr)])
     baby1['sub'] = baby1['Name'].str.extract(r'(\w+\s*)', expand=False)

     r3= pd.merge(baby3, baby1, left_on='baby2', right_on='sub',suffixes=('_f', '_c'))
     r4.append(r3)
out2_df = pd.concat(r4)
out2_df.to_csv('C:\\child2.csv')

I want to put below code in a loop and go through the file and check it, based on naming process and detect other fathers and babies till it finished. however this code is not customized and doesn't have the exact result as i expected. my question is about how to make the loop?

I should go through the path and also consider regstr value for any string.

#check in all the csv file and find anything which starts with the second part of child1 which is A2 and A4

out_df["baby2"] = out_df['Name_baby'].str.extract(r'^(?:[^-]*-){2}\s*([^-]+)', expand=False)
baby3= out_df["baby2"]
r4 = []
for f in out_df["baby2"]:
    #I want to exclude A0 which has been detected. 
     l = ['A0']  
     regstr = '|'.join(l)
     baby1=(df[df['Name'].str.startswith(f) & ~df['Name'].str.contains(regstr)])
     baby1['sub'] = baby1['Name'].str.extract(r'(\w+\s*)', expand=False)

     r3= pd.merge(baby3, baby1, left_on='baby2', right_on='sub',suffixes=('_f', '_c'))
     r4.append(r3)
out2_df = pd.concat(r4)
out2_df.to_csv('C:\\child2.csv')


from Finding relationships between values based on their name in Python with Panda

No comments:

Post a Comment