Tuesday 8 December 2020

Understanding FeatureHasher, collisions and vector size trade-off

I'm preprocessing my data before implementing a machine learning model. Some of the features are with high cardinality, like country and language.

Since encoding those features as one-hot-vector can produce sparse data, I've decided to look into the hashing trick and used python's category_encoders like so:

from category_encoders.hashing import HashingEncoder
ce_hash = HashingEncoder(cols = ['country'])
encoded = ce_hash.fit_transform(df.country)
encoded['country'] = df.country
encoded.head()

When looking at the result, I can see the collisions

    col_0  col_1  col_2  col_3  col_4  col_5  col_6  col_7 country
0       0      0      1      0      0      0      0      0      US <━┓
1       0      1      0      0      0      0      0      0      CA.  ┃ US and SE collides 
2       0      0      1      0      0      0      0      0      SE <━┛
3       0      0      0      0      0      0      1      0      JP

Further investigation lead me to this Kaggle article. The example of Hashing there include both X and y.

  • What is the purpose of y, does it help to fight the collision problem?
  • Should I add more columns to the encoder and encode more than one feature together (for example country and language)?

Will appreciate an explanation of how to encode such categories using the hashing trick.

Update: Based on the comments I got from @CoMartel, Iv'e looked at Sklearn FeatureHasher and written the following code to hash the country column:

from sklearn.feature_extraction import FeatureHasher
h = FeatureHasher(n_features=10,input_type='string')
f = h.transform(df.country)
df1 = pd.DataFrame(f.toarray())
df1['country'] = df.country
df1.head()

And got the following output:

     0    1    2    3    4    5    6    7    8    9 country
0 -1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 -1.0  0.0      US
1 -1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 -1.0  0.0      US
2 -1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 -1.0  0.0      US
3  0.0 -1.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0      CA
4  0.0  0.0  0.0  0.0  0.0  0.0  0.0  1.0 -1.0  0.0      SE
5  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0      JP
6 -1.0  0.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0      AU
7 -1.0  0.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0      AU
8 -1.0  0.0  0.0  0.0  0.0  0.0  1.0  0.0  0.0  0.0      DK
9  0.0  0.0  0.0  0.0  0.0  0.0  0.0  1.0 -1.0  0.0      SE
  • Is that the way to use the library in order to encode high categorical values?
  • Why are some values negative?
  • How would you choose the "right" n_features value?
  • How can I check the collisions ratio?


from Understanding FeatureHasher, collisions and vector size trade-off

No comments:

Post a Comment