Thursday 26 November 2020

Floats converting to symbols reading from .dat file. Unsure of encoding

I am attempting to read MapInfo .dat files into .csv files using Python. So far, I have found the easiest way to do this is though xlwings and pandas.

When I do this (below code) I get a mostly correct .csv file. The only issue is that some columns are appearing as symbols/gibberish instead of their real values. I know this because I also have the correct data on hand, exported from MapInfo.

import xlwings as xw
import pandas as pd

app = xw.App(visible=False)

tracker = app.books.open('./cable.dat')


last_row = xw.Range('A1').current_region.last_cell.row

data = xw.Range("A1:AE" + str(last_row))


test_dataframe = data.options(pd.DataFrame, header=True).value

test_dataframe.columns = list(schema)
test_dataframe.to_csv('./output.csv')

The float columns in the output csv

When I compare to the real data, I can see that the symbols do actually map the correct number (meaning that (1 = ­?, 2=@, 3=@, etc.)

Below is the first part of the 'dictionary' as to how they map: My dictionary

My question is this: Is there an encoding that I can use to turn these series of symbols into their correct representation? The floats aren't the only column affected by this, but they are the most important to my data.

Any help is appreciated.



from Floats converting to symbols reading from .dat file. Unsure of encoding

No comments:

Post a Comment