Monday, 3 April 2023

Python 3 SQLAlchemey UnicodeDecodeError: 'ascii' codec can't decode byte

I have some code which I'm finally getting around to updating to Python 3.

It runs in Docker using ubuntu:20.04 as the base image.

I am using SQLAlchemy query .all() method method sqlalchemy.orm.Query.all()

The code works with Python 2 however, it fails on Python 3 with the following error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe7 in position 1: ordinal not in range(128)

I am struggling to understand why the code is trying to decode to ascii.

I've set the locale in the Dockerfile and running locale.getlocale() on the line above the SQLAlchemy line prints ('en_US', 'UTF-8')

The SQLAlchemy connection url specifies utf8:

sqlalchemy.url='mysql+pymysql:{server}&charset=utf8&binary_prefix=true

I've read all the other similar questions but still cannot get this working.

Update:

I've tracked it down to one column:

from sqlalchemy import PickleType


class Schedule(OrgRefMixin, DeclarativeBase):
    __tablename__ = 'flights'
    ...
        routing = Column(PickleType())
    ...

This is a blob column in the database.

Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/venv/lib/python3.8/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/venv/lib/python3.8/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/venv/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/venv/lib/python3.8/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/venv/lib/python3.8/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/code/flights/flights/json.py", line 182, in view_wrapper
    return view_func(*args, **kwargs)
  File "/code/flights/flights/blueprints/flights.py", line 62, in get_flights
    flights = Flights.get(filters, **valid_args)
  File "/code/flights/flights/resources/flights.py", line 254, in get
    response = query.all()
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2643, in all
    return list(self)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 90, in instances
    util.raise_from_cause(err)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 77, in instances
    rows = [keyed_tuple([proc(row) for proc in process])
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 77, in <listcomp>
    rows = [keyed_tuple([proc(row) for proc in process])
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 77, in <listcomp>
    rows = [keyed_tuple([proc(row) for proc in process])
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py", line 93, in __getitem__
    return processor(self._row[index])
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/sql/sqltypes.py", line 1478, in process
    return loads(value)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe7 in position 1: ordinal not in range(128)


from Python 3 SQLAlchemey UnicodeDecodeError: 'ascii' codec can't decode byte

No comments:

Post a Comment