Thursday, 16 June 2022

PythonOperator fails where normal python code succeeds for google Bigquery

I have a custom pythonoperator in an airflow instance that supposed to pull a list of records from bigquery that i can reference in a API look up.

    os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/home/airflow/airflow/keys/the_JSON.json"

    client = bigquery.Client()
    geoclient = GeocodioClient('apikey')

    bq_sql = "SELECT ATTOM_ID, ADDRESS, CITY, STATE, ZIP FROM `project.dataset.tbl` Where GEOCODE_DT < DATE_SUB(CURRENT_DATE(), INTERVAL 2 month)  and ATTOM_ID IN (1, 2)"
    query_job = client.query(bq_sql)
    print(f'query job: {query_job}')
    results = query_job.result()  # Waits for job to complete.

ERROR - 400 Syntax error: Expected end of input but got identifier "Sedgwick" at [4:37]

"Sedgwick" is part of a value in a record.

When I attempt to run this code in the python interpreter my airflow server is at, it works fine. but in a dag pythonoperator, I get the error.

What can I do? thanks!



from PythonOperator fails where normal python code succeeds for google Bigquery

No comments:

Post a Comment