Wednesday, 1 December 2021

Can I send binary fles using the SimpleHttpOperator in Airflow?

I would like to upload binary data with a REST API using an Airflow's SimpleHttpOperator.
With python's requests, I would do that by doing

upload_response = requests.put(
        upload_url, data=Path(f"{filename}").read_bytes(), headers=headers
    )

In Airflow I have a first task downloading the file with an SFTPOperator and then trying to upload it with the SimpleHttpOperator:

get_file = SFTPOperator(
        task_id         = 'get_file',
        ssh_conn_id     = 'my_sftp_conn',
        remote_filepath = f"{templ_remote_filepath}/{filename}",
        local_filepath  = f"{basepath}/{filename}",
        operation       = 'GET',
    )

upload_file = SimpleHttpOperator(
        task_id      = 'upload_file',
        http_conn_id = "my_http_conn",
        endpoint     = get_upload_url.output,
        method       = 'PUT',
        headers      = {"Content-MD5": md5_hash_b64, "Content-Type": "application/text"},
        data         = Path(get_file.output).read_bytes(),
        log_response = True,
    )

I would like to be able to input get_file.output (the downloaded filepath) as a string instead of an XComArgs as it is. How can I do this?



from Can I send binary fles using the SimpleHttpOperator in Airflow?

No comments:

Post a Comment