I would like to upload binary data with a REST API using an Airflow's SimpleHttpOperator.
With python's requests, I would do that by doing
upload_response = requests.put(
upload_url, data=Path(f"{filename}").read_bytes(), headers=headers
)
In Airflow I have a first task downloading the file with an SFTPOperator and then trying to upload it with the SimpleHttpOperator:
get_file = SFTPOperator(
task_id = 'get_file',
ssh_conn_id = 'my_sftp_conn',
remote_filepath = f"{templ_remote_filepath}/{filename}",
local_filepath = f"{basepath}/{filename}",
operation = 'GET',
)
upload_file = SimpleHttpOperator(
task_id = 'upload_file',
http_conn_id = "my_http_conn",
endpoint = get_upload_url.output,
method = 'PUT',
headers = {"Content-MD5": md5_hash_b64, "Content-Type": "application/text"},
data = Path(get_file.output).read_bytes(),
log_response = True,
)
I would like to be able to input get_file.output (the downloaded filepath) as a string instead of an XComArgs as it is. How can I do this?
from Can I send binary fles using the SimpleHttpOperator in Airflow?
No comments:
Post a Comment