Thursday 5 August 2021

Airflow SSHOperator's Socket exception: Bad file descriptor

In Airflow, I use an SSHOperator to call an API that works on some automation work. The work ran successfully and the report did generate, but Airflow returns the task failed due to the Socket exception.

This error sometimes occurs, and I would like to know the reason that caused it.

The error message received:

[2021-07-20 08:00:07,345] {ssh.py:109} INFO - Running command: curl -u <user:pw> <URL>
[2021-07-20 08:00:07,414] {ssh.py:145} WARNING -   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[2021-07-20 08:00:08,420] {ssh.py:145} WARNING - 
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
[2021-07-20 08:00:09,421] {ssh.py:145} WARNING - 
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
[2021-07-20 08:00:10,423] {ssh.py:145} WARNING - 
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
[2021-07-20 08:00:10,615] {ssh.py:141} INFO - Report Sent Successfully.
[2021-07-20 08:00:10,616] {transport.py:1819} ERROR - Socket exception: Bad file descriptor (9)
[2021-07-20 08:00:10,633] {taskinstance.py:1481} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/u01/airflow-venv/lib/python3.8/site-packages/airflow/providers/ssh/operators/ssh.py", line 152, in execute
    stdout.channel.close()
  File "/u01/airflow-venv/lib/python3.8/site-packages/paramiko/channel.py", line 671, in close
    self.transport._send_user_message(m)
  File "/u01/airflow-venv/lib/python3.8/site-packages/paramiko/transport.py", line 1863, in _send_user_message
    self._send_message(data)
  File "/u01/airflow-venv/lib/python3.8/site-packages/paramiko/transport.py", line 1839, in _send_message
    self.packetizer.send_message(data)
  File "/u01/airflow-venv/lib/python3.8/site-packages/paramiko/packet.py", line 431, in send_message
    self.write_all(out)
  File "/u01/airflow-venv/lib/python3.8/site-packages/paramiko/packet.py", line 367, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/u01/airflow-venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1137, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/u01/airflow-venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/u01/airflow-venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1336, in _execute_task
    result = task_copy.execute(context=context)
  File "/u01/airflow-venv/lib/python3.8/site-packages/airflow/providers/ssh/operators/ssh.py", line 171, in execute
    raise AirflowException(f"SSH operator error: {str(e)}")
airflow.exceptions.AirflowException: SSH operator error: 

--- edit ---

generate_report = SSHOperator(
    task_id = 'generate_report',
    ssh_conn_id = 'ssh_123',
    command = curl -u user:password "http://localhost:1234/path/to/trigger/report_creation_API?async=false",
)


from Airflow SSHOperator's Socket exception: Bad file descriptor

No comments:

Post a Comment