Monday, 17 January 2022

Docker containers randomly shutting down

So I've had this system where I had 3 docker containers running at the same time to host my APIs, one for traefik, and the other two for different APIs (written in python using FastAPI). It was working fine for a while, however now all the docker containers seem to be randomly shutting down after a couple hours at the same.

This is the error message all of them output:


Here is my docker-compose for traefik:
Exception in thread Thread-5:
Traceback (most recent call last):
  File "urllib3/connectionpool.py", line 677, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1277, in request
  File "http/client.py", line 1323, in _send_request
  File "http/client.py", line 1272, in endheaders
  File "http/client.py", line 1032, in _send_output
  File "http/client.py", line 972, in send
  File "docker/transport/unixconn.py", line 43, in connect
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "requests/adapters.py", line 449, in send
  File "urllib3/connectionpool.py", line 727, in urlopen
  File "urllib3/util/retry.py", line 410, in increment
  File "urllib3/packages/six.py", line 734, in reraise
  File "urllib3/connectionpool.py", line 677, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1277, in request
  File "http/client.py", line 1323, in _send_request
  File "http/client.py", line 1272, in endheaders
  File "http/client.py", line 1032, in _send_output
  File "http/client.py", line 972, in send
  File "docker/transport/unixconn.py", line 43, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "threading.py", line 926, in _bootstrap_inner
  File "threading.py", line 870, in run
  File "compose/cli/log_printer.py", line 168, in tail_container_logs
  File "compose/cli/log_printer.py", line 185, in wait_on_exit
  File "compose/container.py", line 268, in wait
  File "docker/utils/decorators.py", line 19, in wrapped
  File "docker/api/container.py", line 1305, in wait
  File "docker/utils/decorators.py", line 46, in inner
  File "docker/api/client.py", line 233, in _post
  File "requests/sessions.py", line 578, in post
  File "requests/sessions.py", line 530, in request
  File "requests/sessions.py", line 643, in send
  File "requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

Here is my docker-compose for my first API (call it "Hierarchy"):

services:

  backend:
    build: ./
    restart: always
    labels:
      - traefik.enable=true

      - traefik.http.services.hierarchy_app.loadbalancer.server.port=80

      - traefik.http.routers.hierarchy-http.entrypoints=http
      - traefik.http.routers.hierarchy-http.rule=Host(`api.mydomain.me`)
      - traefik.docker.network=traefik-public

      - traefik.http.routers.hierarchy-https.entrypoints=https
      - traefik.http.routers.hierarchy-https.rule=Host(`api.mydomain.me`)
      - traefik.http.routers.hierarchy-https.tls=true

      - traefik.http.routers.hierarchy-https.tls.certresolver=le

      - traefik.http.middlewares.https-redirect.redirectscheme.scheme=https
      - traefik.http.middlewares.https-redirect.redirectscheme.permanent=true

      - traefik.http.routers.hierarchy-http.middlewares=https-redirect

    networks:
      - traefik-public

    volumes:
      - ${PWD}/hierarchy_app/commands.json:/hierarchy_app/commands.json:Z

networks:
  traefik-public:
    external: true

Here is my docker-compose for my second API (call it "Voicerooms"):

services:

  backend:
    build: ./
    restart: always
    labels:
      - traefik.enable=true

      - traefik.http.services.voicerooms_app.loadbalancer.server.port=80

      - traefik.http.routers.voicerooms-http.entrypoints=http
      - traefik.http.routers.voicerooms-http.rule=Host(`api.mydomain2.app`)
      - traefik.docker.network=traefik-public

      - traefik.http.routers.voicerooms-https.entrypoints=https
      - traefik.http.routers.voicerooms-https.rule=Host(`api.mydomain2.app`)
      - traefik.http.routers.voicerooms-https.tls=true

      - traefik.http.routers.voicerooms-https.tls.certresolver=le

      - traefik.http.middlewares.https-redirect.redirectscheme.scheme=https
      - traefik.http.middlewares.https-redirect.redirectscheme.permanent=true

      - traefik.http.routers.voicerooms-https.middlewares=https-redirect
    networks:
      - traefik-public

    volumes:
      - ${PWD}/voicerooms_app/commands.json:/voicerooms_app/commands.json:Z

networks:
  traefik-public:
    external: true

Any help would be greatly appreciated, I've been trying to figure out this problem for a while :)

Edit: After scrolling up a bit more, I found this in the error message:

backend_1  | [2022-01-10 02:04:51 +0000] [1] [INFO] Handling signal: term
backend_1  | [2022-01-10 02:04:51 +0000] [7] [INFO] Shutting down
backend_1  | [2022-01-10 02:04:51 +0000] [8] [INFO] Shutting down

I'm assuming my docker container is getting terminated for some reason, what's going on here?



from Docker containers randomly shutting down

No comments:

Post a Comment