Nginx taking way more time than uwsgi

We are running nginx and uwsgi behind Load Balancer (AWS ELB). uwsgi is able to process request in lesses than 100 ms but nginx is adding some overhead and final request time is becoming way more than uwsgi time. This is not happening for all of the requests but for only 2-3 % of requests.

Log Format

'$status [$time_local] "$request" $body_bytes_sent $request_length $request_time $upstream_response_time $upstream_connect_time $upstream_header_time $upstream_status $pipe';


200 [21/Sep/2022:11:46:46 +0000] "POST api_end_point HTTP/1.1" 7238 1546 24.848 0.036 0.000 0.036 200 .

200 [21/Sep/2022:11:46:57 +0000] "POST api_end_point HTTP/1.1" 1130 1558 2.178 0.044 0.000 0.040  200 .

200 [21/Sep/2022:11:46:56 +0000] "POST api_end_point HTTP/1.1" 1130 1565 10.212 0.028 0.000 0.024  200 .

Log 1: upstream request time is 36 ms and upstream connect time is 0 but nginx request time is 24.848 seconds.

Log 2: upstream request time is 44 ms and upstream connect time is 0 but nginx request time is 2.178 seconds.

Log 3: upstream request time is 28 ms and upstream connect time is 0 but nginx request time is 10.212 seconds.

Nginx Config:

error_log  /var/log/nginx/error.log info;
worker_processes  auto;
worker_rlimit_nofile 30000;

events {
   worker_connections  1000;
   use epoll;
   multi_accept on;
}

http {
   include       /etc/nginx/mime.types;
   default_type  application/json;
   client_max_body_size 5m;
   client_body_buffer_size      256k;
   client_header_buffer_size    1k;
   large_client_header_buffers  8 64k;
   client_header_timeout  1m;
   client_body_timeout    2m;
   send_timeout           2m;
   reset_timedout_connection on;
   server_tokens off;

   sendfile        on;
   tcp_nopush      on;
   tcp_nodelay     on;

   keepalive_timeout  200;
   keepalive_requests  2000;

   log_format mycombined '$status [$time_local] "$request" $body_bytes_sent $request_length $request_time $upstream_response_time $upstream_connect_time $upstream_header_time $upstream_status $pipe';
   map $status $loggable {
       ~^[23]  0;
       default 1;
   }

   include /etc/nginx/conf.d/*.conf;
}


server {
    listen       our_custom_port;
    server_name  localhost;
    access_log /var/log/nginx/access.log mycombined;
    location api_end_point {
         include uwsgi_params;
         uwsgi_read_timeout  5s;
         uwsgi_pass  unix:/opt/apis/server/uwsgi_socket.sock;
         uwsgi_ignore_client_abort on;
    }
    location /_health {
        include uwsgi_params;
        uwsgi_read_timeout  5s;
        uwsgi_pass  unix:/opt/apis/server/uwsgi_socket.sock;
        uwsgi_ignore_client_abort on;
   }
}

I tried using gunicorn also instead of uwsgi. Similar issue persists with nginx + gunicorn also.

Any help would be highly appreciated.

from Nginx taking way more time than uwsgi