Tuesday, 21 May 2019

Why would I get a Forbidden message from AWS API Gateway, even though things are working internally?

I have AWS API gateway setup for a public endpoint with no auth. It connects to a websocket that triggers a Lambda.

I was creating connections with Python's websocket-client lib at https://pypi.org/project/websocket_client/.

I noticed that connections would fail ~10% of the time, and get worse as I increased load. I can't find anywhere that would be throttling me seeing as my general API Gateway settings say Your current account level throttling rate is 10000 requests per second with a burst of 5000 requests.. That’s beside the point that just 2-3 requests per second would trigger issue fairly often.

Meanwhile the failure response would be like {u'message': u'Forbidden', u'connectionId': u'Z2Jp-dR5vHcCJkg=', u'requestId': u'Z2JqAEJRvHcFzvg='}

I went into my CloudWatch log insights and searched for the connection ID and request ID. The log group for the API gateway would find no results with either ID. Yet a search on my Lambda that fires on websocket connect, would have a log with that connection ID. The log showed everything running as expected on our side. The lambda simply runs a MySQL query that fires.

Why would I get a response of forbidden, despite the lambda working as expected?

The existing question over at getting message: forbidden reply from AWS API gateway, seems to address if it's ALWAYS returning forbidden for some private endpoints. Nothing lined up with my use case.

UPDATE

I think this may be related to locust.io, or python, which I'm using to connect every second. I installed https://www.npmjs.com/package/wscat on my machine and am connecting and closing as fast as possible repeatedly. I am not getting a Forbidden message. It's just extra confusing since I'm not sure how the way I connect would randomly spit back a Forbidden message some of the time.

class SocketClient(object):
    def __init__(self, host):
        self.host = host
        self.session_id = uuid4().hex

    def connect(self):
        self.ws = websocket.WebSocket()
        self.ws.settimeout(10)
        self.ws.connect(self.host)

        events.quitting += self.on_close

        data = self.attach_session({})
        return data

    def attach_session(self, payload):
        message_id = uuid4().hex
        start_time = time.time()
        e = None
        try:
            print("Sending payload {}".format(payload))
            data = self.send_with_response(payload)
            assert data['mykey']

        except AssertionError as exp:
            e = exp
        except Exception as exp:
            e = exp
            self.ws.close()
            self.connect()
        elapsed = int((time.time() - start_time) * 1000)
        if e:
            events.request_failure.fire(request_type='sockjs', name='send',
                                        response_time=elapsed, exception=e)
        else:
            events.request_success.fire(request_type='sockjs', name='send',
                                        response_time=elapsed,
                                        response_length=0)
        return data

    def send_with_response(self, payload):
        json_data = json.dumps(payload)

        g = gevent.spawn(self.ws.send, json_data)
        g.get(block=True, timeout=2)
        g = gevent.spawn(self.ws.recv)
        result = g.get(block=True, timeout=10)

        json_data = json.loads(result)
        return json_data
    def on_close(self):
        self.ws.close()

class ActionsTaskSet(TaskSet):
    @task
    def streams(self):
        response = self.client.connect()
        logger.info("Connect Response: {}".format(response))

class WSUser(Locust):
    task_set = ActionsTaskSet
    min_wait = 1000
    max_wait = 3000

    def __init__(self, *args, **kwargs):
        super(WSUser, self).__init__(*args, **kwargs)
        self.client = SocketClient('wss://mydomain.amazonaws.com/endpoint')

enter image description here



from Why would I get a Forbidden message from AWS API Gateway, even though things are working internally?

No comments:

Post a Comment