Monday, 24 October 2022

Timeout Issues inside Kubernetes Cluster Powerdns

I'm running PowerDNS recursor inside my k8s cluster. My python script is on a different pod that is doing rdns to my powerdns rescursor app. I have my hpa Max replica set to 8. However, I do not think the load is the problem here. I'm unsure what to do to resolve this timeout error that I'm getting below. I can increase the replicas to solve the problem temporarily, and then it would happen again.

[ipmetadata][MainThread][source.py][144][WARNING]: dns_error code=12, message=Timeout while contacting DNS servers

It seems like my pods are rejecting incoming traffic therefore it's outputting the dns_error code=12.

Here is part of my script that's running the rdns

        return_value = {
            'rdns': None
        }
        try:
            async for attempt in AsyncRetrying(stop=stop_after_attempt(3)):
                with attempt:
                    try:
                        if ip:
                            result = await self._resolver.query(ip_address(ip).reverse_pointer, 'PTR')
                            return_value['rdns'] = result.name
                        return return_value
                    except DNSError as dns_error:
                        # 1  = DNS server returned answer with no data
                        # 4  = Domain name not found
                        # (seems to just be a failure of rdns lookup no sense in retrying)
                        # 11 = Could not contact DNS servers
                        if int(dns_error.args[0]) in [1, 4, 11]:
                            return return_value
                        LOG.warning('dns_error code=%d, message=%s, ip=%s', dns_error.args[0], dns_error.args[1], ip)
                        raise

        except RetryError as retry_ex:
            inner_exception = retry_ex.last_attempt.exception()
            if isinstance(inner_exception, DNSError):
                # 12 = Timeout while contacting DNS servers
                LOG.error('dns_error code=%d, message=%s, ip=%s', inner_exception.args[0], inner_exception.args[1], ip)
            else:
                LOG.exception('rnds lookup failed')
            return return_value



from Timeout Issues inside Kubernetes Cluster Powerdns

No comments:

Post a Comment