I'm running PowerDNS recursor inside my k8s cluster. My python script is on a different pod
that is doing rdns to my powerdns
rescursor
app. I have my hpa Max replica
set to 8
. However, I do not think the load is the problem here. I'm unsure what to do to resolve this timeout error that I'm getting below. I can increase the replicas to solve the problem temporarily, and then it would happen again.
[ipmetadata][MainThread][source.py][144][WARNING]: dns_error code=12, message=Timeout while contacting DNS servers
It seems like my pods are rejecting incoming traffic therefore it's outputting the dns_error code=12.
Here is part of my script that's running the rdns
return_value = {
'rdns': None
}
try:
async for attempt in AsyncRetrying(stop=stop_after_attempt(3)):
with attempt:
try:
if ip:
result = await self._resolver.query(ip_address(ip).reverse_pointer, 'PTR')
return_value['rdns'] = result.name
return return_value
except DNSError as dns_error:
# 1 = DNS server returned answer with no data
# 4 = Domain name not found
# (seems to just be a failure of rdns lookup no sense in retrying)
# 11 = Could not contact DNS servers
if int(dns_error.args[0]) in [1, 4, 11]:
return return_value
LOG.warning('dns_error code=%d, message=%s, ip=%s', dns_error.args[0], dns_error.args[1], ip)
raise
except RetryError as retry_ex:
inner_exception = retry_ex.last_attempt.exception()
if isinstance(inner_exception, DNSError):
# 12 = Timeout while contacting DNS servers
LOG.error('dns_error code=%d, message=%s, ip=%s', inner_exception.args[0], inner_exception.args[1], ip)
else:
LOG.exception('rnds lookup failed')
return return_value
from Timeout Issues inside Kubernetes Cluster Powerdns
No comments:
Post a Comment