Aggregating a subquery with multiple OuterRefs in Django

For the following models:

class Merchant(models.Model):
    ...

class Batch(models.Model):
    merchant = models.ForeignKey(Merchant, ...)
    ...

class Tab(models.Model):
    is_void = models.BooleanField()
    ...

class Payment(models.Model):
    batch = models.ForeignKey(Batch, ...)
    tab = models.ForeignKey(Tab, ...)
    payment = models.ForeignKey("self", ...)
    payment_type = models.CharField(choices=..., ...)
    status = models.CharField(choices=..., ...)
    amount = models.FloatField()

I'm trying to solve an N+1 selects issue. Currently, the code looks something like this, somewhat simplified - the code is really in a DRF serializer and does some other things:

for payment in Payment.objects.filter(...):
    refunds_total = Payment.objects.filter(
        batch__merchant=obj.batch.merchant,
        payment=payment,
        payment_type="refund",
        status="success",
    ).exclude(
        tab__is_void=True,
    ).aggregate(
        total=Sum('amount')
    ).get('total', 0.0)
    # Do something with refunds_total.

The performance is really bad. What I'd like to do is something like this:

refunds = Payment.objects.filter(
    batch__merchant=OuterRef("batch__merchant"),
    payment=OuterRef("id"),
    payment_type="refund",
    status="success",
).exclude(
    tab__is_void=True
).aggregate(total=Coalesce(Sum("amount"), 0.0))
    ​
payments = Payment.objects.filter(
    ...
).annotate(total_refunds=Subquery(refunds))

But Django doesn't allow aggregating in a subquery, as aggregating immediately evaluates the QuerySet. I've tried two approaches. The first is following this documentation. However, I'm not sure how I need to put the query together as there are multiple OuterRefs required. My first shot at a subquery looks like this:

refunds = CustomerPayment.objects.filter( 
    batch__merchant=OuterRef("batch__merchant"), 
    payment=OuterRef("id"), 
    payment_type="refund", 
    status="success", 
).exclude( 
    tab__is_void=True 
).order_by().values( 
    "batch__merchant", "payment", 
).annotate(total_refunds=Coalesce(Sum("amount"), 0.0)).values("amount")

But however I try to get it working, it always returns multiple rows rather than single row with the total, so I can't use it in a subquery.

I've also tried making my own Subquery subclass but I couldn't put together anything that worked.

from Aggregating a subquery with multiple OuterRefs in Django