django rest framework - backward serialization to avoid prefetch_related

I have two models, Item and ItemGroup:

class ItemGroup(models.Model):
   group_name = models.CharField(max_length=50)
   # fields..

class Item(models.Model):
   item_name = models.CharField(max_length=50)
   item_group = models.ForeignKey(ItemGroup, on_delete=models.CASCADE)
   # other fields..

I want to write a serializer that will fetch all item groups with their item list as a nested array.

So I want this output:

[ {group_name: "item group name", "items": [... list of items ..] }, ... ]

As I see, I should write this with django rest framework:

class ItemGroupSerializer(serializers.ModelSerializer):
   class Meta:
      model = ItemGroup
      fields = ('item_set', 'group_name') 

Means, I have to write a serializer for ItemGroup (not for Item). To avoid many queries I pass this queryset:

ItemGroup.objects.filter(**filters).prefetch_related('item_set')

The problem that I see is, for a large dataset, prefetch_related results in an extra query with a VERY large sql IN clause, which I could avoid with the query on the Item objects instead:

Item.objects.filter(**filters).select_related('item_group')

Which results in a JOIN which is way better.

Is it possible to query Item instead of ItemGroup, and yet to have the same serialization output?

from django rest framework - backward serialization to avoid prefetch_related