In my app I have some Users and Groups, that are related by a many to many relation through GroupMember.

class User(models.Model):
    ...

class Group(models.Model):
    ...
    users = models.ManyToManyField(User, through='groups.GroupMember', related_name='groups')


class GroupMember(models.Model):
    ...
    user = models.ForeignKey(User, related_name='group_members', on_delete=models.CASCADE)
    group = models.ForeignKey(Group, related_name='members', on_delete=models.CASCADE)

Now I am trying to build an endpoint that retrieves a list of user instances that are in any one of the groups that your user is in. So if user A and B are in group1, user A and C are in group2, and C and D are in group3, retrieving the users as A would return B and C (since those are in groups with A).

That filter part can be done in multiple ways, but the part where I am stuck is how to show which groups were the cause of the relation. I tried to use prefetch_related with a custom filter, to fetch only the groups that have user A in them, like this:

User.objects.exclude(id=user.id).annotate(
    # Annotate the amount of groups related between the users
    related_group_count=Count('groups', filter=Q(groups__users=user))
).filter(
    # Only users that have at least 1 relation with this user
    related_group_count__gt=0
).prefetch_related(
    # Prefetch the specific groups to show
    Prefetch('groups', User.groups.all(), to_attr='related_groups'),
)

The returned users will not have any related_groups, it will be an empty qs. When I include the request user (by removing the exclude part on the query), the groups will be available for that user, but still not any other. This is my current response (including the requesting user "A"):

{
    "first_name": "A",
    "related_groups": [
        {
            "name": "group1",
        },
        {
            "name": "group2",
        }
    ]
},
{
    "first_name": "B",
    "related_groups": []
},
{
    "first_name": "C",
    "related_groups": []
}

But I'm trying to achieve this:

{
    "first_name": "B",
    "related_groups": [
        {
            "name": "group1",
        }
    ]
},
{
    "first_name": "C",
    "related_groups": [
        {
            "name": "group2",
        }
    ]
}

The problem lies with the filter on the Prefetch queryset, but I can't figure out what I'm doing wrong or if this is even possible. I tried different way of filtering (also with Group.objects.filter(users=user)) or even annotating the group members, counting if the given user is in that group and filtering on that, but with no success...