Managed identity doesn't allow permissions to join subnet, but it's owner on that vnet/subnet

I have an Azure managed application, deployed from the Azure Marketplace with an ARM template and with a managed identity assigned to a VM, that lives in the managed resource group. The managed identity has Owner permissions over the resource group so it should be able to do anything - I can create/delete vnets/subets with az cli freely.

This dfa2e8dd-df89-40e2-bbb2-9b77cdcb4d7c is the VM object managed identity.

Now I am trying to add backend servers to an application gateway backend pool using:

az network application-gateway address-pool create --gateway-name $APP_GATEWAY_NAME -n mybackendpooltest -g $RG --servers $IPS

What I am getting back is:

(ApplicationGatewayInsufficientPermissionOnSubnet) Client with object id dfa2e8dd-df89-40e2-bbb2-9b77cdcb4d7c does not have permission on the Virtual Network resource /subscriptions/XXXXXXXXXXXXX/resourceGroups/YYYYYYYYYYYYYY/providers/Microsoft.Network/virtualNetworks/AppGatewayVnet/subnets/AppGatewaySubnet to perform action Microsoft.Network/virtualNetworks/subnets/join/action. For details on the required permissions, please visit https://aka.ms/agsubnetjoin.

When I run the same command with the external tenant that has full access over the managed application - the command succeeds. I have the same permissions as the external object (Owner) of the resource group and all resources.

I tried manually assigning this system managed identity explicit access over the Vnet and Subnet with:

• "Network Contributor" role - no help
• "Domain Services Contributor" - no help
• "Owner" role - no help

The reason I am adding the backend servers later on are that during the ARM template deployment the machines are not known yet and are created afterwards.

What can I do to unblock this?