Table ImageColumn defaultImageUrl with a many-to-many relationship

Is it possible to use defaultImageUrl on a Table ImageColumn that is stacked on a many-to-many relationship? For example, if I have a Pet and User table and the Pet can belong to many Users, how can I display a default image for each User that doesn't have an avatar? When I use defaultImageUrl on the ImageColumn it displays the default for the pet, not each user. Thanks!