c# - IIS impersonation. Cannot get user principal when authenticated from outside network

Our WebApi is using Windows Authentication Impersonation. When user connects to application I am getting user information as:

        var ctx = new PrincipalContext(ContextType.Domain, System.Environment.UserDomainName.ToUpper());
        var userInfo = UserPrincipal.FindByIdentity(ctx, System.Environment.UserName.ToUpper());
        // Some logic to detect which AD groups user belongs to so we can check which areas of application user can access to

It works flawless when user is accessing the site within our network, however when user attempts to access application within outside network and specifying proper credentials I am receiving exception when it executes UserPrincipal.FindByIdentity:

An operations error occurred.\r\n","type":"System.DirectoryServices.DirectoryServicesCOMException","stacktrace":" at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)\r\n at System.DirectoryServices.DirectoryEntry.Bind()\r\n at System.DirectoryServices.DirectoryEntry.get_AdsObject()\r\n at System.DirectoryServices.PropertyValueCollection.PopulateList()\r\n at System.DirectoryServices.PropertyValueCollection..ctor(DirectoryEntry entry, String propertyName)\r\n at System.DirectoryServices.PropertyCollection.get_Item(String propertyName)\r\n at System.DirectoryServices.AccountManagement.PrincipalContext.DoLDAPDirectoryInitNoContainer()\r\n at System.DirectoryServices.AccountManagement.PrincipalContext.DoDomainInit()\r\n at System.DirectoryServices.AccountManagement.PrincipalContext.Initialize()\r\n at System.DirectoryServices.AccountManagement.PrincipalContext.get_QueryCtx()\r\n at System.DirectoryServices.AccountManagement.Principal.FindByIdentityWithTypeHelper(PrincipalContext context, Type principalType, Nullable`1 identityType, String identityValue, DateTime refDate)\r\n at System.DirectoryServices.AccountManagement.Principal.FindByIdentityWithType(PrincipalContext context, Type principalType, String identityValue)\r\n at System.DirectoryServices.AccountManagement.UserPrincipal.FindByIdentity(PrincipalContext context, String identityValue)\

I can see that values at System.Environment.UserDomainName and System.Environment.UserName are proper domain name and user name.