Error when selecting new SSL certificate for SSTP VPN in RRAS – Web Listener HTTP.sys

The certificate used for Secure Socket Tunneling Protocol (SSTP) is different than the certificate bound to the SSL (web listner, HTTP.sys).

Recently I had to install a new SSL certificate in a server that was an SSTP VPN server. This server is running Windows Server 2012 R2 Essentials. The server was previously configured for Anywhere Access, but we decided to utilize SSTP VPN instead.

I installed the new certificate and private key to the machine’s local certificate store. Upon trying to change the SSL certificate, the RRAS server properties would return the following error:

sstp_cert1

“The certificate used for Secure Socket Tunneling Protocol (SSTP) is different than the certificate bound to the SSL (web listner, HTTP.sys). Configure SSTP to use the default certificate or the certificate bound to SSL. You can configure web server application to use the same certificate used by SSTP.”

After poking around, this seems to be happening due to the old expired certificate being assigned to the IIS Default Web Site. We will need to change that assignment and then we’ll be able to change the certificate that RRAS uses for SSTP connections.

To assign the new certificate that we’ve already installed, we’ll need to open IIS management, and in the left-hand column, we’ll need to navigate to the Server name > Sites > Default Web Site, then click Edit Site > Bindings in the right-hand column. sstp_cert2

Select the HTTPS binding and click the Edit… button.

sstp_cert3

For SSL certificate, it should list the old expired certificate. We want to have it use the new certificate we’ve installed. To do this, click the “Select” button.

sstp_cert4

Select the new certificate and click the OK button.

sstp_cert5

The certificate is now selected for this site binding. Click OK.

sstp_cert6

Click the Close button to close the Site Bindings window.

sstp_cert7

In the right-hand column, under Manage Website, click Restart.

sstp_cert8

Close the IIS management window and go back to the Routing and Remote Access MMC window. Right click the server name in the left-hand column and click Properties.

sstp_cert9

Upon clicking the Security tab where we can specify which SSL certificate is used for SSTP VPN, you’ll notice that we get the same error as we did when we tried changing to the new certificate.

sstp_cert10

After clicking OK in that dialog box, we will now be able to select our new certificate.

sstp_cert11

Once the new certificate has been selected, click Apply.

sstp_cert12

After clicking Apply, the RRAS service will want to restart. Click Yes to let the service restart.

sstp_cert13

The router will restart.

sstp_cert14

Once the service restarts, click OK to close the Properties window. sstp_cert15

Now, the new certificate should be applied to RRAS. Test SSTP VPN connection to the server. It should work!

Windows Server Backup failing to back up one virtual machine – 0x80070050

Windows Server Backup failing with error code 0x80070050

This particular machine has a pretty simple backup setup: a Windows Server 2012 R2 host with two 2012 R2 guest VMs. Integration services are installed on the two VMs and Backup (volume checkpoint) is enabled on both VMs in the integration services settings. Windows Server Backup is installed on the host, and it’s configured to back up the VMs and host system state to an external drive once daily.

This server was not completing backups for one virtual machine. The other virtual machine was backing up properly on this host.

WSBEC1

I took a look at the event logs to find any clues as to why the backup might be failing for this particular VM. One particular event stood out for me.

WSBEC2

Log Name: Microsoft-Windows-Hyper-V-Worker-Admin
Source: Microsoft-Windows-Hyper-V-Worker
Date: 4/17/2018 9:00:43 PM
Event ID: 3280
Task Category: None
Level: Error
Keywords: 
User: NT VIRTUAL MACHINE\94FEE3F0-459E-498D-96E4-BF8272BAC254
Computer: HV01
The description for Event ID 3280 from source Microsoft-Windows-Hyper-V-Worker cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

DC2
94FEE3F0-459E-498D-96E4-BF8272BAC254
%%2147942480
0x80070050

The locale specific resource for the desired message is not present

Unfortunately, the event log didn’t give much information other than some error codes at the bottom of the event. Looking at the 0x80070050 error, that code means “The file exists”. (to find that, you can either Google the code, or you can use this guide to find out what the code means).

To figure out why “The file exists” matters, we need to understand how the backup process works when a backup is taken of a VM. Very roughly speaking, when a backup is created of a virtual machine, the host creates a checkpoint of the VM. The checkpoint creates a differencing disk (AVHDX) linked to the original VHDX. The parent VHDX is then backed up/copied. Once the backup is completed, the host merges the differencing disk back into the parent VHDX, and eliminates the checkpoint.

When Windows Server backup tries to create a checkpoint of the VM, it creates a differencing disk (AVHDX) which is the name of the original VHDX file with “-AutoRecovery” appended to the end of the file name.

In this particular case, there was an old AVHDX file that existed for that VM already. That must be what’s causing our “The file exists” error!

WSBEC3

Before we just get rid of it, I went to the HyperV settings of the VM to see if it was in use.

WSBEC4

If the AVHDX were in use, we would see it here instead of the regular VHDX file. The AVHDX file was not in use on this VM, and that AVHDX file had not been modified in a long time, so I deemed it safe to rename/delete it. After I deleted the file, I let the backup schedule run its course.

Veeam Agent failing to back up to Synology NAS – Invalid Signature

I came across a Windows Server 2016 VM that had Veeam Agent installed for backups. The backup destination was a Synology NAS dedicated for backups running DSM 6.1.5-15254 Update 1 (most current version at the time of writing this). The backups were not completing, and in Event Viewer, each failure would log the following event:

veeamfail1

Veeam Agent 'Backup Job FS1' finished with Error and will be retried.

Job details: Error: Shared memory connection was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: Invalid Signature.
Failed to write data to the file [\\nas\backups\FS1\Backup Job FS1\Backup Job FS12018-03-11T203041.vbk].
Failed to download disk.

The Invalid Signature part stuck out to me. I started looking into SMB signing and found this Microsoft page: https://support.microsoft.com/en-us/help/2686098/system-error-2148073478-extended-error-or-invalid-signature-error-mess

The article states that some third party file servers are not compatible with SMB signing. The SMB client expects a signed response, but the 3rd party file server never gives it to the client, so the transfer fails.

I logged into the Synology NAS, went to Control Panel > File Services > SMB > Advanced settings.

synologysmb1

Transport Encryption mode was set to Force, so I set it to Auto. Now the Veeam Agent backups are completing successfully!

IPv6 DNS record being published on local AD DNS

I recently was working on a domain controller migration from a SBS 2008 to a 2008 R2 machine. I promoted the new domain controller successfully. Before switching the clients to use the IP of the new DC as their DNS servers, I did my usual checks of the DNS records and made sure that the new DC was returning DNS records properly.

I started by running nslookup and setting the server to my new DC. From there, I did a DNS lookup for the local domain name. Normally, this should return the IPs of the DCs. When I did this, the result returned the local IPs of the DCs, but also returned an IPv6 address starting with 2002:. This isn’t good, since any clients trying to update their group policy may resolve the local domain to that IPv6 address, which could cause some problems down the road.

I checked the IPv6 settings for each NIC on both DCs, but none of the NICs had public IPv6 addresses assigned, just link local addresses (starting with fe80:). I did an ipconfig /all on one server, and I found that that IPv6 address was coming from a Microsoft 6to4 tunneling adapter. The IPv6 address that was returned by nslookup was the same as the 6to4 adapter. Now, why was this 6to4 adapter enabled in the first place?

I found this article: https://blogs.technet.microsoft.com/askpfeplat/2013/11/17/ipv6-for-the-windows-administrator-the-2002-6to4-tunnel-address-and-its-impact/

The article states that whenever a machine has a public IPv4 address assigned to its NIC, it will generate a 6to4 tunneling address. Sure enough, the machine that I was working on here had a 2nd NIC which was not connected to the network, and that NIC had a static public IP assigned to its interface. The IP was from this client’s previous internet service and was no longer needed. I changed the IPv4 configuration from static to DHCP. After that, I ran an ipconfig /registerdns from an elevated command prompt, which got rid of that IPv6 address from AD’s DNS. Woo!

Site to Site IPSec VPN Tunnel between two Ubiquiti EdgeRouters

I recently needed to configure an IPSec VPN tunnel between two Ubiquiti EdgeRouters. I decided to write a post describing my setup process from start to finish.

Intro/Background

I recently needed to configure an IPSec VPN tunnel between two Ubiquiti EdgeRouters. I did some searching, and the documentation was lacking or incomplete. I decided to write a post describing my setup process from start to finish.

In my test setup, I’ve got two Ubiquiti EdgeRouters. One is an EdgeRouter Lite, the other is an EdgeRotuer X, both on firmware 1.9.1 (the most current firmware as of when this was written). Both routers have external IPs on their WAN interfaces, and are not behind any other NAT. Both routers have been configured with the basic WAN+2LAN wizards, which adds a basic stateful firewall and NAT masquerade on the WAN interface. R1 has an external IP of 1.1.1.1 on eth0, and an internal subnet of 192.168.1.0/24. R2 has an external IP of 2.2.2.2 on eth1, and an internal subnet on 10.1.1.0/24. In the real world, you would ideally have static external IPs. This might work using hostnames with dynamic IPs, but I haven’t tested it.

Configuring the tunnel

I found this guide on Ubiquiti’s website, which shows some CLI commands to run. The screenshots of the configurations on that page must be dated though, because some of the commands aren’t correct.

I’m going to start by configuring the tunnels in the routers. SSH into the routers, and apply these commands:

For R1:

configure
set vpn ipsec esp-group vpntunnel
set vpn ipsec esp-group vpntunnel compression disable
set vpn ipsec esp-group vpntunnel proposal 1
set vpn ipsec esp-group vpntunnel proposal 1 encryption aes128
set vpn ipsec esp-group vpntunnel proposal 1 hash sha1
set vpn ipsec ike-group vpntunnel
set vpn ipsec ike-group vpntunnel proposal 1
set vpn ipsec ike-group vpntunnel proposal 1 encryption aes128
set vpn ipsec ike-group vpntunnel proposal 1 hash sha1
set vpn ipsec ipsec-interfaces interface eth0
set vpn ipsec site-to-site peer 2.2.2.2
set vpn ipsec site-to-site peer 2.2.2.2 local-address 1.1.1.1 
set vpn ipsec site-to-site peer 2.2.2.2 authentication mode pre-shared-secret
set vpn ipsec site-to-site peer 2.2.2.2 authentication pre-shared-secret SuperSecretKey123
set vpn ipsec site-to-site peer 2.2.2.2 default-esp-group vpntunnel
set vpn ipsec site-to-site peer 2.2.2.2 ike-group vpntunnel
set vpn ipsec site-to-site peer 2.2.2.2 tunnel 1
set vpn ipsec site-to-site peer 2.2.2.2 tunnel 1 esp-group vpntunnel
set vpn ipsec site-to-site peer 2.2.2.2 tunnel 1 local prefix 192.168.1.0/24 
set vpn ipsec site-to-site peer 2.2.2.2 tunnel 1 remote prefix 10.1.1.0/24
commit
exit

For R2:

configure
set vpn ipsec esp-group vpntunnel
set vpn ipsec esp-group vpntunnel compression disable
set vpn ipsec esp-group vpntunnel proposal 1
set vpn ipsec esp-group vpntunnel proposal 1 encryption aes128
set vpn ipsec esp-group vpntunnel proposal 1 hash sha1
set vpn ipsec ike-group vpntunnel
set vpn ipsec ike-group vpntunnel proposal 1
set vpn ipsec ike-group vpntunnel proposal 1 encryption aes128
set vpn ipsec ike-group vpntunnel proposal 1 hash sha1
set vpn ipsec ipsec-interfaces interface eth1
set vpn ipsec site-to-site peer 1.1.1.1
set vpn ipsec site-to-site peer 1.1.1.1 local-address 2.2.2.2 
set vpn ipsec site-to-site peer 1.1.1.1 authentication mode pre-shared-secret
set vpn ipsec site-to-site peer 1.1.1.1 authentication pre-shared-secret SuperSecretKey123
set vpn ipsec site-to-site peer 1.1.1.1 default-esp-group vpntunnel
set vpn ipsec site-to-site peer 1.1.1.1 ike-group vpntunnel
set vpn ipsec site-to-site peer 1.1.1.1 tunnel 1
set vpn ipsec site-to-site peer 1.1.1.1 tunnel 1 esp-group vpntunnel
set vpn ipsec site-to-site peer 1.1.1.1 tunnel 1 local prefix 10.1.1.0/24 
set vpn ipsec site-to-site peer 1.1.1.1 tunnel 1 remote prefix 192.168.1.0/24
commit
exit

Now that we have committed the changes and we have exited configuration mode, we’ll run show vpn ipsec status to check the status of the tunnel on R2.

show vpn ipsec status

IPSec Process Running PID: 11410
0 Active IPsec Tunnels
IPsec Interfaces :

Configuring the WAN_IN firewall to allow IPSec traffic

The tunnel isn’t up! What we have to do now is configure the WAN_LOCAL firewall on both routers to allow IPSec traffic in to the router. Make sure that the firewall rule numbers you configure are higher priority (smaller number) than the default “Drop invalid state” rule.

configure

set firewall name WAN_LOCAL rule 15 
set firewall name WAN_LOCAL rule 15 action accept
set firewall name WAN_LOCAL rule 15 description AllowIPSec
set firewall name WAN_LOCAL rule 15 log disable
set firewall name WAN_LOCAL rule 15 protocol udp
set firewall name WAN_LOCAL rule 15 destination port 500,4500

set firewall name WAN_LOCAL rule 16 
set firewall name WAN_LOCAL rule 16 action accept
set firewall name WAN_LOCAL rule 16 description AllowESP
set firewall name WAN_LOCAL rule 16 log disable
set firewall name WAN_LOCAL rule 16 protocol esp

commit
exit

Now that I’ve applied the above rules to both routers, let’s try that show vpn ipsec status command again.

show vpn ipsec status

IPSec Process Running PID: 11140
1 Active IPsec Tunnels
IPsec Interfaces :

The tunnel is now up! Let’s try to ping the local IP of R1 from R2.

ping 192.168.1.1

PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
--- 192.168.1.1 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5008ms

No responses were received, so I interrupted the pings with control-c. The requests timed out. Let’s see if the traffic is actually traversing the tunnel. We’ll check this by running show vpn ipsec sa on R2.

show vpn ipsec sa

peer-1.1.1.1-tunnel-1: #2, ESTABLISHED, IKEv1, bb126d66e9ca4bdf:f2fccad0bcf04771
  local  '2.2.2.2' @ 2.2.2.2
  remote '1.1.1.1' @ 1.1.1.1
  AES_CBC-128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
  established 246s ago, reauth in 27505s
  peer-1.1.1.1-tunnel-1: #1, INSTALLED, TUNNEL, ESP:AES_CBC-128/HMAC_SHA1_96/MODP_1024
    installed 246 ago, rekeying in 2298s, expires in 3354s
    in  caa4c413,      0 bytes,     0 packets
    out ccaa54eb,      0 bytes,     0 packets
    local  10.1.1.0/24
    remote 192.168.1.0/24

Adding NAT Masquerade exclusion rules

Why are there 0 packets going out when I’m trying to ping across the tunnel? The packets need to be excluded from the NAT masquerade.

For R1:

configure
set service nat rule 5001 
set service nat rule 5001 description ExcludeIPSecTunnel
set service nat rule 5001 inbound-interface eth0
set service nat rule 5001 log disable
set service nat rule 5001 protocol all
set service nat rule 5001 type masquerade
set service nat rule 5001 destination address 10.1.1.0/24
set service nat rule 5001 exclude
commit
exit

For R2:

configure
set service nat rule 5001 
set service nat rule 5001 description ExcludeIPSecTunnel
set service nat rule 5001 inbound-interface eth1
set service nat rule 5001 log disable
set service nat rule 5001 protocol all
set service nat rule 5001 type masquerade
set service nat rule 5001 destination address 192.168.1.0/24
set service nat rule 5001 exclude
commit
exit

You may also have to recreate the default NAT masquerade with a lower priority (higher number) than this NAT exclusion rule. NAT masquerade rules need to be between 5000-10000. In this example, my default NAT masquerade on my WAN interface eth0 on R1 is rule 5000. I’m going to delete rule 5000 and recreate it as rule 5002, which is lower priority than the rule I created in the previous step.

For R1:

configure
delete service nat rule 5000
set service nat rule 5002
set service nat rule 5002 outbound-interface eth0
set service nat rule 5002 type masquerade
commit
exit

For R2:

configure
delete service nat rule 5000
set service nat rule 5002
set service nat rule 5002 outbound-interface eth1
set service nat rule 5002 type masquerade
commit
exit

Let’s try pinging the local IP of R1 from R2 again.

ping 192.168.1.1

PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
--- 192.168.1.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4006ms

Pings still aren’t going through, so I interrupted the pings. Let’s check the VPN tunnel stats on R2 with show vpn ipsec sa.

show vpn ipsec sa

peer-1.1.1.1-tunnel-1: #2, ESTABLISHED, IKEv1, bb126d66e9ca4bdf:f2fccad0bcf04771
  local  '2.2.2.2' @ 2.2.2.2
  remote '1.1.1.1' @ 1.1.1.1
  AES_CBC-128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
  established 2280s ago, reauth in 25471s
  peer-1.1.1.1-tunnel-1: #1, INSTALLED, TUNNEL, ESP:AES_CBC-128/HMAC_SHA1_96/MODP_1024
    installed 2280 ago, rekeying in 264s, expires in 1320s
    in  caa4c413,      0 bytes,     0 packets
    out ccaa54eb,    420 bytes,     5 packets,     4s ago
    local  10.1.1.0/24
    remote 192.168.1.0/24

Allowing traffic from remote subnet through firewall

Progress! Pings are now traveling out the IPSec tunnel, but nothing is coming back. Now we’ll add rules to the WAN_IN and WAN_LOCAL firewalls on both routers to allow traffic in from the remote subnets. The WAN_LOCAL rule will allow us to communicate with the router on the other side of the tunnel, and the WAN_IN rule will allow us to communicate with the rest of the devices in that remote subnet. In the commands below, I’ve given the rules number 18. Make sure that 18 is higher priority (lower number) than the default drop invalid state rule.

For R1:

configure
set firewall name WAN_LOCAL rule 18 
set firewall name WAN_LOCAL rule 18 action accept
set firewall name WAN_LOCAL rule 18 description AllowVPNsubnet
set firewall name WAN_LOCAL rule 18 log disable
set firewall name WAN_LOCAL rule 18 protocol all
set firewall name WAN_LOCAL rule 18 source address 10.1.1.0/24

set firewall name WAN_IN rule 18 
set firewall name WAN_IN rule 18 action accept
set firewall name WAN_IN rule 18 description AllowVPNsubnet
set firewall name WAN_IN rule 18 log disable
set firewall name WAN_IN rule 18 protocol all
set firewall name WAN_IN rule 18 source address 10.1.1.0/24
commit
exit

For R2:

configure
set firewall name WAN_LOCAL rule 18 
set firewall name WAN_LOCAL rule 18 action accept
set firewall name WAN_LOCAL rule 18 description AllowVPNsubnet
set firewall name WAN_LOCAL rule 18 log disable
set firewall name WAN_LOCAL rule 18 protocol all
set firewall name WAN_LOCAL rule 18 source address 192.168.1.0/24

set firewall name WAN_IN rule 18 
set firewall name WAN_IN rule 18 action accept
set firewall name WAN_IN rule 18 description AllowVPNsubnet
set firewall name WAN_IN rule 18 log disable
set firewall name WAN_IN rule 18 protocol all
set firewall name WAN_IN rule 18 source address 192.168.1.0/24
commit
exit

Now I’m able to ping the router and a remote device across the IPSec tunnel. Here’s a test ping from R2 to a device in R1’s private subnet.

ping 192.168.1.2

PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_req=1 ttl=127 time=16.5 ms
64 bytes from 192.168.1.2: icmp_req=2 ttl=127 time=19.9 ms
64 bytes from 192.168.1.2: icmp_req=3 ttl=127 time=18.9 ms
64 bytes from 192.168.1.2: icmp_req=4 ttl=127 time=17.1 ms
64 bytes from 192.168.1.2: icmp_req=5 ttl=127 time=20.5 ms

Conclusion

Finally, once everything is tested to be working properly, you’ll want to save your configs. I’ve intentionally not saved my configs, just in case I mess something up and need to revert back. When you save, your configurations will be retained after reboots. To save, enter:

configure
save
exit

For fun, I set up a continuous ping on the Windows machine (192.168.1.2) and pinged a network printer (10.1.1.2) in the remote subnet over the span of a few days.

Ping statistics for 10.1.1.2: 
Packets: Sent = 311538, Received = 311242, Lost = 296 (0% loss), 
Approximate round trip times in milli-seconds: Minimum = 9ms, Maximum = 1165ms, Average = 24ms

There were very few dropped packets, so I consider this tunnel to be pretty stable. Both sites have pretty stable internet connections.

You will need to modify the commands above accordingly to match your unique environment. Also, many/all of the above commands can be done in the GUI, too. The following is provided only as a reference. As always, test in a lab environment first!