Server 2012 Network Virtualization with SCVMM 2012 SP1

imageOne of the building blocks of multi-tenant cloud is Network Virtualization. Dynamic VLAN configuration has always been cumbersome, complex, and hard to manage. Now with Network Virtualization we can run multiple virtual networks on a physical network.

In this post, we will look at how pieces come together with the help of SCVMM 2012 SP1 and will go over creating Logical Network and VM Network setup with NVGRE.

First, there are two methods for Network Virtualization:

IP rewrite: customer IP address gets rewritten on each packet before leaving the fabric.

IP Encapsulation: It encapsulates all VM packets with a header over Provider Address.  (Generic Routing Encapsulation (NVGRE))

With SCVMM 2012 SP1, Default method is to encapsulate packets using NVGRE

With these methods, now we can run VMs independent of their IP address configuration, and can host customers in the cloud without worrying about the IP address space of the customer…

What we will try to achieve is that with NVGRE we will create two virtual networks, Blue and Red. They will both use the same IP address space but will be isolated from each other. These can be different divisions/departments in your scenario of private cloud, or different companies like in public clouds.

image

Before we start, just to briefly summarize:

  • Logical Networks define physical network
  • VM Networks define VM connectivity
  • Logical Switches define port capability

Logical Network:

Logical Networks is the abstraction of physical network and defines the type of a network, like External, Internal, Prod, Backup, DMZ etc. We will also need an IP pool here which will serve us as a Provider Address (PA) by VMM. PA is the address used in the packets on the wire. VMs are not aware of the PA in any way. Logical NW can have multiple logical network definitions which can have multiple subnets and multiple IP pools (PA). This is important when you have multiple sites so SCVMM uses definitions to connect VM to correct physical network.

To create a Logical Network in SCVMM: In the fabric, click on Logical Networks. Create a logical NW and then create an IP address pool for the logical network.

image

Network site is basically the definition of logical networks.

image

The same can be done with PS:

 $logicalNetwork = New-SCLogicalNetwork -Name "EXTERNAL-PA-Site1" -LogicalNetworkDefinitionIsolation $false -EnableNetworkVirtualization $true -UseGRE $true
$allHostGroups = @()
$allHostGroups += Get-SCVMHostGroup
$allSubnetVlan = @()
$allSubnetVlan += New-SCSubnetVLan -Subnet "192.168.1.0/24" -VLanID 0
New-SCLogicalNetworkDefinition -Name "PA_SITE1" -LogicalNetwork $logicalNetwork -VMHostGroup $allHostGroups -SubnetVLan $allSubnetVlan

Next, we need to create an IP pool with gateway, DNS/WINS addresses: Right click the logical network created and then hit create IP Pool

image

image

image

image

$logicalNetwork = Get-SCLogicalNetwork
$logicalNetworkDefinition = Get-SCLogicalNetworkDefinition -LogicalNetwork $logicalNetwork -Name "PA_SITE1"
# Gateways
$allGateways = @()
$allGateways += New-SCDefaultGateway -IPAddress "192.168.1.1" -Automatic
# DNS servers
$allDnsServer = @("192.168.1.100", "192.168.1.1")
# DNS suffixes
$allDnsSuffixes = @()
# WINS servers
$allWinsServers = @()
New-SCStaticIPAddressPool -Name "IPPOOL_PA_SITE1" -LogicalNetworkDefinition $logicalNetworkDefinition -Subnet "192.168.1.0/24" -IPAddressRangeStart "192.168.1.200" -IPAddressRangeEnd "192.168.1.250" -DefaultGateway $allGateways -DNSServer $allDnsServer -DNSSuffix "" -DNSSearchSuffix $allDnsSuffixes -RunAsynchronously

VM Network

VM networks defines the routing domain which can contain multiple subnets which can have multiple IP pools (Customer Address). Customer address gets assigned by IP pools associated with VM Network. CA only visible to virtual Machines. VM Network have 1-1 relationship with Logical Network. To create:

image

image

VM Subnet definitions:

image

We wont have any gateway device for now : )

image

$logicalNetwork = Get-SCLogicalNetwork -Name "EXTERNAL-PA-Site1"

$vmNetwork = New-SCVMNetwork -Name "RedVMs" -LogicalNetwork $logicalNetwork -IsolationType "WindowsNetworkVirtualization" -CAIPAddressPoolType "IPV4" -PAIPAddressPoolType "IPV4"

$subnet = New-SCSubnetVLan -Subnet "10.0.1.0/24"
New-SCVMSubnet -Name "Red_Subnet_1" -VMNetwork $vmNetwork -SubnetVLan $subnet

$subnet = New-SCSubnetVLan -Subnet "10.0.2.0/24"
New-SCVMSubnet -Name "Red_Subnet_2" -VMNetwork $vmNetwork -SubnetVLan $subnet
#ıppool
$vmNetwork = Get-SCVMNetwork -Name "RedVMs"
$vmSubnet = Get-SCVMSubnet -Name "Red_Subnet_2"
$allGateways = @()
$allGateways += New-SCDefaultGateway -IPAddress "10.0.2.1" -Automatic
# DNS servers
$allDnsServer = @("192.168.1.100")
# DNS suffixes
$allDnsSuffixes = @()
# WINS servers
$allWinsServers = @()
New-SCStaticIPAddressPool -Name "Red_IPPool_2" -VMSubnet $vmSubnet -Subnet "10.0.2.0/24" -IPAddressRangeStart "10.0.2.2" -IPAddressRangeEnd "10.0.2.254" -DefaultGateway $allGateways -DNSServer $allDnsServer -DNSSuffix "" -DNSSearchSuffix $allDnsSuffixes -RunAsynchronously

After the basic configuration done, you need to assign logical networks to the Hyper-V hosts.

Right Click on Hyper-V host, select Properties, go to hardware, and under networking adapters bind the logical networks to the NICs:

image

This completes the configuration.

TEST

4 VMs as shown, red1, red2, blue1, blue2

image

One a new VM provisioned:

image

if we look at the ipconfig, we will see that it gets IP address from DHCP based on the definitions (CA). You can run series of ping test to verify the configurations by placing the VMs on the same or different hosts.

image

This is the BlueVM2 running on another host:

image

If we capture traffic on VMs, they don’t see anything other than CA addresses.

image

When I tried to move one of the VMs to another host (it is not cluster), the wizard had an error: “Windows network virtualization is not enabled on a host NIC available for this placement” as shown below:

image

In this case you need to enable Windows Network Virtualization Filter drivers on the NICs:

image

To locate which IP addresses has been assigned:

Get-SCIPAddress -StaticIPAddressPool $ippool | ft -property Address,AssignedToType,State

Address AssignedToType State
——- ————– —–
10.0.1.2 VirtualNetworkAdapter Assigned
10.0.1.3 VirtualNetworkAdapter Assigned

To check Lookup Records, Virtual Subnet IDs, Customer IDs:

Get-NetVirtualizationLookupRecord

CustomerAddress : 10.0.1.1
VirtualSubnetID : 8377539
MACAddress : 005056000000
ProviderAddress : 1.1.1.1
CustomerID : {49A46A4E-01FC-4E1A-AD90-4D19A4C4A3B4}
Context : SCVMM-MANAGED
Rule : TranslationMethodEncap
VMName : GW
UseVmMACAddress : False

CustomerAddress : 10.0.1.2
VirtualSubnetID : 8377539
MACAddress : 00155d01b40a
ProviderAddress : 192.168.1.200
CustomerID : {49A46A4E-01FC-4E1A-AD90-4D19A4C4A3B4}
Context : SCVMM-MANAGED
Rule : TranslationMethodEncap
VMName : Blue1-win2k3-1
UseVmMACAddress : False

CustomerAddress : 10.0.1.3
VirtualSubnetID : 8377539
MACAddress : 00155d01b40b
ProviderAddress : 192.168.1.201
CustomerID : {49A46A4E-01FC-4E1A-AD90-4D19A4C4A3B4}
Context : SCVMM-MANAGED
Rule : TranslationMethodEncap
VMName : Blue2-win2k3-2
UseVmMACAddress : False

CustomerAddress : 10.0.1.2
VirtualSubnetID : 15139587
MACAddress : 00155d01b40c
ProviderAddress : 192.168.1.202
CustomerID : {1C95F727-3160-4912-B6B1-DF55E6D85E85}
Context : SCVMM-MANAGED
Rule : TranslationMethodEncap
VMName : Red1-win2k3-3
UseVmMACAddress : False

CustomerAddress : 10.0.1.1
VirtualSubnetID : 15139587
MACAddress : 005056000001
ProviderAddress : 1.1.1.1
CustomerID : {1C95F727-3160-4912-B6B1-DF55E6D85E85}
Context : SCVMM-MANAGED
Rule : TranslationMethodEncap
VMName : GW
UseVmMACAddress : False

It is also possible to create an netsh trace to see the packets. I will write about this in another post. It is here:

Hope helps Smile

Bulent Tolu

Bulent Tolu

Bulent is an IT professional with Master's in MIS and 10-years of experience in broad range of Information Technologies. He is exposed to engineering/architecting, implementation/integration, and administration of various high-available IT systems and infrastructure. He has a passion to continually research, test and evaluate new technologies and follow industry best practices to secure and optimize IT systems. Currently, he lives in Istanbul and works as a Sr. Cloud Computing and Virtualization Consultant. He has a diverse knowledge and interest in Virtualization and Cloud Computing.

:: VMware Certified Advanced Professional 5
Data Center Administration (VCAP5-DCA)
Data Center Design (VCAP-DCD)
:: VTSP (VMware Certified Technical Sales Professional 5)
:: MCSE(Microsoft Certified Solutions Expert) – Private Cloud 2012
:: CCEE(Citrix Certified Enterprise Engineer)
Bulent ToluBulent ToluBulent ToluBulent Tolu Bulent Tolu
Bulent Tolu

16 Comments

  1. Pat says:

    Hi,

    I am hoping you can help with a issue I have.

    I have setup SCVMM 2012 with SP1 network virtualization with 2 Hosts and 4 VMs with 2 different VM Subnets.

    I have setup the network virtualization and if I have all 4 VMs on the one Host I can ping between the VMs on the same subnet.

    The problem I have is if the VM or VMs are on the other Host I cannot ping the same VMs on the same subnet.

    I have checked the LookupRecord and all VMs have the correct PA and CA addressing and VirtualSubnetID

    If you have any idea what the issue maybe I would be very grateful

    Thanks
    Pat

    • Bulent Tolu Bulent T says:

      Hi Pat, I am assuming hosts are bound to same logical network. Can you check whether Windows Network Virtualization Filter drivers on the NICs on both hosts are enabled? In most cases, VMM takes care of it but in my case I had to enable it manually. Since lookup records are created on both hosts I think your setup should be okay. Have you tried migrating VMs and checking the records again? Hope helps :)

  2. Pat says:

    Hi Bulent,

    When I created the virtual switches on the hosts I checked that the NV Filter got enabled through scvmm

    The hosts are not bound to the logical network. I have 3 NICs in each host with two for Network Virtualization, does the Host need to be on the same Logical Network?

    On the same Host the VMs on the same subnet can ping each other through different NICs.

    When I migrate they stop pinging but the lookup records are exactly the same on both hosts

    I did a packet capture and noticed on the same Host there doesn’t appear to be GRE routing when I ping from VM to VM on the same subnet. But when I migrated a VM to another Host it appears in the packet capture but the ping response stops, only shows requests.

    Hope this helps a bit better with the setup.

    Thanks

    • Bulent Tolu Bulent T says:

      Hi Pat, I am a little confused now :) :) Network virtualization gets enabled by logical networks, and they need to be bound to NICs on the hosts (hardware tab of the host properties – for VMs only). This actually tells hosts to use GRE and provides PA address range that the hosts can use. Then, when you create a “VM Network” with the logical network created (ie, RedVMs/CompanyA, BlueVMs/CompanyB etc.), this creates “routing domains” per VM Networks and Virtual SubnetIDs for subnets defined, so they are isolated. Can you create a logical network and bind to the both hosts? Thanks.

  3. Pat says:

    P.S All Firewalls have been turned off.

  4. Pat says:

    Sorry about the confustion :(

    All the NICs have the logical network assigned that I created when I setup SCVMM.
    They are all getting a PA address from the IP Pool setup.
    The VMs are all getting the correct CA from the two VM Networks I setup (Blue and Red)
    If I run a Lookup Record I can see the NICs have been assigned a PA address and a CA address.
    The Routing Domain and Virtual SubnetID are the same for the Blue Subnet VMs. The Red one is also correct.
    On the same host the VM Network Blue can ping each other same with Red VM Network.

    Its when they are moved to another Host, the routing domain and virtual subnetid are all correct on the other host.

    Running a Packet Capture I can see the packet hitting the Host but reports destination unreachable (Host Administratively Prohibited)

    I can ping all the PA IP addresses from each host without error.
    I have included the Lookup Record from each Host.

    Host 2 (VM Blue-02)

    CustomerAddress : 192.168.12.12
    VirtualSubnetID : 10535999
    MACAddress : 001dd8b71c03
    ProviderAddress : 192.168.50.246
    CustomerID : {112D9D66-7D4C-4622-8B32-EEE39ED743D1}
    Context : SCVMM-MANAGED
    Rule : TranslationMethodEncap
    VMName : Blue-02
    UseVmMACAddress : False

    CustomerAddress : 192.168.12.11
    VirtualSubnetID : 10535999
    MACAddress : 001dd8b71c01
    ProviderAddress : 192.168.50.241
    CustomerID : {112D9D66-7D4C-4622-8B32-EEE39ED743D1}
    Context : SCVMM-MANAGED
    Rule : TranslationMethodEncap
    VMName : Blue-01
    UseVmMACAddress : False

    CustomerAddress : 192.168.12.1
    VirtualSubnetID : 10535999
    MACAddress : 005056000000
    ProviderAddress : 1.1.1.1
    CustomerID : {112D9D66-7D4C-4622-8B32-EEE39ED743D1}
    Context : SCVMM-MANAGED
    Rule : TranslationMethodEncap
    VMName : GW
    UseVmMACAddress : False

    Host 1 (VMs Blue-01, Red-01, Red-02)

    CustomerAddress : 192.168.12.11
    VirtualSubnetID : 10535999
    MACAddress : 001dd8b71c01
    ProviderAddress : 192.168.50.241
    CustomerID : {112D9D66-7D4C-4622-8B32-EEE39ED743D1}
    Context : SCVMM-MANAGED
    Rule : TranslationMethodEncap
    VMName : Blue-01
    UseVmMACAddress : False

    CustomerAddress : 192.168.14.1
    VirtualSubnetID : 7285158
    MACAddress : 005056000000
    ProviderAddress : 1.1.1.1
    CustomerID : {8E0F472E-E934-4BD9-93FE-B1D8C6838BA1}
    Context : SCVMM-MANAGED
    Rule : TranslationMethodEncap
    VMName : GW
    UseVmMACAddress : False

    CustomerAddress : 192.168.12.12
    VirtualSubnetID : 10535999
    MACAddress : 001dd8b71c03
    ProviderAddress : 192.168.50.246
    CustomerID : {112D9D66-7D4C-4622-8B32-EEE39ED743D1}
    Context : SCVMM-MANAGED
    Rule : TranslationMethodEncap
    VMName : Blue-02
    UseVmMACAddress : False

    CustomerAddress : 192.168.14.12
    VirtualSubnetID : 7285158
    MACAddress : 001dd8b71c02
    ProviderAddress : 192.168.50.244
    CustomerID : {8E0F472E-E934-4BD9-93FE-B1D8C6838BA1}
    Context : SCVMM-MANAGED
    Rule : TranslationMethodEncap
    VMName : Red-02
    UseVmMACAddress : False

    CustomerAddress : 192.168.12.1
    VirtualSubnetID : 10535999
    MACAddress : 005056000001
    ProviderAddress : 1.1.1.1
    CustomerID : {112D9D66-7D4C-4622-8B32-EEE39ED743D1}
    Context : SCVMM-MANAGED
    Rule : TranslationMethodEncap
    VMName : GW
    UseVmMACAddress : False

    CustomerAddress : 192.168.14.11
    VirtualSubnetID : 7285158
    MACAddress : 001dd8b71c00
    ProviderAddress : 192.168.50.240
    CustomerID : {8E0F472E-E934-4BD9-93FE-B1D8C6838BA1}
    Context : SCVMM-MANAGED
    Rule : TranslationMethodEncap
    VMName : Red-01
    UseVmMACAddress : False

    sorry about the long comment

    • Bulent Tolu Bulent T says:

      Hi Pat,

      I have been trying to repro it but could not. All output above looks okay to me. Mine is the same… Are you using logical switch? If so, can you check if uplink profile has NV enabled… I know you confirmed but I would double-check if the switch created on the hosts have NV filter enabled. I will email some screenshots from my demo … :D Thought It may give you a hint hopefully… Hope helps…

  5. Pat says:

    Hi Bulent,

    I have checked and checked again the settings for the uplink profile NV and the Hosts NV settings, all are enabled.

    All logical switches are correct and working.

    Thanks for the screen shots, I have followed the instructions and still the same result.

    I am running the SCVMM 2012 as a VM, not sure if that makes a difference.

    Just seems strange it works on the same host with different NICs (Logical Switches) but not on different Hosts.

    Even if I check the View Network in SCVMM 2012 it shows the correct VMs connected to the correct VM Networks.

    Thanks
    Pat

  6. Alex says:

    Thank you very much for the post! I have the following question:

    1. Did you use VM templates in order to deploy the VMs or you configure already deployed VMs in the “One a new VM provisioned:” step?

    Thank you very much!

  7. Alex says:

    Hi Pat!
    I have to Hyper-V hosts with 1 physical NIC each. I want test the network virtualization scenario which you kindly shared in your blog.
    1. How should I configure the external virtual switch and internal virtual switch on each of the Hyper-V hosts so that I would be able to implement this network design?
    2. Is 1 physical NIC per the Hyper-V host sufficient for that scenario?
    Many thanks in advance!

  8. Alex says:

    Hi Pat!
    I have 2 (two) Hyper-V hosts with 1 physical NIC each. I want test the network virtualization scenario which you kindly shared in your blog.
    1. How should I configure the external virtual switch and internal virtual switch on each of the Hyper-V hosts so that I would be able to implement this network design?
    2. Is 1 physical NIC per the Hyper-V host sufficient for that scenario?
    Many thanks in advance!

  9. CypherBit says:

    Great article. Could you perhaps expand on this topic with the newly released Preview of VMM R2 and the use of a Gateway?

  10. xdaniel says:

    Hi,

    Can you post a wireshark capture with the traffic between host when doing a ping?

  11. Bulent Tolu Bulent T says:

    @xdaniel Hello, unfortunately, dont have the same environment right now. When I build another cluster, I will try to post/answer this as well.

  12. krishna says:

    wonderfull article , thanks a lot , you simply explained it , Kudos

Leave a Reply