window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-16803030-1');

Security in a Virtualized World

For our August Lunch & Learn presentation “Security in a Virtualized World,”  TBL’s virtualization expert Harley Stagner was joined by Bryan Miller of Syrinx Technologies.    The speakers each brought their own unique perspectives to the subject matter, as Harley’s job is to help clients  to build virtual infrastructures, and Bryan’s job is to see if he can break into them (with his clients’ permission, of course).

Harley and Bryan discussed the challenges of managing a virtualized infrastructure while maintaining system security.  Focus areas included patching issues and best practices for auditing and hardening your system.  In addition, Harley and Bryan covered  vulnerabilities that are germane to virtualized environments, while also reviewing the newest compliance guidelines.

To conclude, Harley and Bryan provided a live demonstration of vMotion, and explained how it can be infiltrated.  Bryan demonstrated how during a typical vMotion session information could be easily exposed without taking security measures.

A big TBL “Thank You” to all those who attended for sessions in Virginia Beach and Richmond in August.  In September, TBL Networks is participating in two very exciting events.  On Wednesday, September 14, TBL is hosting an exclusive Lunch & Learn in Virginia Beach entitled “Cisco Collaboration Meets Virtualization.”  This presentation will be lead by TBL Networks’ CCIE and Collaboration Practice Lead Engineer Patrick Tredway.  For our Richmond readers, please join us at RichTech’s Annual TechLinks Golf Tournament on Monday, September 12th.   TBL Network is serving as a Reception Sponsor for this event, and we look forward to meeting everyone in the technology field in Central Virginia for a great day of golf and socializing.


For more information on Bryan Miller and Syrinx Technologies, please to

Two TBL Engineers Named vExperts 2011

TBL Networks is very proud to announce that two of our Data Center Engineers, Sean Crookston and Harley Stagner, have been named as vExperts 2011 by VMware. Sean and Harley received this designation as recognition of their contributions to the VMware, virtualization, and cloud computing communities.

According to VMware, “the vExperts are people who have gone above and beyond their day jobs in their contributions to the virtualization and VMware user community. vExperts are the bloggers, the book authors, the VMUG leaders, the tool builders and town criers, the tinkerers and speakers and thinkers who are moving us all forward as an IT industry.”

TBL Networks’ Solutions Engineer Sean Crookston also has the title of VMware Certified Advanced Professional in Data Center Administration (VCAP-DCA). Sean is only the 47thperson worldwide to achieve this elite virtualization certification.

TBL Networks’ Account Engineer Harley Stagner is the first VMware Certified Design Expert (VCDX) in Virginia, and just the 46th person worldwide to hold this title.  The VCDX is the highest certification available from VMware.

Congratulations again to Sean Crookston and Harley Stagner – vExperts 2011.

A Structured Virtual Infrastructure Approach Part III: Compute Platform Software

In Part II of the Structured Virtual Infrastructure Approach Series, we explored the Cisco Unified Computing System (UCS) hardware. This post will explore the UCS management software. Up to 20 chassis can be managed with a single instance of the UCS Manager. The UCS Manager is included with the 6100 series fabric interconnects. All of the blades in the infrastructure can be managed through this single interface. Below, we’ll discuss some of the features that make this interface unique among compute platform management interfaces.

Complete Compute Infrastructure Management

  • All the chassis / blades (up to 20 chassis worth) are managed in this single interface.
  • The management is not “per chassis” like legacy blade systems.
  • Consolidated management means efficient management for the entire compute platform.

Service Profiles

  • All of the items that make a single server (blade) unique are abstracted with a service profile.
  • This may include WWN, MAC, Bios Settings, Boot Order, Firmware Revisions, etc.
  • WWN’s and MAC’s are pulled from a pool that can be defined.
  • Even the KVM management IP’s are pulled from a pool so the administrator does not have to manage those IP’s at all.
  • You can create a Service Profile template with all of these characteristics and create Service Profiles from the template.
  • When you need to deploy a new blade all of the unique adjustments are already completed from the Service Profile template.
  • With the M81KR Virtual Interface Card (VIC) the number of interfaces assigned to a blade can be defined in the Service Profile template.
  • Even though the a single mezzanine card in a blade will only have (2) 10Gb ports, the M81KR VIC allows you to define up to 56 FC/Ethernet ports. This allows for more familiar vSphere Networking setups like the one below:


The diagram above is a setup that can be used with the Cisco Nexus 1000v. It would be impossible to do this setup on the UCS B-Series without the M81KR VIC. We’ll explore why a networking setup like this may be necessary when we get to the vSphere specific posts in this series.

Role Based Access Control

  • Even though the components are converging (storage, compute, networking) the different teams responsible for those components can still maintain access control for their particular area of responsibility.
  • The UCS manager has permissions that can be applied in such a way that each team only has access to the administrative tab(s) that they are responsible for.
  • Network team –> Network Tab, Server Team –> Server Tab, Storage Team –> Storage Tab.
  • The UCS manager also supports several different authentication methods, including local and LDAP based authentication.

What vSphere does for the Operating System instances, the UCS does for the blade hardware. It abstracts the unique configuration items into a higher software layer so that they can be easier managed from a single location.  The next post in this series will take a look at some storage platform hardware. It’s not just about carving out disks for the virtual infrastructure any longer. We’ll take a look at some options that integrate well with a modern vSphere infrastructure.

Cisco Expands UC Virtualization Support

Stand back….this is a pretty big announcement!  As of June 7, 2011 Cisco began support for some Collaboration (formerly Unified Communications) applications running in a virtual environment on hardware other than their own Unified Computing System (UCS). The is the first in hopefully many steps to come in widening support for benefits we often realize with typical desktop and server applications running on a VMware hypervisor. The details are as follows.


Cisco is pleased to announce expanded virtualization of Cisco Unified Communications starting Jun 7, 2011.

On Jun 7 Cisco will add two additional virtualized UC offers. Customers will then have three deployment options:

1. UC on UCS – Tested Reference Configurations

2. UC on UCS – Specs-based VMware hardware support

3. HP and IBM – Specs-based VMware hardware support

Phase 1 support begins Jun 7, 2011 and should include the following (see for final products and versions supported):

– Cisco Unified Communications Manager 8.0.2+ and 8.5.1

– Cisco Unified Communications Manager – Session Management Edition 8.5.1

– Cisco Unified Communications Management Suite

– Cisco Unity Connection 8.0.2+ and 8.5.1

– Cisco Unity 7.0.2+ (with Fiber Channel SAN only)

– Cisco Unified Contact Center Express and IP IVR 8.5.1

Support for additional products and versions will phase in over rest of CY11.

Specs-based VMware hardware support adds the following

– UC Compute support for UCS, HP, IBM servers on VMware’s hardware compatibility list and running Intel Xeon 5600 / 7500 family CPUs

– UC Network support for 1Gb through 10Gb NIC, CNA, HBA and Cisco VIC adapters that are supported by above servers

– UC Storage support for DAS, SAN (Fiber Channel, iSCSI, FCoE) and NAS (NFS).

– More co-resident UC VMs per physical server if more powerful CPUs are used

– Note that UC / non-UC / 3rd-party co-residency is still not supported.

– Note that hardware oversubscription is still not supported by UC.

– No changes to VMware product, version or feature support by UC


This most certainly gives us far more agility for the manner in which we deploy these applications. More info to come as I get it…

A Structured Virtual Infrastructure Part I: Physical Infrastructure

Server virtualization is infectious. It is a technology that tends to take off in record pace in IT organizations that have adopted it as part of their infrastructure. It has been my experience that organizations fall into one of two broad categories when it comes to their virtualization initiatives. They either look at server virtualization as a “Strategic Initiative” or they use server virtualization as a “Tactical Tool.” Let’s explore these categories and then I’ll discuss some infrastructure options for a structured virtual infrastructure.

Server Virtualization as a “Tactical Tool”

I have seen this in many organizations. The IT group needed to test a new application or needed to spin up a new server quickly. What’s the quickest way to spin up a new server? Server virtualization, of course. So, here is how I see many infrastructures get started:

  • IT department downloads the free vSphere Hypervisor
  • IT department proceeds to click next until the hypervisor is installed
  • IT department spins up a few virtual machines on the hypervisor
  • “Life is good. That was easy wasn’t it?”
  • “It’s so easy and cool that more demand creeps up for further virtual machines
  • Pretty soon the IT department wants to host production workloads on the hypervisor
  • “But wait? What about failover, live migration, etc. Don’t I need a SAN for that?”
  • How “much” storage do I need?
  • IT department calculates how much space they are using on their servers, or worse yet, how much disk space in total is available on all of their servers combined
  • “Wow! I need a lot of space to host all of those servers”
  • IT department buys large slow “shared disks” of some variety to satisfy the SAN requirement
  • IT department sets up vCenter on a spare server
  • IT department clicks next until a few hypervisors are installed and added to the new cluster complete with “shared storage”
  • Now there is some equipment and software in place to host virtual machines
  • IT department spins up new virtual machines until they are suddenly out of capacity or things are “just slow and error prone”
  • Virtualization stalls because there is no more capacity and there is a lack of trust in the virtual infrastructure as it stands
  • IT department starts purchasing physical servers again for “critical” applications
  • “Now DR must be provided for those “critical” applications. How can we protect them?”
  • “The easiest thing to do would be to leverage virtualization, but we’re out of capacity and the platform has been problematic”
  • “What do we need to do to leverage virtualization on a larger scale in our infrastructure?”

It’s a vicious cycle and it is why I continue to see companies only 20-40% virtualized. It is great that server virtualization technology has been embraced. However, without proper planning and a structured approach to building and maintaining the virtual infrastructure, many organizations will continue to be only 20-40% virtualized. They are leaving the many benefits of server virtualization and even money on the table if they stall.

So, this series of posts will explore the alternative of server virtualization as a “Strategic Initiative”. This is the approach that I take with my clients at TBL to either build a structured virtual infrastructure from the ground up or remediate a “tactical tool” virtual infrastructure to the point that it becomes an effective platform to host the organizations infrastructure moving forward.

Physical Infrastructure Choices

There are many options when it comes to virtual infrastructure hardware. Before any hardware choices are made, a capacity planning engagement should occur. Notice that capacity planning was not mentioned at all in the “Server Virtualization as a Tactical Tool” scenario. Look at this infrastructure as if it is going to host all of your physical servers even if you will not start there. How else does one determine if the infrastructure purchased for a new virtual infrastructure is sufficient if capacity planning is not performed? I can’t count the number of times that I have heard the equivalent of the below phrase:

  • “These host servers and storage should do since my physical servers don’t really do much.”

How do you know that your host servers don’t do much unless you have performed capacity planning? Is it a gut feeling? I have seen many gut feelings cause server virtualization stall. We need to examine the four “core” resources (CPU, RAM, DISK, and NETWORK) to determine not only our capacity but the level of performance needed. After a proper capacity planning engagement we can determine the “feeds and speeds” of our hardware. However, the hardware choice becomes about more than just raw specs in a structured virtual infrastructure. Let’s examine some options.

Traditional Rackmount Server Infrastructure

This is the standard virtual infrastructure that has been around for a while. With this approach, you take rackmount servers as hosts and provide shared storage via iSCSI, NFS, or Fibre Channel. A diagram of this approach can be seen below.


This infrastructure is well understood. However, the scalability is somewhat limited. Typically, a virtual infrastructure host will have eight to ten cables attached to it in a 1Gbe environment. This is due to the way that traffic should be separated in a virtual infrastructure. This is fine for a few hosts. As the infrastructure is scaled, the number of cables and ports required becomes problematic. I have seen environments where shortcuts were taken to provide enough ports by combining virtual infrastructure traffic types even though they should be separated. As more hosts are needed a better solution to scaling the infrastructure needs to be in place.

Converged Rackmount Server Infrastructure

This infrastructure consolidates the traditional 1GbE infrastructure into a 10GbE infrastructure by connecting to an FCoE or straight 10GbE switch. This allows more bandwidth and cuts down on the port count required as the infrastructure scales.


As this infrastructure is scaled, the number of cables and ports required is much more manageable. It must be noted that the cable infrastructure still scales linearly with the hosts. Port count can still be an issue in larger environments. Also, we really haven’t added anything new on the server management front in this design choice. Again, for smaller, relatively static environments this can work nicely. If the infrastructure needs to be able to scale quickly and efficiently, there are better options.


Converged Blade Infrastructure

Large scale ease of management, efficient scaling, and massive compute capacity can be achieved without the inherent cable / port count problems with a converged blade infrastructure. In the example below, a Cisco UCS B-Series converged blade infrastructure to achieve these benefits.


Let’s look at the components of this infrastructure model.

  • The UCS 6100 series would be similar to the FCoE switches in the Converged Rackmount Infrastructure. I say similar because it is ideal to still have upstream SAN and LAN switches. In this scenario the 6100 pair act like a host (or multiple hosts) attached to the Fibre Channel Fabric. They accomplish this with an upstream switch that is NPIV capable.
  • The blade chassis provide the backplane connectivity for your compute resources or blades. Each chassis can have up to (8) 10Gb FCoE ports for connectivity. The blades share that connectivity to the upstream 6100’s.
  • The 6100’s then take that FCoE traffic and split it into Fibre Channel to connect to the upstream SAN Fabric and Ethernet to connect to the upstream LAN Fabric.
  • Instead of calculating bandwidth / port counts at the server level as you would in a traditional rack mount scenario, you calculate bandwidth needs at the 6100 level.
  • Less cabling, more scalability, easier management, smaller footprint.

With the up front investment in the 6100’s in this architecture, the solution scales out with only incremental cost very nicely. Also, the 6100’s are the single point of management using the UCS Manager in this infrastructure. The UCS abstracts unique information that would identify a server into a service profile. The types of data in the service profile may include items such as:

  • Bios Settings
  • WWN
  • MAC Addresses
  • Boot Order
  • Firmware Revisions

This way, settings that would normally be configured after a server arrives can be pre-configured. When a new blade arrives you can simply slide the blade into the chassis, assign the service profile, boot it and it is ready to install an OS in minutes. If this OS is ESXi, then that only takes about 5 minutes to install as well.

With the Converged Blade Infrastructure we set up a foundation for ease of incremental scalability when the environment grows. Using this as the model infrastructure, the upcoming posts will examine the different components involved in more detail so that you can get a holistic view of the entire virtual infrastructure as a structure approach is taken to building this out.

Sean Crookston Awarded VCAP-DCA

TBL Networks’ Solutions Engineer Sean Crookston recently attained  the title of VMware Certified Advanced Professional in Datacenter Administration (VCAP-DCA). Sean is only the 47th person worldwide to achieve this elite virtualization certification.

The VMware Certified Advanced Professional 4 – Datacenter Administration (VCAP4-DCA) is designed for Administrators, Consultants and Technical Support Engineers capable of working with large and more complex virtualized environments and can demonstrate technical leadership with VMware vSphere technologies.

Sean put together many hours of study and research to reach this achievement.  Sean has documented much of this work on his website –

Congratulations to Sean Crookston, VCAP-DCA #47.

Follow Sean Crookston on Twitter at

Follow TBL Networks on Twitter at

What I Learned – EMC VNX and The Green Hornet

Virtualization. Storage. Seth Rogen? You might not associate movie theaters and superheroes with Unified Storage, but for one day, TBL Networks found a way to bring them together. On February 17th, TBL Networks’ Harley Stagner and EMC’s Steve Woods provided an exclusive presentation the new EMC VNX series of storage solutions at Movieland at Boulevard Square. Following the presentation, the attendees watched a private screening of The Green Hornet.

Here are a few things that I learned from the EMC VNX/Green Hornet event.

– The VNX series is the new mid-tier storage platform built for the demanding virtualized data center. It has a fully redundant, multi-controller design that scales and scales.

– Just because you have giant bucket of popcorn, you are not obligated to eat the entire bucket.

– Even if you ask nicely, Steve and Harley will not add select scenes from Black Swan to their presentation.

– The VNXe series has a series of best practice wizards so you can configure your storage with just a few clicks.

– I am a sucker for any technology that comes with a wizard.

– Movieland is not planning to screen The Cannonball Run as part of their MOVIES & MIMOSAS® series.

– The VNX series provides the broadest range of unified storage platforms in the industry.

Thanks to everyone who came out to hear Harley and Steve and to watch the movie. Special thanks to the team at Movieland for a providing a great facility and experience.

First Ask Harley Session Question and Answer Summary

On Thursday, January 13th I had my first Ask Harley Session. This was a session where I answered virtualization and VMware related questions on Twitter. I received a lot of great questions during this session. Thank you to all who participated. Below are the questions and their answers in case you missed them on Twitter.


Ask Harley: Question 1 – What common issues or mistakes do you see with your customers who have setup VMware infrastructure or are looking to setup VMware?


NOTE: This answer was originally provided over a series of Tweets by Harley Stagner on 1/13/11 at TBL Networks’ Twitter site as part of our “Ask Harley” series.



What common issues or mistakes do you see with your customers who have setup VMware infrastructure or are looking to setup VMware?


Most of the issues in an initial deployment occur from a lack of capacity, application, and infrastructure planning.

Consider the 4 core (CPU, RAM, DISK, NET) resources from a capacity standpoint. Consider application requirements (MS Clustering, Dongles, Vendor Support, Etc.).

Consider scalability and ease of management from the infrastructure standpoint. Infrastructure item examples: Scale up vs scale out(more hosts = more DRS opportunities,Less hosts = more risk).

Details. Details. Details. Example- Do I have enough space for VMDK and Swap files? Do I have a syslog server for ESXi?

Keep it simple. Avoid Resource Pools, Reservations, and Limits unless they are needed.

Resource pools are NOT for organization. That’s worth repeating. Resource pools are NOT for organization. Folders are.

There is more involved in a virtualization design / deployment than clicking next.


Ask Harley: Question 2 – Why would you use Virtual Port-ID NLB instead of IP-Hash NLB?


NOTE: This answer was originally provided over a series of Tweets by Harley Stagner on 1/13/11 at TBL Networks’ Twitter site as part of our “Ask Harley” series.



Why would you use Virtual Port-ID NLB instead of IP-Hash NLB?


The summary answer would be simplicity. Port-ID is the default load balancing and good in a wide range of use cases.

Port-ID Advantage: Simple, effective. Port-ID Disadvantage: Only egress traffic is load balanced as it depends on the source virtual port id

IP-Hash has an upstream dependency on 802.3ad static link aggregation. An example is etherchannel on Cisco Switches. Even if the dependency is met. You may not be load balancing as efficiently as you think. You need MANY destinations in order for IP-Hash maximum effectiveness.

Why? Because IP-Hash algorithm uses an Xor of source and destination IP using the least significant byte (LSB) of both addresses. Then, the modulo of the Xor result is computed over the number of physical NICs.

Formula- (“LSB of Source IP of VM” xor “LSB of Destination IP”) mod “Number of Physical NICs in the team” = Modulo. If the Modulo is the same among two VM’s they will choose the same physical NIC for traffic. If the Modulo is different, they will choose different physical NICs.

IP-Hash Advantage: Ingress and Egress load balancing. IP-Hash Disadvantage: Upstream dependencies. More complexity and planning involved.

For further detail beyond Twitter see Ken Cline’s excellent post on Network Load Balancing here ->


Ask Harley: Question 3 – What are some of the most difficult parts of the journey to becoming a VCDX?


NOTE: This answer was originally provided over a series of Tweets by Harley Stagner on 1/13/11 at TBL Networks’ Twitter site as part of our “Ask Harley” series.


What are some of the most difficult parts of the journey to becoming a VCDX?


I can only speak from my experience on the VCDX journey.

If you are well prepared and study the written portion of the tests (and to some extent the lab), while challenging are nothing compared to the application and defense.

The application and design submission itself requires a significant amount of work.

Whatever you calculate the work effort to be, you may want to double or quadruple it.

I spent about four weeks worth of man-hours on my application and design.

Make sure you meet the application requirements in your design documentation and then go beyond. Leave nothing blank.

Know your design for the defense. This is worth repeating. Know your design cold for the defense. Nothing can prepare you for the defense other than knowing your design and significant field experience.

You don’t know what the panelists will throw at you, so you must have a breadth of knowledge.

By far the most challenging of all may be getting a handle on your nerves during the panel defense.

My detailed experience is here ->

There is also a nice roundup of experiences here ->


Ask Harley: Question 4 – Are there significant performance advantages on ESXi using Dell Equalogic MPIO drivers over native VMware Round Robin drivers?


NOTE: This answer was originally provided over a series of Tweets by Harley Stagner on 1/13/11 at TBL Networks’ Twitter site as part of our “Ask Harley” series.


Are there significant performance advantages on ESXi using Dell Equalogic MPIO drivers over native VMware Round Robin drivers?


I have not tested the performance of the Dell Equalogic MPIO drivers and a quick search did not net any official benchmarks. In general, a properly implemented and tested third party MPIO solution like the Equalogic MEM or Powerpath VE should be better at making path selections.

The storage vendor should be more familiar with its array than VMware. I have experience with Powerpath VE and the installation is fairly easy. Once it is installed there is very little management besides just updating the software from time to time.

Any other third party plugin should have a similar ease of use / management story.Consult the vendor.

I did find one unofficial performance testing post here ->


Ask Harley: Question 5 – What about using multiple vendor MPIO drivers, have you ever experienced issues in a mixed environment?



NOTE: This answer was originally provided over a series of Tweets by Harley Stagner on 1/13/11 at TBL Networks’ Twitter site as part of our “Ask Harley” series.



What about using multiple vendor MPIO drivers, have you ever experienced issues in a mixed environment?




I have not tested using multiple MPIO drivers.

However, I would not recommend that scenario as a host can only have one path selection policy.

If you have multiple hosts using different path selection policies, then performance or availability can be impacted. You should always use the same path selection policy for all of the hosts in a cluster to avoid path confusion.

Consistency is the key in a virtual infrastructure


Ask Harley: Question 6: With Cisco load balancers in places, do I still specify direct URL’s to the two security servers for #VMwareView or use LB URL?

NOTE: This answer was originally provided over a series of Tweets by Harley Stagner on 1/13/11 at TBL Networks’ Twitter site as part of our “Ask Harley” series.


With Cisco load balancers in places, do I still specify direct URL’s to the two security servers for #VMwareView or use LB URL?


With Cisco load balancers in front of the view security servers, you would specify the load balancer URL.

Cisco Nexus 1000v and vSphere HA Slot Sizes

When implementing the Cisco Nexus 1000v, High Availability (HA) Slot Sizes can be affected on your vSphere Cluster. HA slot sizes are used to calculate failover capacity for HA when the “Host Failures” HA setting is used. By default the slot size for CPU is the highest reservation of CPU among all VM’s in the cluster (or 256 MHz if no per VM reservations exist). The slot size for Memory is the highest reservation of Memory among all VM’s in the cluster (or 0 MB + Memory Overhead if no per VM reservations exist). If you want to really dive further into HA and slot size calculations, I would highly recommend reading Duncan Epping’s HA Deepdive at


Each Cisco Nexus 1000v Virtual Supervisor Module (VSM) has a reservation of 1500 MHz on the CPU and 2048 MB on the Memory so that these resources can be guaranteed for the VSM’s. So, the slot size will be affected accordingly (1500 MHz for CPU and 2048 MB for Memory).


If you do not want your slot size affected and you do not want to enable the advanced slot size parameters das.slotCpuInMHz or das.slotMemInMB there is another solution that will allow you to still guarantee resources for the Nexus 1000v VSM’s and maintain smaller slot sizes.

Create a separate resource pool for each VSM with a CPU reservation of 1500 MHz and Memory reservation of 2048 MB.




Since slot sizes are only affected by per VM reservations, the resource pools will give you the desired results without increasing your HA slot sizes. I wouldn’t recommend doing this with too many VM’s as this turn into a management nightmare very quickly. However, for the two VSM’s in a Cisco Nexus 1000v infrastructure it is manageable.

Now your Nexus 1000v VSM’s are happy and you are not wasting resources in your HA cluster by calculating larger slot sizes than are required.