MPLS L3VPN’s – Part 2

In Part 1 of MPLS L3VPN’s we talked about the different components and technologies that form the IP VPN service, the role of the devices in the service provider connection model and how they interact between each other. In this second part, we will look at four design cases that use MPLS L3VPN’s in today’s modern networks.

Case 1: Service provider – VPN Service

The most popular and proven use case of L3VPN is within the service provider network used as a transit VPN service. Typically, the aim of a service provider offering is to meet the requirements of their customers by offering a service that can:

  • Support a large number of sites per customer
  • Provide customers with a service that can differentiate and transport traffic with end-to-end QoS
  • Provide a flexible and single infrastructure that can serve various media access methods for all customers
  • Reduce the time to provision new sites or to increase bandwidth throughput

All these goals can be achieved by using the following overlay model, also discussed in Part 1 of this blog:


In this architecture, the provider edge nodes (PE’s) receive the customer traffic from the customer edge nodes (CE’s), carries it across the provider nodes (P’s) to its endpoint PE’s using the relevant VPN and transport MPLS labels to finally inject it into the appropriate CE. In this model, the network is used as a transit service only and does not provide services to the customers themselves.

Case 2: Service Provider – Internet and Application Service

The flexibility provided by MPLS VPN infrastructure allows the provider to offer to the customer multiple additional services like Internet access or Application as a service. There are multiple design options to allow the extension of an application through L3VPN’s but the general model can be seen below:


The PE’s at remote sites will exchange routes and data with the PE’s inside the Service Provider data-center. The PE’s in the service provider data-center can be considered as Autonomous System Boundary Router’s (ASBR’s) as they will be at the edge of the Application or Internet service provided to the customers. This use-case is most popular in providing Internet as an additional service to the already existing transit VPN’s to other sites.

Case 3: Enterprise – WAN and Campus Segmentation/Virtualization

Some enterprise opt to use L3 MPLS over their own WAN or over a Service Provider network to offer L3 segmentation and virtualization up to their Campus networks. This is often done in the case of a merger if there is a need to keep the routing infrastructure separated end to end (due to isolated network for example).

In the case where the enterprise has their own managed WAN network the design will be the same as the service provider network (Case 1) or also called a self-deployed L3 MPLS network. If the enterprise opts instead to use a service provider as transit with a L2 VPN service such as VPLS or EoMPLS, the design will look like the picture below.

From the service provider’s perspective:


From the enterprise perspective:


As you can see, the CE’s from the perspective of the service provider are considered the PE’s for the enterprise. This allows the enterprise to run MPLS and MP-BGP between their PE’s over the WAN of the service provider’s network and extend it inside of the campus network. Using L3 VPN’s and extending them to the campus network from PE1 allows the use of an overlapping infrastructure:


In some cases, the WAN service offered might be a L3 service itself or the enterprise might run a pure IP core backbone and running MPLS natively over the transit network won’t be feasible. In this case, a GRE or mGRE tunnel over MPLS design model can be used as seen below:


One last design option is the MPLS over GRE + IPSEC model or even the MPLS over DMPVN (2548oDMVPN) design with a public network as transit, like the the internet for example. This offers a low cost, scalable MPLS overlay.

In general, L3VPN’s within the enterprise is not as popular due to the lack of MPLS knowledge for enterprise engineers. Another factor contributing to the lack of popularity of this design is that the number of different VPN’s to be maintained within most enterprise networks is often fairly low so scalability is not an issue. If scalability is not a design concern and MPLS is not a familiar technology by the supporting staff, a VRF-lite only design model often becomes a better option in terms of manageability.

Case 4: Data-Center – Multi-Tenancy

In the past few years, the CLOS or Spine/Leaf architecture has been increasing in popularity in the data-center environments due to its several benefits to bandwidth availability and efficiency in east/west traffic patterns.  Although a L2 model using TRILL/Fabricpath/SPB could be used, L3 ECMP is another option available that offers most of the same benefits. It is also possible to use L3 ECMP with VRF-lite to create hop-by-hop segmentation with an IP data-plane to provide multi-tenancy, however we will focus on the option of using MPLS with L3VPN’s. In fact, MPLS L3VPN’s offer several benefits that VRF-lite segmentation does not. The first one being scalability which is very important for most host/cloud providers due to the sheer number of customers that they have to support. The second being convergence time due to the availability of MPLS tools like Fast Reroute and Path/Node protection. Lastly, there is more complexity involved with the VRF-lite model due to the requirement from customers to stitch VRF-lite to MPLS segments at the data-center edge. Let’s take a look at the architecture itself:


Comparing to the spine/leaf architecture, the PE’s would be represented by the Leaf nodes and the P’s would be the Spine nodes. As usual, alink-state routing protocol underlay would run between P devices and PE to P devices with an MP-BGP overlay between PE devices. If there is high number of PE’s, it is possible to use a hierarchical model by using two of the devices as Route Reflectors for the VPNv4 peering’s.

In conclusion, in this post we covered different use-cases for L3VPN’s in the service provider, enterprise and data-center environments. In the next post, we will cover several deployment scenarios and issues encountered in different L3VPN’s topologies.


MPLS L3VPN’s – Part 1

MPLS Layer 3 VPN’s (MPLS L3VPN) or also called MPLS IP VPN, is one of the most used application of MPLS in today’s networks. This service is most popular with Service Provider’s but can also be found in Enterprise WAN and Data-center environments. This post will focus on an overview of the different technologies used to create an MPLS IP VPN network and how they interact with each other.

MPLS L3 VPN is, like the name says, a way to create a Layer 3 VPN by harnessing the power of MPLS. The terminology “L3” or Layer 3 refers to the third layer of the OSI model and it is basically just a fancy way of saying that we are segregating at the network layer by creating a separate routing and forwarding table for each VPN. This means that for each VPN we can have an overlapping IP addressing topology running on a shared backbone network.

MPLS VPN Connection Model:

Let’s start by looking at a common topology for an MPLS VPN network. This is what a simplified Service Provider network would look like:

2015-11-14 12_40_05-untitled

The purpose of the network in this example is to send traffic for each customer from one site to another over the routed Service Provider network while maintaining a separate routing and forwarding instance for each customer. In an enterprise environment, the same topology could be used but with the goal being instead of separating traffic between different departments, applications, groups, services or any other logical domain that might be a requirement of the business. There are different components and technologies used in this network and we will explore them one by one separately.

The first part of this network is the Core MPLS Network, made of Provider (P) devices.

2015-11-14 14_29_57-untitled

P devices sit inside of the MPLS network and run a link-state routing protocol (OSPF or IS-IS) with an MPLS distribution protocol on all of their interfaces. There are a couple of MPLS distribution protocols that can be used: Label Distribution Protocol (LDP), Resource Reservation Protocol with Traffic Engineering Extensions (RSVP-TE), Constraint-based routed LDP (CR-LDP), Multiprotocol BGP (MP-BGP) and Segment Routing, but we will focus on LDP for simplicity and because it is possibly the most widely used.

A quick overview of LDP:

Label Distribution Protocol (LDP) is a protocol used to form Label-Switched Paths (LSP’s) by using the existing routing table made by an IGP. LSP’s are sequences of MPLS enabled devices that forward packets of a certain Forwarding Equivalence Class (FEC). A FEC is a set of packets that a single device forwards the same next hop, with the same interface and the same treatment. If this all sounds confusing and unclear, just think of an LSP as a unidirectional tunnel between two devices that share common characteristics. Without going into too much details on this technology, LDP will create an LSP using P devices and it will be used to forward customer traffic across the MPLS network.

The second part of this network is the Provider Edge (PE) devices.

2015-11-14 14_31_29-untitled

PE devices sit at the edge of the MPLS network in between Provider (P) and Customer Edge (CE) devices. They use MPLS/LDP to the P devices and IP to the CE devices. By running LDP with the P devices, the PE’s are able to create an LSP to the other PE’s. The labels forming this LSP will be used for forwarding packets over the MPLS core and are more commonly called the IGP label or Transport Label.

PE’s also need to create an MP-BGP connection to other PE’s to exchange VPNv4 information as follow:

2015-11-14 15_40_24-untitled

We will see later on what is contained exactly in these BGP updates but one of the key information exchanged will be the VPN address that will be imposed as a label, more commonly known as the VPN label.

PE devices also have another very important role and that is to hold a separate routing instance for every CE. This is made possible by using a separate Virtual Routing and Forwarding (VRF) instances for each customer.

A quick overview of VRF’s:

Virtual Routing and Forwarding (VRF) is a technology that allows multiple routing and forwarding tables to exist on a single device. It is basically the equivalent of a VLAN for Layer 3 segmentation. The use of a VRF alone is called VRF-lite, but within the context of MPLS it is just called a VRF. The difference really is that VRF-lite does not make use of two critical components used in MPLS IP VPN’s: the Route Distinguisher’s (RD’s) and Route Target’s (RT’s).

Route Distinguishers (RD’s) are defined within a VRF’s configuration and their only purpose in L3VPN’s is to make an IPv4 prefix globally unique for route exchange. This is important so that the service provider can distinguish between two same prefixes from different customers.

Let’s say Customer 1 and Customer 2 both have the network and are advertising this prefix through the Service Provider network. How would the Service Provider know from what customer this route came from? It can’t, unless there is something to differentiate both routes. That is exactly why the route distinguisher exists. During the route exchange process, the route distinguisher will be appended on the IPv4 address as follows:

2015-11-14 16_49_08-Untitled - Notepad

Route Target’s (RT’s) are also defined within a VRF configuration and in a similar format as the Route Distinguisher but they have a completely different purpose. Their role is to tell PE device which prefixes should be exported or imported in the VPNv4 routing table using a BGP extended Community Attribute.

Again, we will see how this whole process works later on this post but for now just remember that the Route Distinguisher makes an IPv4 address unique across the MPLS network and the Route Target defines which prefixes get imported or exported on the PE devices.

The third and final part of the network that we need to look at are the CE devices:

2015-11-14 17_22_57-untitled

CE devices sit at the edge of the network and they exchange IP routes with the PE devices using an IP routing protocol (BGP, OSPF, EIGRP, IS-IS, RIP). On the CE devices, the routing is done in the global routing table (normal non-VRF routing table) but on the PE, the routing is done in a separate VRF for each customer. VRF’s are locally significant and this means that the CE does not need to know if the PE interface is in a VRF or not. From the perspective of the customer, the routes advertised to the PE device are tunneled transparently across the service provider network and advertised to the CE at the other end without any special configuration.

To better understand the whole process, let’s look at the life of an IP packet as it moves across the Service Provider network from its signaling (control-plane) to it’s forwarding (forwarding plane). For simplicity, we will only look at Customer1 in a single direction (uni-directional traffic).

  1. The first step is to have LDP to do it’s magic. LDP Label Binding and signaling will be done hop by hop from the loopback’s of a PE to the other PE device, forming an LSP.

2015-11-15 12_37_42-untitled

In this case the LSP is created for Loopback0 ( prefix) and the labels signaled are: Null -> 33 -> 16 -> 40. The first label is always signaled as Null because of the Penultimate hop popping (PHP) function. This process is simply an optimization to avoid a double lookup of the label at the last hop device of the LSP; the Label Edge Router (LER). You can read more on this feature if you check RFC3031 section 3.16

  1. In the second step, the customer sends a route update from the CE device and it is advertised through the routing protocol configured between the CE and PE device. When it reaches the PE device, it is signaled through MP-BGP to the other PE as a VPNv4 route with a Route-Distinguisher attached to the prefix, a Route-Target and a Label. In this case, the route 200:1: is sent with Route-Target 200:1 and Label 21.

2015-11-15 12_57_24-untitled

The update packet in Wireshark  looks like this:


In the first box you can see the Route-Target value of 200:1. You can also see in the third box the Label value sent of 21 for the IPv4 route Finally, in the last box you can see the Route-Distinguisher value that makes the route globally unique.

  1. Finally, the customer at the opposite side sends packets for the signaled route. These packets are forwarded from the CE using IPv4 to the PE. From the PE, the IPv4 packets will be encapsulated with two labels to be forwarded over the MPLS backbone, the VPN and the IGP label. Again, the VPN label was signaled by MP-BGP and tells us in which VRF of the opposing PE to send the traffic into. The IGP label was signaled by LDP and is used to forward packets hop by hop across the MPLS backbone.

2015-11-15 13_15_33-untitled

In this example, when the IPv4 packet for destination enters the ingress PE device from the CE, the PE device looks up the destination IP address ( in its forwarding table and finds the correct VRF to send the traffic by looking at which interface the route entered the PE device in the first place. It encapsulates the packet with both IGP and VPN labels (21 and 40) and sends it to the first P device. Hop by hop, the P devices will change the IGP label based on the BGP next-hop IP address ( and finally remove it at the last P before entering the PE device because of the Null label. The last hop PE device looks up the VPN label in its forwarding table and forwards the packet without any labels (as an IP packet) towards the CE device.

Again, this example only showed unidirectional forwarding. If bi-directional traffic was required, Step 1 and Step 2 would have to happen again in the reversed direction for the route back towards the source to be installed in the control-plane of the remote CE device.

In conclusion, MPLS L3VPN is a mix of several different protocols forming a highly scalable and flexible VPN solution. In Part 2, I will be looking at different use cases for MPLS L3VPN’s and some of the most common deployment scenarios.


During my CCIE R&S studies I purposely avoided creating any content related to MPLS (Multiprotocol Label Switching). The problem with MPLS is that once you start talking about it, you are opening the doors to a much larger and deeper discussion on various other technologies. At the time, I didn’t feel quite prepared to embark on this path but after spending several months exploring this technology, I think I can bring some valuable input on the matter. This will be the first out of several posts on MPLS.

I still remember first hearing about MPLS from other engineers back when I was still new in the networking industry. They explained to me that it could do VPN’s, traffic engineering, advanced switching, guarantee bandwidth and basically solve any problem I’ve ever had or would ever have. I was really impressed by all of this and wanted to learn more so I looked it up on my good old search engine to find some more information. To my surprise, the only information I could find on MPLS was that it was some kind of technique to direct packets based on labels. I did find a lot of other documentation talking about MPLS and MP-BGP, VRF’s, RSVP, LDP/TDP, etc. but all of that was way over my head and too complicated for me at the time. Fast forward a couple of years and I wish that at the time those network engineers would have explained MPLS to me in a clearer manner for me to grasp. Let me attempt to explain MPLS in the way I think they should of.

What is MPLS?

Multiprotocol Label Switching (MPLS) is, like I said before, a technique to direct packets based on labels. The key word to remember in this sentence is “technique” and note that technique does not mean it is a service. What i’m trying to say here is that MPLS doesn’t really do anything on its own, it is simply a way to enable services using other technologies and protocols. In fact, the real use case of MPLS is that it creates a new forwarding paradigms that is not available in conventional IP routing. The role of this paradigm is to de-couple IP packet forwarding from the information carried in the packet header itself thus allowing us to forward packets based on labels.

So if MPLS is just a technique and does not do anything on its own, why should I care to implement it in my network?

Well the benefits of MPLS, using this new forwarding paradigm, is that it enables you to create new services for your network. Some of these services are:


-Traffic Engineering and congestion management: MPLS-TE, RSVP-TE

-Failover services: FRR, Link and Node protection

How does it work?

This is what an MPLS label looks like:


Pretty simple isn’t it? This label is inserted between the Layer 2 Header and Layer 3 portion of a packet by using protocol’s like Label Distribution Protocol (LDP) or Tag Distribution Protocol (TDP) as below:


Once packets are tagged or labeled, there are different ways to create MPLS services by using BGP, RSVP, VRF’s and other protocols but I will cover this in future blogs.

The important point to remember from all of this is that MPLS is a technique that creates a new forwarding paradigm and uses several other protocols and technologies to enable services used in today’s modern networks. In my next post, I will go over one of the more popular use-cases for MPLS right now: MPLS L3VPN’s.

Hour 80: Routing Traffic Isolation using VRF’s

A lot of engineers when they hear Virtual Routing and Forwarding (VRF’s) they think MPLS VPN’s and associate it to a Service Provider feature. Although VRF’s are widely used with MPLS in SP environments, in today’s networks, a lot of companies will opt to use the VRF feature alone in order to obtain L3 isolation. Similar to VLAN’s that isolate broadcast domains at L2, VRF’s (also called VRF-lite) isolate routing information at L3 by creating local virtual routing tables.

The main reason, that I’ve personally implemented VRF’s in networks, was in order to join several private overlapping address topologies. Even though VRF-lite is only locally significant, GRE tunnels can be used to extend VRF information. In fact, you can make a GRE tunnel VRF-aware with the use of the “tunnel vrf <vrf name>” command. While VRF’s offer a layer of security by isolating routing information locally, it does not offer any kind of security when passing data from one point to the other of the network, and for that reason, I would always suggest coupling GRE tunnels with IPSEC.

A common misconception I’ve heard from network engineers is that since a network is already using VRF’s you might as well implement MPLS to increase the forwarding speed of the network. This is a bogus argument as with CEF enabled on all routers, MPLS will not offer any speed increase at the forwarding level. In fact, you will lose speed as the overhead of the MPLS + IP calculation factor will make your forwarding decision process a little slower. It’s very important to note that MPLS and VRF are two different separate technologies that aren’t interdependent of each other at all but rather compliment one another in certain situations.