CDN Performance Analyzer

DESCRIPTION AND METHODOLOGY

CDN Performance Analyzer  

Discover how CDNs can improve web application performance around the world

Modern web applications are very often a mix of elements provided by both the application’s owner and third-parties service providers. For example, even a simple e-commerce application might include a shopping cart from one service provider, directions from a map provider, advertisements from an ad service provider, personalization from a recommendation service, usage of a content delivery network (CDN), and web analytics. The existence of some of these third-party services will be obvious to even the casual user: Google Maps, for example, has a distinct look and feel--and carries the Google logo. Others are much less obvious or completely anonymous to all but the most sophisticated user. In fact, our research shows that commercial web sites use an average of more than 9 different services supplied from outside the domain of the application’s owner. A significant percentage of these third-party services make calls to Cloud infrastructure or platform service providers such as Amazon or Google.

We call these web applications “borderless applications” to highlight their dependency on services that are outside the organizational and geographic “borders” of the applications’ owners. (The term “composite application” is also sometimes used to describe these applications, but is less precise and fails to capture a critical attribute of these applications: it isn’t so much that they are a mix of elements, but that the elements may be outside the direct control of the application’s owner.) 

The performance of a borderless application is influenced by a number of factors, including coding style and browser characteristics, but is highly dependent upon the performance of the services it employs. The effect of any individual service on an application’s performance is neither linear nor independent of the performance of other services. For example, a 50% decrease in static image load time will not necessarily translate into a 50% decrease in page load time. It may for some users, but for others it may have absolutely no impact. Page rendering may also dependent on all services required to render the page and their performance.

It is impractical to attempt to illustrate the effect of all of these factors on end user experience in a single demonstration. Instead, the CDN Performance Analyzer concentrates on services commonly used in deployed applications. Specifically, in this release we focus edge caching services.

Methodology

Overview | Target Application | Reported Metrics | Comments 

The CDN Performance Analyzer focuses on visualizing the dynamic nature of borderless application performance as seen by a geographically diverse user community. A reference application is deployed to a single, globally accessible hosting site. Web transactions are run continuously against the reference application using the Gomez Performance Network – Compuware's global network of instrumented transaction agents. The data collected from these agents are collected and aggregated into a cloud performance database. The CDN Performance Analyzer enables users to interact with the data from the Cloud performance database.

Reference Application (Borderless Application)

The reference application is designed to be a representative proxy for the type of applications that practitioners are currently deploying using web and Cloud services. Specifically, the reference application is a trivial two-page catalog web site that contains:

  • Static content, include text and images of various sizes
  • Mash-content from common map providers
  • Advertisements from commercial ad providers
  • Analytics

This reference application can be deployed without modification to either IaaS of PaaS service providers. It can readily take advantage of edge caching delivery services which are available from multiple providers. The reference application contains static content of various sizes and types on two pages to avoid favoring any specific edge caching content delivery network (CDN) provider.

The reference application is complex enough to be somewhat representative of an actual e-commerce application, thereby avoiding the perils of micro-benchmarks, but simple enough to be readily understood and easily deployed.

The first page of the reference application (see below) contains small static images interspersed with text. The second page contains a single large (approximately 1MByte) image, an advertisement supplied by an ad service, and a map provided by a map service. Analytics are embedded on both pages.

PAGE 1PAGE 2

 

Hosting Site

To minimize the number of extraneous variables, allowing CDN Performance Analyzer users to focus on the contribution of other Cloud services such as CDNs, map and advertisement services, this visualization uses a single infrastructure as a service (IaaS) provider to host the reference application. The reference application is being hosted in a virtual machine located in Amazon’s EC2 East Region.

 

Test Transaction Profiles

Three test transaction profiles have been defined for this release of CDN Performance Analyzer. The profiles are identical except for the services used to in the transaction. Each of the reference application's two pages are completely loaded including the pages’ root object as well as all referenced objects, JavaScript, Cascading Style Sheets, embedded services, and any other related content. The transaction completes when the page is completely loaded.

This release focuses on Content Delivery Network (CDN) service performance. As such, test transaction profiles vary just the CDN used to cache the pages’ static content. The three profiles are:

  • Origin only (Amazon EC2 East - No CDN)
  • CDNetworks
  • Amazon’s CloudFront

To minimize the number of extraneous variables, the service configuration is specified to the reference application in the URL. Specifically, a query string parameter is provided on the URL to select the CDN to be used. JavaScript on the page modifies the base URL of all the static content that can be loaded via the CDN, and the rest of the page is loaded accordingly.

Measurement Approaches

When comparing performance of different Cloud delivery systems, the key measurements are end-user performance (or end-user experience), coupled with consistency of performance. Users demand fast and reliably consistent page load times.

To collect these measurements, Compuware drives test transactions continuously against the reference application using the Gomez Performance Network (GPN). The CDN Performance Analyzer uses Gomez Last Mile peers to measure end-user experience. Since service providers use different delivery approaches, we ensure that our methodology doesn't unfairly favor one over another by always reporting response times from the end-user perspective as collected by Gomez Last Mile Peers.

Gomez Active Backbone nodes measure provider consistency and availability as perceived at key backbone locations.

Gomez Last Mile Peers

Gomez Last Mile Peers are 150,000+ real, consumer-grade desktops connected to 2,500+ local ISPs and wireless carriers located in 168+ countries. Since these agents are running on real user machines, they represent the type of experience actual users might encounter interacting with the test target application.

To enable the user to get a sense for real end user experience, reported end-user experience times are only reported from Last Mile Peers. This is especially important when dealing with services such as CDN's, who often locate their servers in the same data centers as the Gomez backbone nodes. Co-location eliminates network-induced latency and variability, which would result in unrealistic end-user performance measurements.

Only high-bandwidth peers are used to run test transactions. Results are then filtered based upon reported peer network throughput to eliminate peers suffering with grossly underperforming network connections. Individual transactions run from last mile peers exhibit a misleading amount of variability, so aggregated results are always reported. (The one exception of this is the "Ticker" feed at the bottom of the application, which provides a near real-time view of Last Mile data tests). Because of resource constraints and the requirement to filter and aggregate responses from multiple last mile peers for each reported result, last mile results are reported on a regional basis.

Gomez Active Backbone Nodes

Gomez Active Backbone nodes are enterprise-class servers located in data centers with high-bandwidth, direct connections to the Internet backbone. Since these nodes are resourced-managed and use high-bandwidth connections, they generate highly accurate and consistent test loads with little network-induced variability. This makes them ideal for understanding intra-service consistency.

Test transactions are from 35 backbone nodes located around the world: 18 nodes are located in the United States and 17 are located outside of the United States. The distribution of the nodes located in the United States is designed to be representative of six geographic regions described by http://www.fcc.gov/oet/info/maps/areas/.

The distribution of backbone nodes located outside the United States is selected to represent major population centers.

The following is a list of defined regions and the backbone nodes that are within those regions.

RegionIncluded Countries or U.S. StatesTest RateBackbone Nodes [hosting]
CAN  (Canada)Canada6 / hourToronto, Canada  [Fusepoint/Savvis]
EAS  (East Asia)China, Hong Kong, Japan, Macao, “Taiwan, Province Of China”6 / hourBeijing, China  [China Unicom]
Tokyo, Japan  [NTT]
EEU  (Eastern Europe)Albania, Bulgaria, Belarus, Czech Republic, Estonia, Finland, Greece, Hungary, Lithuania, Latvia, "Moldova, Republic of", "Macedonia, the Former Yugoslav Republic Of", Montenegro, Norway, Poland, Romania, Serbia, Slovakia, Sweden, Ukraine6 / hourHelsinki, Finland  [TeliaSonera]
Oslo, Norway  [Song Networks]
Stockholm, Sweden  [Telia]
 
LAM  (Latin America)Aruba, Netherlands Antilles, Argentina, Antigua and Barbuda, Bahamas, Brazil, Barbados, Chile, Colombia, Costa Rica, Cuba, Dominica, Dominican Republic, Ecuador, Grenada, Guatemala, Honduras, Jamaica, Mexico, Martinique, Nicaragua, Panama, Peru, Puerto Rico, Paraguay, El Salvador, Trinidad and Tobago, Uruguay, Venezuela,"Bolivian Republic of”6 / hourBuenos Aires, Argentina  [Telefonica]
Sao Paulo, Brazil  [Global Crossing]
 
MEA  (Middle East)Afghanistan, United Arab Emirates, Bahrain, Cyprus, Egypt, "Iran, Islamic Republic Of", Israel, Jordan, Kuwait, Oman, Pakistan, Qatar, Saudi Arabia, Syrian Arab Republic, Turkey, Yemen6 / hour   N/A
NCA  (North-Central Asia)Georgia, Kazakhstan, Mongolia, Russian Federation, Uzbekistan6 / hourN/A
OCE  (Australia-Oceania)Australia, “Micronesia, Federated States Of”, New Caledonia, New Zealand, Vanuatu6 / hourSydney, Australia  [Telstra]
SAS (South Asia)Bangladesh, India, Sri Lanka, Nepal6 / hour Mumbai, India  [VSNL]
 
SEA (South-East Asia)Brunei Darussalam, Indonesia, Cambodia, "Lao People's Democratic Republic", Malaysia, Philippines, Singapore, Thailand, Timor-Leste, Viet Nam6 / hour Kuala Lumpur, Malaysia  [VSNL]
Singapore, Singapore  [SingTel]
US (Central/Mountain)                            Arizona, Colorado, Iowa, Kansas, Missouri, Montana, Nebraska, New Mexico, North Dakota, Oklahoma, South Dakota, Texas, Utah, Wyoming6 / hour

Mesa, AZ
Dallas, TX
Houston, TX
 

US (Great Lakes)Illinois, Indiana, Michigan, Minnesota, Ohio, Wisconsin6 / hourChicago, IL
Kansas City, MO
US (Mid-Atlantic)Deleware, Kentucky, Maryland, New Jersey, North Carolina, Virginia, Washington D.C., West Virginia6 / hour

Philadelphia, PA
Reston, VA
Washington, D.C.

US (Northeast)Connecticut, Maine, Massachusetts, New Hampshire, New York, Pennsylvania, Rhode Island, Vermont6 / hour

Boston, MA
Newark, NJ
New York, NY

US (Pacific)Alaska, California, Hawaii, Idaho, Nevada, Oregon, Washington6 / hourDenver, CO
Los Angeles, CA
San Diego, CA
San Jose, CA
Seattle, WA
US (Southeast)Alabama, Arkansas, Florida, Georgia, Louisiana,    Mississippi, South Carolina, Tennessee6 / hourAtlanta, GA
St. Louis, MO
WEA  (Western Europe)Andorra, Austria, Belgium, Bosnia and Herzegovina, Switzerland, Germany, Denmark, Spain, France, United Kingdom, Guernsey, Gibraltar, Croatia, Isle of Man, Ireland, Iceland, Italy, Jersey, Luxembourg, Malta, Netherlands, Portugal, Slovenia6 / hourCopenHagen, Denmark  [LambdaNet]
Paris, France  [France Telecom]
Frankfurt, Germany  [Deutsche Telecom]
Bern, Switzerland  [Sunrise]
London, United Kingdom  [Telstra]
 
                                                                                                                                                                        

 

  

Reported Metrics

Gestalt World View

The world-view map presents regional response times in one of four colors: green, yellow, red, or gray. The map is intended to provide context for additional exploration. The colors are intended to give an overall, subjective perspective on performance rather than an objective measurement. The thresholds for green, yellow, and red, defined in the Response Time Key at left, are admittedly arbitrary and reflect the large number of objects in the reference application. No data is available from grayed-out regions.

The default world-view is colored based on data retrieved using the origin-only test transaction profile. As such, it reflects the end-user experience for an application that is not utilizing edge caching. Selecting the “Edge Cached” view from the radio box in the legend causes the world view to be re-colored based upon each region's best performing edge caching service. The color of two adjacent regions may be based on the results retrieved from two different CDN services. Pop-up displays (tooltips) in each region identify the best performing CDN for that region (see below).

Regional End-User Experience Tooltips

Hovering the mouse over any non-gray region displays the regional end-user experience tooltip. Three numbers are reported in this tooltip:

Best Performing Service Response Time 
The average response time recorded for this region over the last 12 hours (30 minute accuracy) for the best performing edge cache service in this highlighted region. The value is colored to match the User Experience and Consistency charts (described below). The name of the best performing edge cache service for that region is displayed under the value. To provide better consistency without directly removing outliers, only tests with a bandwidth at the test client between 2000 and 5000 kb/s (kilo bits per second) are used (the Gomez GPN provides a bandwidth test for each last mile client immediately before and after each test, and this measured bandwidth is averaged for use in this filtering).

Origin-only response time
The average response time recorded for this region for the origin-only profile

Percent improvement
The improvement over origin-only
 

Service Availability Tooltip

Hovering the mouse over a backbone node (indicated by a black dot on the map) causes the availability tooltip to be displayed. Availability is calculated as the percentage of successful tests of the web site, where “success” is defined as the successful loading of all pages and all of their objects from the web site, and displaying the pages within a reasonable time frame (currently 160 seconds per page maximum). A single test is a transaction across two pages, so the availability requires success of each page in order for the test to be a success. Errors in certain page objects can exist, so long as they do not block the rest of the objects from loading and the Browser client can respond with an OnLoad complete event.

Regional End-User Experience Trend & Consistency Charts

Left clicking on any non-gray region causes the trend charts to be updated to reflect the clicked region and closest city.

User Experience Chart
This chart displays the average response time for each of the edge caching service providers, as well as for the origin / baseline (Amazon EC2 East) for comparison.  Response times are averaged over intervals of 12 hours covering a 7 day period.  Test failures are not included in the average response time calculations.  To provide better consistency without directly removing outliers, only tests with a bandwidth at the test client between 2000 and 5000 kb/s (kilo bits per second) are used (the Gomez GPN provides a bandwidth test for each last mile client immediately before and after each test, and this measured bandwidth is averaged for use in this filtering).

User Experience Chart – Load Time Details
The Load Time Details button on this chart opens a new pane that shows comparative performance analysis for the origin and edge cached sources as measured from the currently-selected region. The intent is to help viewers understand where edge caching will improve performance. For each source, the CDN Performance Analyzer shows four activity timelines, one for each of the following content types:

  • Other: represents analytics and other third-party services
  • Map: represents third-party map services
  • Ad: represents third-party advertising services
  • Static: represents cacheable content provided by the site owner

The bold portions of each timeline represent when that content type is actively transferring data to or from the browser. The thinner portions of the line reflect total page load time, and the overall length of all four lines in a group will always be the same length.

Click on “More Details” to see a complete Object Breakdown by Page, as hosted on the Gomez Performance Network portal. This is the most granular view of page performance, by object and connection. (Note that the User Experience Chart – Load Time Details transaction time shown at the bottom of the waterfall page, which matches the transaction time from the Detail View, is the transaction time for the two measured pages – denoted as “Page 0: …” and “Page 1: …”. However, the waterfall includes 3 additional “pages” that are calls to ad, map, and analytics sources – the Detail View uses these calls internally to clarify the distribution of time attributed to these three sources.) As a viewer of this waterfall, you will be most interested in the objects that precede the start of “Page 2:”

Consistency Chart
The intent of this chart is to provide insight into how consistent a particular system is operationally; whereas user experience measures performance from an end user's perspective, this data, like the data used in the calculations of the Service Availability Tooltip, is derived from tests at Gomez Backbone nodes (also described above).

The chart displays data over a 7 day period, with intervals of 1 hour.  Actual tests are taken at each backbone location four times per hour, allowing for an average of 4 tests per data point.

The data displayed is the percentage difference from the average data for each test scenario, thereby “normalizing” the graph lines to center around a 0 percentage line [horizontally].   Individual test results are considered outliers and removed from the calculations when they are beyond 6 standard deviations for the 7 day period and there are no adjacent tests outside 6 standard deviations.   This removes anomalies, but allows for the capture of significant events that measure the operational consistency of the service.

The more a line stays close to the zero (0) line, the more “consistent” the service is.   This chart provides a quick, visual comparison of the consistency of the services in a specific geography (backbone node location).  

Consistency Chart - Standard Deviation

To simplify the assessment of what the consistency chart shows, CloudSleuth reports a reading of standard deviation for the measurements over the displayed 7-day period– lower standard deviations indicate better consistency.

Comments

Further discussion regarding the CDN Performance Analyzer methodology can be found here - comments