UI Performance Analysis via Selenium WebDriver

The article from the series Automation Tools reviews different approaches to check the UI performance of web apps reusing your existing functional Selenium WebDriver tests. We will investigate the native ChromeDriver GetLog feature, the new Selenium 4 DevTools protocol integration. The central part of the publication is dedicated to the integration of Google Lighthouse and Selenium. We will create a full-fledged testing library out of this integration. Moreover, in the end, you will be able to run the Lighthouse analysis against pages behind login after executing any required steps.

User-centric Performance Metrics

As you know the performance of our website apps is essential. But when we use the word "fast" in this context what do we mean?

Well, the performance turned out is something relative. A website might be fast for one user situated in Europe and slow for another who lives on an island. Two websites may finish loading relatively at the same time however one of them may seem that it is loading much faster. Another may appear to load fast but to respond poorly on user interactions.

Over the past few years, members of the Chrome team—in collaboration with the W3C Web Performance Working Group—have been working to standardize a set of new APIs and metrics that more accurately measure how users experience the performance of a web page. They tried to give answers to a few questions: "Is it happening?", "Is it useful?", "Is it usable?", "Is it delightful?".

Types of Metrics

There are several other types of metrics that are relevant to how users perceive performance.

Perceived load speed: how quickly a page can load and render all of its visual elements to the screen.

Load responsiveness: how quickly a page can load and execute any required JavaScript code for components to respond promptly to user interaction

Runtime responsiveness: after page load, how quickly can the page respond to user interaction.

Visual stability: do elements on the page shift in ways that users don't expect and potentially interfere with their interactions?

Smoothness: do transitions and animations render at a consistent frame rate and flow fluidly from one state to the next?

Important Metrics to Measure

First contentful paint (FCP) measures when the page starts loading to when any part of the page's content is rendered on the screen.

Largest contentful paint (LCP): measures when the page starts loading to when the largest text block or image element is rendered on the screen.

First input delay (FID): measures when a user first interacts with your site (i.e., when they click a link, tap a button, or use a custom, JavaScript-powered control) to the time when the browser is actually able to respond to that interaction.

Time to Interactive (TTI): measures the time from when the page starts loading to when it's visually rendered, its initial scripts (if any) have loaded, and it's capable of reliably responding to user input quickly.

Total blocking time (TBT): measures the total amount of time between FCP and TTI where the main thread was blocked for long enough to prevent input responsiveness.

Cumulative layout shift (CLS): measures the cumulative score of all unexpected layout shifts that occur between when the page starts loading and when its lifecycle state changes to hidden.

You can read the full article here.

Test Case

For the examples, I created a simple test against our BELLATRIX demo website. The test simply logs into my account section. I wanted to do a real-world example where we don't test just a simple not-protected web page.

bellatrix-login

Here is the main body of the test - nothing really special to mention. We use the WebDriverManager library to download the proper driver. The interesting part comes next - how to get the performance data about the page.

private static ChromeDriver _driver;
[SetUp]
public void Setup()
{
new DriverManager().SetUpDriver(new ChromeConfig(), VersionResolveStrategy.MatchingBrowser);
var chromeDriverService = ChromeDriverService.CreateDefaultService();
ChromeOptions options = new ChromeOptions();
_driver = new ChromeDriver(chromeDriverService, options);
_driver.Manage().Window.Maximize();
}
[TearDown]
public void TearDown()
{
_driver?.Close();
_driver?.Quit();
}
[Test]
public void LighthouseMetricsCheck()
{
_driver.Navigate().GoToUrl("http://demos.bellatrix.solutions/");
var myAccountLink = _driver.FindElement(By.LinkText("My account"));
myAccountLink.Click();
var userName = _driver.FindElement(By.Id("username"));
userName.SendKeys("info@berlinspaceflowers.com");
var password = _driver.FindElement(By.Id("password"));
password.SendKeys("@purISQzt%%DYBnLCIhaoG6$");
var loginButton = _driver.FindElement(By.XPath("//button[@name='login']"));
loginButton.Click();
WaitAndFindElement(By.LinkText("Orders"));
}
private IWebElement WaitAndFindElement(By by)
{
var webDriverWait = new WebDriverWait(_driver, TimeSpan.FromSeconds(30));
return webDriverWait.Until(SeleniumExtras.WaitHelpers.ExpectedConditions.ElementExists(by));
}

Chrome Performance Logs

To turn on the native Chrome performance logs, you need to make some changes to the ChromeOptions. First, we need to enable the profiling and call the SetLoggingPreferences method. I added a few more arguments so that the tests run a little bit faster.

[SetUp]
public void Setup()
{
new DriverManager().SetUpDriver(new ChromeConfig(), VersionResolveStrategy.MatchingBrowser);
var chromeDriverService = ChromeDriverService.CreateDefaultService();
ChromeOptions options = new ChromeOptions();
options.AddAdditionalCapability(CapabilityType.EnableProfiling, true, true);
options.SetLoggingPreference("performance", LogLevel.All);
options.AddArgument("--disable-gpu");
options.AddArgument("--no-sandbox");
options.AddArguments("--disable-storage-reset");
_driver = new ChromeDriver(chromeDriverService, options);
_driver.Manage().Window.Maximize();
}

This is how we access the logs.

Console.WriteLine("Chrome Performance Metrics");
var logs = _driver.Manage().Logs.GetLog("performance");
for (int i = 0; i < logs.Count; i++)
{
Debug.WriteLine(logs[i].Message);
}
File.WriteAllLines("chromeMetrics.txt", logs.Select(m => m.Message));

The output is too large to include it here. You need to check it yourself to find out whether it is helpful to you or not. To be honest, I prefer the Lighthouse approach more, but this might be useful too.

Chrome DevTools Protocol Performance Checks

As you probably already know, the new Selenium 4 provides an integration with the Chrome DevTools Protocol. At the time of writing is still in Beta, so from version to version, there are many changes. However, you can check the article - Most Complete Selenium WebDriver 4.0 Overview to get insights on what to expect. The protocol allows us to use the Chrome performance tooling.

The DevTools main class is Chrome version-specific. This is why the using statement needs to mention the version: 

using DevTools = OpenQA.Selenium.DevTools.V91;

This is how we need to change the setup to use the DevTools performance tools.

private static ChromeDriver _driver;
private static IDevToolsSession _devTools;
private static DevTools.DevToolsSessionDomains _devToolsSession;
[SetUp]
public void Setup()
{
new DriverManager().SetUpDriver(new ChromeConfig(), VersionResolveStrategy.MatchingBrowser);
var chromeDriverService = ChromeDriverService.CreateDefaultService();
ChromeOptions options = new ChromeOptions();
_driver = new ChromeDriver(chromeDriverService, options);
_driver.Manage().Window.Maximize();
_devTools = _driver.GetDevToolsSession();
_devToolsSession = _devTools.GetVersionSpecificDomains<DevTools.DevToolsSessionDomains>();
_devToolsSession.Performance.Enable(new DevTools.Performance.EnableCommandSettings());
_devToolsSession.Network.Enable(new DevTools.Network.EnableCommandSettings());
}

At the end of the test you can call the following code to display or save the performance data.

Console.WriteLine();
Console.WriteLine("DevTools Performance Metrics");
var metrics = _devToolsSession.Performance.GetMetrics();
foreach (var metric in metrics.Result.Metrics)
{
Console.WriteLine($"{metric.Name} = {metric.Value}");
}
File.WriteAllLines("devToolsMetrics.txt",
_devToolsSession.Performance.GetMetrics().Result.Metrics.Select(m => $"{m.Name} = {m.Value}"));

Below you can find the output:

Timestamp = 18072.3841
AudioHandlers = 0
Documents = 7
Frames = 2
JSEventListeners = 197
LayoutObjects = 265
MediaKeySessions = 0
MediaKeys = 0
Nodes = 2829
Resources = 114
ContextLifecycleStateObservers = 21
V8PerContextDatas = 4
WorkerGlobalScopes = 0
UACSSResources = 0
RTCPeerConnections = 0
ResourceFetchers = 7
AdSubframes = 0
DetachedScriptStates = 2
ArrayBufferContents = 6
LayoutCount = 70
RecalcStyleCount = 39
LayoutDuration = 0.061348
RecalcStyleDuration = 0.056053
DevToolsCommandDuration = 0.126399
ScriptDuration = 0.350914
V8CompileDuration = 0.003751
TaskDuration = 0.864955
TaskOtherDuration = 0.26649
ThreadTime = 0.764974
ProcessTime = 1.109375
JSHeapUsedSize = 9161296
JSHeapTotalSize = 14876672
FirstMeaningfulPaint = 0
DomContentLoaded = 18072.116968
NavigationStart = 18069.00376

What Is Google Lighthouse?

Built by Google Lighthouse is an open-source auditing tool for developers. It works over the chrome-dev-tools and runs a series of audits to measure the web application under test on the following: performance, accessibility, SEO, best practices, PWA (progressive web applications). Lighthouse works as a Chrome/Firefox add-ons or in a CLI mode, or as a node application. 

Requirements: Node.js + Lighthouse CLI

npm, install -g lighthouse

To create a solution out of this, one needs some fundamental understanding of how Lighthouse works. 

google-lighthouse-architecture

As could be seen from the diagram, the chrome dev-tools protocol enables us to run the audits. Hence we understand the need for chrome debugs here. Finally, converting the audits into meaningful reporting by comparing the lighthouse standards is done.

One of the primary outputs is a detailed HTML report containing various screenshots, recommendations and scores.

You can run Lighthouse from the shell using the following command:

lighthouse https://demos.bellatrix.solutions/my-account/ --port=5333 --emulated-form-factor=desktop --output=html --output-path=./bellatrix-account.html

There are many arguments and options. Check the official documentation for more information.

Selenium + Google Lighthouse

Lighthouse is using its own Chrome to perform its checks. However, it is possible to use the Chrome debug to connect it to and existing started Chrome on its debug port. This is how we will integrate the both tools.

Workflow

1. Start Chrome in debug mode, exposing a port
2. Perform our WebDriver test
3. Call Lighthouse CLI from C# against the Chrome debug port, specifying that we want not only HTML output but also a JSON one
4. Read the JSON file. Then, deserialize it to C# objects.
5. Perform assertions against the Lighthouse output C# objects

Lighthouse Integration Implementation

We need two helper methods. One for getting a free port and one for executing CLI commands from C#.

private void ExecuteCommand(string command, bool shouldWaitToExit = false)
{
var p = new Process();
var startInfo = new ProcessStartInfo();
startInfo.FileName = "cmd.exe";
startInfo.Arguments = @"/c " + command;
p.StartInfo = startInfo;
p.Start();
if (shouldWaitToExit)
{
p.WaitForExit();
}
}
private static int GetFreeTcpPort()
{
Thread.Sleep(100);
var tcpListener = new TcpListener(IPAddress.Loopback, 0);
tcpListener.Start();
int port = ((IPEndPoint)tcpListener.LocalEndpoint).Port;
tcpListener.Stop();
return port;
}

You can change "cmd.exe /c" with "/bin/bash -c" to be able to run the same command on Linux and OSX systems.

To start Chrome in debug mode you need only two options.

private static ChromeDriver _driver;
private static int freePort;
[SetUp]
public void Setup()
{
new DriverManager().SetUpDriver(new ChromeConfig(), VersionResolveStrategy.MatchingBrowser);
var chromeDriverService = ChromeDriverService.CreateDefaultService();
ChromeOptions options = new ChromeOptions();
freePort = GetFreeTcpPort();
options.AddArgument("--remote-debugging-address=0.0.0.0");
options.AddArgument($"--remote-debugging-port={freePort}");
_driver = new ChromeDriver(chromeDriverService, options);
_driver.Manage().Window.Maximize();
}

With these two lines, we start the Lighthouse analysis. Lighthouse will reuse the current Selenium session and create a new tab in the same browser, which means all cookies and everything will be reused.

ExecuteCommand($"lighthouse {_driver.Url} --output=json,html,csv --port={freePort}", true);
var performanceReport = ReadPerformanceReport();

Here is the method for converting the JSON output to C# objects.

private Root ReadPerformanceReport()
{
string assemblyFolder = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
var directoryInfo = new DirectoryInfo(assemblyFolder);
string pattern = "*.report.json";
var file = directoryInfo.GetFiles(pattern).OrderByDescending(f => f.LastWriteTime).First();
string fileContent = File.ReadAllText(file.FullName);
Root root = JsonConvert.DeserializeObject<Root>(fileContent);
return root;
}

The JSON output is quite large - 10000+ lines. This is why I generated the required C# objects via a website that, by provided JSON, creates the corresponding C# classes. Of course, afterward, I had to clean them a little bit.

json-to-csharp

Here is how you can access the scores and the rest of the data. You can use the same way to peform assertions.

ExecuteCommand($"lighthouse {_driver.Url} --output=json,html,csv --port={freePort}", true);
var performanceReport = ReadPerformanceReport();
Console.WriteLine("Display values");
Console.WriteLine($"{performanceReport.Audits.FirstMeaningfulPaint.Title} = {performanceReport.Audits.FirstMeaningfulPaint.DisplayValue}");
Console.WriteLine($"{performanceReport.Audits.FirstContentfulPaint.Title} = {performanceReport.Audits.FirstContentfulPaint.DisplayValue}");
Console.WriteLine($"{performanceReport.Audits.SpeedIndex.Title} = {performanceReport.Audits.SpeedIndex.DisplayValue}");
Console.WriteLine($"{performanceReport.Audits.LargestContentfulPaint.Title} = {performanceReport.Audits.LargestContentfulPaint.DisplayValue}");
Console.WriteLine($"{performanceReport.Audits.Interactive.Title} = {performanceReport.Audits.Interactive.DisplayValue}");
Console.WriteLine($"{performanceReport.Audits.TotalBlockingTime.Title} = {performanceReport.Audits.TotalBlockingTime.DisplayValue}");
Console.WriteLine($"{performanceReport.Audits.CumulativeLayoutShift.Title} = {performanceReport.Audits.CumulativeLayoutShift.DisplayValue}");

Verify No JavaScript Errors Appeared

As a bonus to the upper-performance verification approaches, I believe it might be helpful on certain occasions to check whether any JavaScript errors appeared.

private static void VerifyNoJavaScriptErrorsAppeared()
{
var errorStrings = new List<string>
{
"SyntaxError",
"EvalError",
"ReferenceError",
"RangeError",
"TypeError",
"URIError",
};
var jsErrors = _driver.Manage().Logs.GetLog(LogType.Browser).Where(x => errorStrings.Any(e => x.Message.Contains(e)));
if (jsErrors.Any())
{
Assert.Fail($"JavaScript error(s): {System.Environment.NewLine}
{jsErrors.Aggregate("", (s, entry) => s + entry.Message)} {System.Environment.NewLine}");
}
}

Next

In the following article from the series, we will discuss how to extend the solution to execute Lighthouse checks when we run tests using Selenium Grid. For this purpose, I will teach you how to create Selenium Grid Hub and Node servlets (plugins).

Online Training

  • C#

  • JAVA

  • NON-FUNCTIONAL

START:8 November 2021

Web Test Automation Fundamentals

LEVEL: 1

  • Java Level 1
  • Java Unit Testing Fundamentals
  • Source Control Introduction
  • Selenium WebDriver- Getting Started
  • Setup Continuous Integration Job
Duration: 20 hours

4 hour per day

-50% coupon code:

START:24 November 2021

Test Automation Advanced

LEVEL: 2

  • Java Level 2
  • WebDriver Level 2
  • Appium Level 1
  • WinAppDriver Level 1
  • WebDriver in Docker and Cloud
  • Test Reporting Solutions and Frameworks
  • Behavior-Driven Development
Duration: 30 hours

4 hour per day

-20% coupon code:

START:8 December 2021

Enterprise Test Automation Framework

LEVEL: 3 (Master Class)

After discussing the core characteristics, we will start writing the core feature piece by piece.
We will continuously elaborate on why we design the code the way it is and look into different designs and compare them. You will have exercises to finish a particular part or extend it further along with discussing design patterns and best practices in programming.

Duration: 30 hours

4 hour per day

-20% coupon code:

START: 13 September 2021

Web Test Automation Fundamentals

LEVEL: 1

  • C# Level 1
  • C# Unit Testing Fundamentals
  • Source Control Introduction
  • Selenium WebDriver- Getting Started
  • Setup Continuous Integration Job
Duration: 20 hours

4 hour per day

-50% coupon code:

START:29 September 2021

Test Automation Advanced

LEVEL: 2

  • C# Level 2
  • WebDriver Level 2
  • Appium Level 1
  • WinAppDriver Level 1
  • WebDriver in Docker and Cloud
  • Test Reporting Solutions and Frameworks
  • Behavior-Driven Development- SpecFlow
Duration: 30 hours

4 hour per day

-20% coupon code:

START:13 October 2021

Enterprise Test Automation Framework

LEVEL: 3 (Master Class)

After discussing the core characteristics, we will start writing the core feature piece by piece.
We will continuously elaborate on why we design the code the way it is and look into different designs and compare them. You will have exercises to finish a particular part or extend it further along with discussing design patterns and best practices in programming.

Duration: 30 hours

4 hour per day

-20% coupon code:

START: 13 September 2021

Performance Testing

  • Fundamentals of Performance Testing
  • Fundamentals of network technologie
  • Performance testing with WebPageTest
  • Performance test execution and automation
  • Introduction to Jmeter
  • Introduction to performance monitoring and tuning
  • Performance testing in the cloud
Duration: 24 hours

4 hour per day

-30% coupon code:

The post UI Performance Analysis via Selenium WebDriver appeared first on Automate The Planet.

UI Performance Analysis via Selenium WebDriver