AWS Device Farm is an Amazing offering from AWS, that enables validation of mobile and web apps, without ever having to have the devices onsite. This article outlines an use case where we use browser testing to validate tracking pixels on the digital ad campaigns that we run.

The best thing about Device Farm is that you can use the tools you already use in your workflows. Here we use a selenium chrome test to navigate to the landing page. Selenium allows access to the logs generated in chrome dev tools. We will extract the network log to validate pixel firing.

Below shows a high level diagram that runs the job. The Kubernetes CronJob schedules a pod to run the job which initiates the set, provision the TestGrid that executes the browser test. The pod extracts the logs, does the analysis and generates reports and notification as necessary.

The first task is to create a TestGrid in device farm and use the url to create a selenium RemoteWebDriver.

import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.logging.LogType;
import org.openqa.selenium.logging.LoggingPreferences;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.WebDriver;

import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.devicefarm.DeviceFarmClient;
import software.amazon.awssdk.services.devicefarm.model.CreateTestGridUrlRequest;
import software.amazon.awssdk.services.devicefarm.model.CreateTestGridUrlResponse;

//---------
//functions
//---------
WebDriver createDriver()  {
    return new RemoteWebDriver(testGridUrl(), getOptions());
}
 
private URL testGridUrl() {
    final String DEVICE_FARM_PROJECT_ARN = "...";
    
    DeviceFarmClient client  = DeviceFarmClient.builder().region(Region.US_WEST_2).build();
    CreateTestGridUrlRequest request = CreateTestGridUrlRequest.builder()
            .expiresInSeconds(300)
            .projectArn(DEVICE_FARM_PROJECT_ARN)
            .build();
    CreateTestGridUrlResponse response = client.createTestGridUrl(request);
    
    return new URL(response.url());
}
 
private ChromeOptions getOptions() {
    LoggingPreferences loggingPreferences = new LoggingPreferences();
    loggingPreferences.enable(LogType.PERFORMANCE, Level.ALL);

    DesiredCapabilities desiredCapabilities = DesiredCapabilities.chrome();
    desiredCapabilities.setCapability(CapabilityType.LOGGING_PREFS, loggingPreferences);

    ChromeOptions options = new ChromeOptions();
    options.merge(desiredCapabilities);
    options.setCapability( "goog:loggingPrefs", loggingPreferences);
    
    return options;
}

Once we have a RemoteWebDriver, the execution is no different from a normal Selenium test grid.


public List<JsonNode> getLandingPageLogs(String landingPageUrl) {
    WebDriver driver = createDriver();
    try {
        driver.get(landingPageUrl);
        return driver.manage().logs().get(LogType.PERFORMANCE).getAll() 
                .stream() //stream of LogEntry
                .map(this::parseLogEntry)
                .collect(Collectors.toList());
    } finally {
        driver.quit();
    }
}

private JsonNode parseLogEntry(LogEntry logEntry) {
    try {
        //Jackson ObjectMapper used to parse json string
        return mapper.readTree(logEntry.getMessage()); 
    } catch (JsonProcessingException e) {
        throw new RuntimeException("Error parsing log message", e);
    }
}

Now that we have the logs, we can look for the network request and response of the pixel url and its http status to verify that the tracker url is firing.

boolean verifyPixelFire(String landingPageUrl, String pixelUrl) {
    List<JsonNode> logs = getLandingPageLogs(landingPageUrl);
    
    //find requestIds matching the pixel URL
    Set<String> reqIds = logs.stream()
        .filter(n -> {
            String method = n.get("message").get("method").asText();
            return "Network.requestWillBeSent".equals(method);
        })
        .map(n -> n.get("message").get("params"))
        .filter(n -> {
            String url = n.get("request").get("url").asText();
            return pp.getPixelUrl().equalsIgnoreCase(url);
        })
        .map(n -> n.get("requestId").asText())
        .collect(Collectors.toSet());
        
    //Now verify that the requests succeeded in responseReceived log entries
    for (String reqId : reqIds) {
        boolean ok = logs.stream()
            .filter(n -> {
                String method = n.get("message").get("method").asText();
                return "Network.responseReceived".equals(method);
            })
            .map(n -> n.get("message").get("params"))
            .anyMatch(n -> {
                String respReqId = n.get("requestId").asText();
                int status = n.get("response").get("status").asInt();
                return reqId.equalsIgnoreCase(respReqId) && status == 200;
            });
        if (!ok) {
            return false;
        }
     }
     return true;
}

We use the approach above to verify the pixel URLs on a periodic basis. The specific Campaign is updated with the status, as well a consolidated report is generated.