Executing Automated Test Cases inside Docker

Before starting the discussion, I must say that the reader should have basic understanding about dockers (https://docs.docker.com/) and test automation. Here I have given example with TFS build, but the same can be implemented using Jenkins. Reader should also have the fair understanding on CICD process.

Since last couple of years, there is a big shift on how software is delivered. As per the agile process, the functionality should be delivered as soon as development is completed. There is no time and even don’t have enough resources to do the manual testing, manual deployment. Thanks to CICD (Continuous Integration Continuous Delivery) for automation testing and automated delivery pipeline. The entire pipeline starting with building the project, deploying into test system and running automated test suites avoids the manual efforts.

Test case execution is a very critical phase in delivery pipeline to maintain quality of the product. Consider a product with multiple roles and users, requires substantial amount of time to complete the test case execution. I have seen, a product with Hospital Management System typically takes 7 to 8 hours to complete the entire flow where each department/modules take around 1 hour. A department can be a Front Office, Laboratory, Nursing, Billing etc. Therefore parallel execution is important to reduce the CICD process time.

We cannot execute multiple test suites in the same system at the same time. We need different system so that each events (keyboard/mouse) cannot interfere with each other. If we have 10 modules, then 10 different systems are required to complete the execution in 1 hour. This is quite a big investment towards infrastructure side. Docker saves our life in this scenario.

Executing automated test cases inside docker, provides a number of benefits –

  1. No intervention between test case runs on different modules as they are on different docker containers.
  2. Downloading and creating the images on the fly helps to achieve the optimal use of infrastructure resources. Cloud resources will be used in optimal manner.
  3. Parallel execution reduces the entire CICD timeline.

I have done the experiment with Selenium based automation test suites. We have used Java to write the test scripts and maven project is created for this. We are using simple maven command “maven install” to start the test case execution.

Execution Dependencies

  1. JDK
  2. Maven Library
  3. Maven Repository in case system is not connected to internet
  4. Chrome

We are using TFS 2017 for our CICD implementation. There is an in-built build task for Maven project. We have specified POM file of the project, maven command as “install” and selected the junit test report xml file to upload the test case results.

We will be using Docker to execute test cases. We need to install TFS Agent inside docker. We will be using different agents for different modules. So, if there are total 10 modules, then total 10 TFS Agents are required. All agents have similar configurations and dependencies. To achieve this, we will create a base image with required dependencies to execute test suites.


FROM mcr.microsoft.com/windows/servercore:ltsc2016
 COPY ./Utilities /Utilities
 COPY ./apache-maven-3.6.0 /apache-maven-3.6.0
 WORKDIR /Utilities
 RUN ["powershell","./jdk.cmd"]
 RUN ["powershell","./ChromeInstall.cmd"]
 RUN ["powershell", "./EnvironmentVariables.ps1"]
 WORKDIR /Users/ContainerAdministrator
 RUN md .m2
 WORKDIR /Users/ContainerAdministrator/.m2
 COPY ./repository repository
 WORKDIR /Utilities

We are using Windows 2016 server core as base image. repository folder contains maven libraries required to run the project. This step is optional. Typically maven downloads the required libraries from central maven repository. In case, if we don’t have internet connection, then required libraries need to be kept inside maven local repository folder (default is C:\Users\<username>\.m2).


start "Installing Chrome Silently" /wait ./76.0.3809.100_chrome_installer.exe /silent /install

We will be using headless mode to execute test cases.


pushd %~dp0
start /wait jdk-8u221-windows-x64.exe INSTALLCFG=%~dp0jdk.cfg




[Environment]::SetEnvironmentVariable("JAVA_HOME", "C:\Program Files\Java\jdk1.8.0_221", [EnvironmentVariableTarget]::Machine)
[Environment]::SetEnvironmentVariable("M2_HOME", "C:\apache-maven-3.6.0", [EnvironmentVariableTarget]::Machine)
[Environment]::SetEnvironmentVariable("MAVEN_HOME", "C:\apache-maven-3.6.0", [EnvironmentVariableTarget]::Machine)

$env:Path += ";C:\Program Files\Java\jdk1.8.0_221\bin;C:\apache-maven-3.6.0\bin"

[Environment]::SetEnvironmentVariable("Path", $env:Path, [EnvironmentVariableTarget]::Machine)

The above powershell script is required to set the environment variables for Java and Maven.

The image build with above Dockerfile, can be termed as testagent. We need to create different test agent for different modules. A VSTS Agent is downloaded and kept inside VSTSAgent folder.

Dockerfile for VSTS Agent

FROM x.x.x.x:5000/testagent:latest
 COPY ./VSTSAgent /VSTSAgent
 RUN ["powershell", "./InstallAgent.ps1"] 


./config.cmd --unattended --url http://TFSServer2017:8080/tfs --auth Negotiate --username "xxx" --password "xxx" --pool DockerTestPool1 --agent DockerAgent1

Once the above image is created successfully, then we can see the DockerAgent1 in TFS Agent list inside pool DockerTestPool1 though the Agent is in stopped mode.

To optimize the use of infrastructure, our containers will be created just before the starting of the test case build in CICD pipeline. Once the execution is completed, we are killing the containers to free the resources. These type of architecture is very much useful in cloud environment where we have to pay as long as we are using the resources. To achieve the dynamic creation of containers, we have installed one TFS agent inside Docker Host system and runs the below powershell script using that agent on Docker Host system. The below script is used to start the containers after downloading the images from local docker registry. Once the execution is completed, we need to stop and destroy the containers and images with passing the parameters “Stop” in the below script.



#$AgentName = "dockeragent7#dockeragent8"
#$AgentImageName = "dockervstsagent7#dockervstsagent8"
#$Flag = "Start"

$EachAgentName = $AgentName.Split("#")
$EachAgentImageName = $AgentImageName.Split("#")

Write-Host $EachAgentName
Write-Host $EachAgentImageName

if($Flag -eq "Start")

$i = 0
foreach($agent in $EachAgentName)
    #$Command = "docker service create --name " + $agent + " --mount type=bind,source=C:\DockerVolumes\"  `
    #            + $agent + "\VSO_CICD,destination=C:\VSO_CICD --mount type=bind,source=C:\DockerVolumes\" `
    #            + $agent + "\temp,destination=C:\temp " + $EachAgentImageName[$i] + " run.cmd"    

	$ImageName = "x.x.x.x:5000/" + $EachAgentImageName[$i]	

	$Command = "docker run -d --name " + $agent + " --mount type=bind,source=C:\DockerVolumes\"  `
                + $agent + "\VSO_CICD,destination=C:\VSO_CICD --mount type=bind,source=C:\DockerVolumes\" `
                + $agent + "\temp,destination=C:\temp " + $ImageName + " run.cmd"        

    $PSCheck = docker ps 

    $AgentCheck = "*" + $agent + "*"
    if($PSCheck -like $AgentCheck)
        Write-Host "Killing the container..."
        docker stop $agent
	docker rm $agent
	docker image rm $ImageName
    docker pull $ImageName
    Write-Host $Command
    Invoke-Expression $Command
    $i = $i + 1
    Start-Sleep -s 300
elseif($Flag -eq "Stop")

$i = 0    
foreach($agent in $EachAgentName)
    $PSCheck = docker ps 
    $AgentCheck = "*" + $agent + "*"
    if($PSCheck -like $AgentCheck)
		$ImageName = "x.x.x.x:5000/" + $EachAgentImageName[$i]

        Write-Host "Killing the container..."
        docker stop $agent
        docker rm $agent
	docker image rm $ImageName

	$i = $i + 1

There are multiple build definitions/jobs need to be created to achieve this entire flow.

  1. Manage Docker Agent definition is required to start/stop docker containers.
  2. Different build definitions are required for different modules. Lets say, if there are 10 test suites, then 10 different definitions are required.
  3. One main build definition is required which will build the code and trigger the other definitions.
  4. We are using “Parallel Builds” plug-in to call multiple build definitions – https://marketplace.visualstudio.com/items?itemName=kneradovsky.viko-build-tasks

Hopefully, reader will find this post helpful for there CICD implementation. Please post your thoughts on comments. Happy Coding!

Learn Dockers

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.