{"id":60163,"date":"2020-11-13T16:55:29","date_gmt":"2020-11-14T00:55:29","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/devops\/?p=60163"},"modified":"2020-11-13T16:55:29","modified_gmt":"2020-11-14T00:55:29","slug":"things-to-consider-when-running-visual-tests-in-ci-cd-pipelines-container-pipeline-edition-part-3","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devops\/things-to-consider-when-running-visual-tests-in-ci-cd-pipelines-container-pipeline-edition-part-3\/","title":{"rendered":"Things to consider when running visual tests in CI\/CD pipelines: Container Pipeline Edition (Part 3)"},"content":{"rendered":"<p>If you haven&#8217;t <em>at least<\/em> read <a href=\"https:\/\/devblogs.microsoft.com\/devops\/things-to-consider-when-running-visual-tests-in-ci-cd-pipelines-getting-started-part-1\/?WT.mc_id=devops-9718-jessde\">the first post in this series<\/a>, I recommend checking it out now. Do it. I&#8217;ll wait. You also might want <a href=\"https:\/\/devblogs.microsoft.com\/devops\/things-to-consider-when-running-visual-tests-in-ci-cd-pipelines-azure-devops-github-actions-part-2\/?WT.mc_id=devops-9718-jessde\">to read the second post in this series<\/a> if you have any investment or interest in traditional pipelines (non-container), but it&#8217;s not a requirement to follow along here today.<\/p>\n<p>This is the final post in the &#8220;Things to consider when running visual tests in CI\/CD pipelines&#8221; series. I think as the age old saying goes, &#8220;They saved the best for last!&#8221; Haha, kidding. I&#8217;m only slightly bias because if you have been following me for any length of time, you may know I am a <em>huge<\/em> fan of all things containers, including container based pipelines.<\/p>\n<p>If you&#8217;ve used Jenkins for awhile now, you might know you can run <a href=\"https:\/\/jenkins.io\/doc\/book\/pipeline\/docker\/\">Jenkins<\/a> within your Kubernetes cluster and take advantage of the Docker socket to run container based pipelines.<\/p>\n<p>What is a container based pipeline? In short, it&#8217;s a pipeline where <em>each<\/em> task runs in a container. The benefit of this is I don&#8217;t need to spend time configuring my build server or build environment with <em>all<\/em> the necessary dependencies and binaries needed for my pipeline. I can simply run a container with those deps\/binaries and then execute my tasks accordingly.<\/p>\n<p>In the <a href=\"https:\/\/go.applitools.com\/200312-Test-Automation-CI-CD-webinar.html\" rel=\"noopener noreferrer\" target=\"_blank\">webinar<\/a> Angie and I did showcasing best practices with visual testing and CI\/CD, I used <a href=\"https:\/\/codefresh.io\/jessica\">Codefresh<\/a> to demonstrate how to run visual tests in a container based pipeline. Codefresh is a Kubernetes native CI\/CD platform, specifically built for microservices and container-based applications. It was also the first CI\/CD platform that puts the container image at the center of your workflow. Another recent container based pipeline solution tool, and one you can see in action at <a href=\"https:\/\/cloud-days.jfrog.com\/microsoft-azure\/\">JFrog&#8217;s DevOps Cloud Days with Azure<\/a> on November 18th, is JFrog Pipelines. As a side note, I do feel Codefresh and JFrog both make it easier to work with container based pipelines than Jenkins (<em>shudders at writing groovy<\/em>), but that&#8217;s a neither here nor there.<\/p>\n<p>Let&#8217;s review the considerations you should be aware of when it comes to visual tests and a container-based (aka docker-based) pipeline.<\/p>\n<ul>\n<li>The obvious: you do not have Chrome locally and easily accessible, unless of course you create a container with Chrome and tools of choice you plan to use.<\/li>\n<\/ul>\n<p>Since my demo uses a Maven Springboot project, I really wanted to use the smallest image possible (smaller images == faster performance and smaller attack surfaces). My demo uses <code>maven:3.6.3-jdk-13<\/code> to run the tests Angie wrote; I run them just as I would if I were in a &#8220;traditional&#8221; environment: simply export the required environment variables and run <code>mvn -f visual_tests\/pom.xml clean test<\/code>.<\/p>\n<ul>\n<li>\n<p>Since I don&#8217;t have chrome local, I can spin up a docker compose file with selenium hub and all necessary browser nodes. Selenium has great documentation on how to do that <a href=\"https:\/\/github.com\/SeleniumHQ\/docker-selenium\/wiki\/Getting-Started-with-Docker-Compose\">here<\/a>.<\/p>\n<\/li>\n<li>\n<p>When using Selenium Hub, especially in a container format, as <a href=\"https:\/\/github.com\/SeleniumHQ\/docker-selenium\/wiki\/Getting-Started-with-Docker-Compose#step-4-running-tests\">the documentation referenced in point 2<\/a> indicates, you have to use a <code>RemoteWebDriver<\/code> to connect to your browser as opposed to the more standard ChromeDriver used in our <code>local<\/code> and <code>pipeline<\/code> environments. Now, when using the <code>RemoteWebDriver<\/code> call as suggested by Selenium&#8217;s docs (using DesiredCapabilities), you might get an error similar to the following:<\/p>\n<\/li>\n<\/ul>\n<pre class=\"prettyprint\">Message: org.openqa.selenium.remote.DesiredCapabilities chrome\n\nINFO: Using new ChromeOptions() is preferred to DesiredCapabilities.chrome() Starting ChromeDriver 2.44.609538 (b655c5a60b0b544917107a59d4153d4bf78e1b90) on port 33954 Only local connections are allowed.<\/pre>\n<p>To resolve that error\/info message, I simply consolidated their example code to the following (which does not use DesiredCapabilities):<\/p>\n<pre class=\"prettyprint\">chromeOptions = new ChromeOptions();\nString Selenium = \"http:\/\/selenium_hub:4444\/wd\/hub\";\ndriver = new RemoteWebDriver(new URL(Selenium), chromeOptions);<\/pre>\n<p>If you read the first post, you can find how I used the above in the getEnvironment() class under the <code>else if (runWhere.equals(\"container\"))<\/code> statement.<\/p>\n<p>Now that we understand the <em>why<\/em> we wrote our <code>container<\/code> condition in our getEnvironment() class, let&#8217;s put it to use and have some fun with our pipeline steps!<\/p>\n<p>If you have watched our webinar, or read part 2 in this series, you know there are 3 basic tasks we need to add into our new or existing pipelines:<\/p>\n<ol>\n<li>HTTP Check<\/li>\n<li>Capture Build Environment Variables<\/li>\n<li>Run Visual Tests<\/li>\n<\/ol>\n<p>However, since we know we also need access to selenium hub for the 3rd task, we have to first define our services. Codefresh makes this super simple since it allows us to define <a href=\"https:\/\/codefresh.io\/docs\/docs\/codefresh-yaml\/service-containers\/\">service containers<\/a> in a format based on docker compose.<\/p>\n<p>If you don&#8217;t have a Codefresh account, and want to play around with it, you can get started for free <a href=\"https:\/\/codefresh.io\/jessica\">here<\/a>. <strong>Disclaimer:<\/strong> This isn&#8217;t an affiliate link and I&#8217;m not paid to promote Codefresh, I just really love all things containers and DevOps, and want to help other engineers get started.<\/p>\n<p>You can take a look at my codefresh.yaml used for this demo <a href=\"https:\/\/github.com\/jldeen\/spring-boot-websocket-chat-demo\/blob\/applitools\/codefresh.yml\">here<\/a>, but for now let&#8217;s just start with our services section. At the very top of our yaml, we&#8217;ll use the following:<\/p>\n<pre class=\"prettyprint\">version: \"1.0\"\nservices:\n  name: selenium_hub\n  composition:\n    selenium_hub:\n      image: selenium\/hub:latest\n      ports:\n        - 4444\n      environment:\n        - SE_OPTS=-debug\n        - GRID_MAX_SESSION=5\n\n    chrome_node:\n      image: selenium\/node-chrome:latest\n      ports:\n        - 5900\n        - 5555\n      command: bash -c \"sleep 5 &#38;&#38; \/opt\/bin\/entry_point.sh\"\n      depends_on: \n        - selenium_hub\n      environment:\n        - HUB_HOST=selenium_hub\n        - REMOTE_HOST=http:\/\/chrome_node:5555\n        - NODE_MAX_SESSION=5\n        - NODE_MAX_INSTANCES=5\n\n    firefox_node:\n      image: selenium\/node-firefox:latest\n      user: 1000:1000\n      ports:\n        - 5900\n        - 5555\n      command: bash -c \"sleep 5 &#38;&#38; \/opt\/bin\/entry_point.sh\"\n      depends_on: \n        - selenium_hub\n      environment:\n        - HUB_HOST=selenium_hub\n        - REMOTE_HOST=http:\/\/firefox_node:5555\n        - NODE_MAX_SESSION=5\n        - NODE_MAX_INSTANCES=5<\/pre>\n<p>See how similar it looks to a docker-compose.yaml file? Sweet, huh? Once we define our service containers, we simply link the necessary step to the <code>selenium_hub<\/code> service we just defined.<\/p>\n<p>First, however, let&#8217;s do our http check and then define our Applitools environment variables.<\/p>\n<pre class=\"prettyprint\">http_check:\n  image: jldeen\/docker-jfrog-cli-java:1.0.4\n  title: \"Http Check\"\n  stage: \"deploy dev\"\n  shell: bash\n  commands:\n    - export attempt_counter=0\n    - export export max_attempts=5\n    - until $(curl --output \/dev\/null --silent --head --fail https:\/\/${{INGRESS_HOSTNAME_DEV}}); do if [ ${attempt_counter} -eq ${max_attempts} ];then echo \"Max attempts reached\"; exit 1; fi; printf '.';         attempt_counter=$(($attempt_counter+1)); sleep 20; done\n\napplitools_task:\n  image: tutum\/curl\n  title: \"Applitools Build Task\"\n  stage: \"deploy dev\"\n  commands:\n    - export APPLITOOLS_BATCH_ID=CF-${{CF_SHORT_REVISION}}\n    - echo \"Setting environment variable APPLITOOLS_BATCH_ID - $APPLITOOLS_BATCH_ID\"\n    - cf_export APPLITOOLS_BATCH_ID=$APPLITOOLS_BATCH_ID\n    - APPLITOOLS_BATCH_NAME=\"Codefresh \/ ${{CF_SHORT_REVISION}}\"\n    - echo \"Setting environment variable APPLITOOLS_BATCH_NAME - $APPLITOOLS_BATCH_NAME\"\n    - cf_export APPLITOOLS_BATCH_NAME=$APPLITOOLS_BATCH_NAME\n    - export APPLITOOLS_SERVER_URL=\"https:\/\/eyes.applitools.com\"\n    - echo \"Setting environment variable APPLITOOLS_SERVER_URL - $APPLITOOLS_SERVER_URL\"\n    - cf_export APPLITOOLS_SERVER_URL=$APPLITOOLS_SERVER_URL\n    - export APPLITOOLS_BATCH_SEQUENCE=CF-${{CF_SHORT_REVISION}}\n    - echo \"Setting environment variable APPLITOOLS_BATCH_SEQUENCE - $APPLITOOLS_BATCH_SEQUENCE\"\n    - cf_export APPLITOOLS_BATCH_SEQUENCE=$APPLITOOLS_BATCH_SEQUENCE<\/pre>\n<p>Notice on how both above tasks are using specific images: <code>jldeen\/docker-jfrog-cli-java:1.0.4<\/code> and <code>tutum\/curl<\/code>.<\/p>\n<p>Both images have only the tools we need for our tasks. I tend to use an image I own and control for a majority of my tasks for security reasons in this specific pipeline, but that&#8217;s a post for another time.<\/p>\n<p>You might also notice the second task has both <code>export<\/code> and <code>cf_export<\/code>. The first <code>export<\/code> only exports the variable for that task, while the <code>cf_export<\/code> will persist the environment variable for subsequent tasks.<\/p>\n<p>The first 2 tasks are self explanatory as far as their function, so now let&#8217;s show the code for our final and most important task: visual tests using maven.<\/p>\n<pre class=\"prettyprint\">visual_testing:\n  image: maven:3.6.3-jdk-13\n  title: \"Running Visual Tests\"\n  stage: \"deploy dev\"   \n  commands:\n    - export RUNWHERE=container\n    - export TEST_START_PAGE=https:\/\/${{INGRESS_HOSTNAME_DEV}}\n    - 'mvn -Dmaven.repo.local=\/codefresh\/volume\/m2_repository -f visual_tests\/pom.xml clean test'\n  services:\n    - selenium_hub<\/pre>\n<p>Notice the the first export command: <code>export RUNWHERE=container<\/code> as well as the last 2 lines:<\/p>\n<pre class=\"prettyprint\">services:\n    - selenium_hub<\/pre>\n<p>That&#8217;s where we link our <code>visual_testing<\/code> step to the services we created at the very top of our pipeline. That&#8217;s it.<\/p>\n<p>Now, when we run our pipeline and get to our visual tests task, the maven will be able to run our tests using the code we defined in our getEnvironment() class under the condition for <code>container<\/code>.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/11\/Screen-Shot-2020-03-11-at-2.13.44-PM.png\"><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/11\/Screen-Shot-2020-03-11-at-2.13.44-PM.png\" alt=\"Image Screen Shot 2020 03 11 at 2 13 44 PM\" width=\"1175\" height=\"376\" class=\"alignnone size-full wp-image-60164\" srcset=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/11\/Screen-Shot-2020-03-11-at-2.13.44-PM.png 1175w, https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/11\/Screen-Shot-2020-03-11-at-2.13.44-PM-300x96.png 300w, https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/11\/Screen-Shot-2020-03-11-at-2.13.44-PM-1024x328.png 1024w, https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/11\/Screen-Shot-2020-03-11-at-2.13.44-PM-768x246.png 768w\" sizes=\"(max-width: 1175px) 100vw, 1175px\" \/><\/a><\/p>\n<p>Now, if you&#8217;re wanting to do this in something like Jenkins or JFrog Pipelines, I will have a future blog post focusing on how to handle container based pipelines in general, but for Jenkins you will have to consider using a <a href=\"https:\/\/plugins.jenkins.io\/docker-compose-build-step\/\">docker compose plugin<\/a> or some other method to spin up services and connect them to running containers.<\/p>\n<p>Remember, when you use docker-compose, you attach defined services to a network so you&#8217;ll have to consider your network configuration and port forwarding.<\/p>\n<p>To give you an idea and some practice, you can play with docker-compose on your system (provided Docker is installed) and <a href=\"https:\/\/github.com\/jldeen\/spring-boot-websocket-chat-demo\">our sample chattybot code<\/a>. You only need two things: &#8211; <a href=\"https:\/\/gist.github.com\/jldeen\/7f2b69b57fdb28d0240d5389bc1d6047\">this docker-compose.yaml<\/a> &#8211; To change line 69 in <code>visual_tests\/src\/test\/java\/base\/BaseTests.java<\/code>.<\/p>\n<pre class=\"prettyprint\"># line 69 for Codefresh\nString Selenium = \"http:\/\/selenium_hub:4444\/wd\/hub\";\n\n# line 69 for local container test\nString Selenium = \"http:\/\/127.0.0.1:4444\/wd\/hub\";<\/pre>\n<p>As a reminder, after making the temporary code change, you will still need to export the <code>RUNWHERE<\/code> environment variable (equal to <code>container<\/code>), along with the <code>batchId<\/code>, <code>TEST_START_PAGE<\/code>, and <code>APPLITOOLS_API_KEY<\/code> variables, and spin up your docker compose service using the following command: <code>docker-compose -f docker-compose.yaml up -d<\/code>.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/11\/docker-compose-selenium.gif\"><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/11\/docker-compose-selenium.gif\" alt=\"Image docker compose selenium\" width=\"1767\" height=\"1080\" class=\"alignnone size-full wp-image-60165\" \/><\/a><\/p>\n<p>Congrats! You just ran visual tests utilizing containers! You&#8217;re cloud native ready now. Time to update that resume. <img decoding=\"async\" src=\"https:\/\/media.giphy.com\/media\/ui1hpJSyBDWlG\/giphy.gif\" alt=\"wink\" \/><\/p>\n","protected":false},"excerpt":{"rendered":"<p>What is a container based pipeline? In short, it&#8217;s a pipeline where *each* task runs in a container. The benefit of this is I don&#8217;t need to spend time configuring my build server or build environment with *all* the necessary dependencies and binaries needed for my pipeline. <\/p>\n","protected":false},"author":43194,"featured_media":60173,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[226,252],"tags":[],"class_list":["post-60163","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ci","category-testing"],"acf":[],"blog_post_summary":"<p>What is a container based pipeline? In short, it&#8217;s a pipeline where *each* task runs in a container. The benefit of this is I don&#8217;t need to spend time configuring my build server or build environment with *all* the necessary dependencies and binaries needed for my pipeline. <\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/60163","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/43194"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=60163"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/60163\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/60173"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=60163"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=60163"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=60163"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}