{"id":60092,"date":"2020-10-30T14:00:14","date_gmt":"2020-10-30T22:00:14","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/devops\/?p=60092"},"modified":"2020-10-30T08:49:14","modified_gmt":"2020-10-30T16:49:14","slug":"things-to-consider-when-running-visual-tests-in-ci-cd-pipelines-azure-devops-github-actions-part-2","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devops\/things-to-consider-when-running-visual-tests-in-ci-cd-pipelines-azure-devops-github-actions-part-2\/","title":{"rendered":"Things to consider when running visual tests in CI\/CD pipelines: Azure Devops &#038; GitHub Actions (Part 2)"},"content":{"rendered":"<p>If you haven&#8217;t yet read <a href=\"https:\/\/devblogs.microsoft.com\/devops\/things-to-consider-when-running-visual-tests-in-ci-cd-pipelines-getting-started-part-1?WT.mc_id=devops-9717-jessde\">the first post<\/a> in this series, I recommend checking it out now. Do it. I&#8217;ll wait, I probably need some more java in my body, anyway. Get it, java == coffee?<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/media.giphy.com\/media\/sbwjM9VRh0mLm\/giphy.gif\" alt=\"eye roll\" \/><\/p>\n<p>Great, now that you have your build environment and code ready to run visual tests, and I have my coffee (<em>whispers<\/em> java), let&#8217;s walk through things you should note when it comes to configuring your pipeline.<\/p>\n<p>If you&#8217;re still new to DevOps, let&#8217;s do a quick recap on what DevOps is.<\/p>\n<p>&#8220;DevOps is the union of people, process, and products to enable continuous delivery of value to our end users.&#8221; &#8211; Donovan Brown<\/p>\n<p>Azure Pipelines and GitHub Actions are both platforms you can use to deliver said value. For those who are unfamiliar with Azure Pipelines, it&#8217;s a service available through Azure DevOps, and for those who are not familiar with GitHub Actions, it allows you to automate your workflow without ever leaving GitHub. Both options are more traditional in the sense you run your tasks on a hosted (provided for you) or private (you own, pay for and maintain) server; we will refer to this method as the term: <code>pipeline<\/code>.<\/p>\n<p>There are 3 basic tasks we need to add into our new or existing pipelines:<\/p>\n<ul>\n<li>HTTP Check<\/li>\n<li>Capture Build Environment Variables<\/li>\n<li>Run Visual Tests<\/li>\n<\/ul>\n<p>These tasks will be added to our dev\/test\/qa environment stage and will run <em>after<\/em> your dev\/test\/qa deployment succeeds.<\/p>\n<p>I&#8217;m a huge fan of YAML, particularly because I can just run bash commands as a native task. Below is an example of the YAML I would add to my existing pipeline:<\/p>\n<h3>Azure DevOps Example<\/h3>\n<p>In order to run the below code, you will need to install the <a href=\"https:\/\/marketplace.visualstudio.com\/items?itemName=applitools.applitools-integration?WT.mc_id=devops-9717-jessde\">Applitools Extension<\/a> to your <a href=\"https:\/\/docs.microsoft.com\/azure\/devops\/marketplace\/install-extension?view=azure-devops&amp;tabs=browser&amp;?WT.mc_id=devops-9717-jessde\">Azure DevOps organization<\/a>.<\/p>\n<p>You will notice there are 3 tasks; each task performs the task we outlined above.<\/p>\n<pre><code>- bash: |\n    # Run check to see when $(hostname) is available\n    attempt_counter=0\n    max_attempts=5\n    until $(curl --output \/dev\/null --silent --head --fail https:\/\/\"$(hostname)\"\/); do\n        if [ ${attempt_counter} -eq ${max_attempts} ];then\n          echo \"Max attempts reached\"\n          exit 1\n        fi\n        printf '.'\n        attempt_counter=$(($attempt_counter+1))\n        sleep 20\n    done\n  displayName: HTTP Check\n\n- task: ApplitoolsBuildTask@0\n\n- task: Maven@3\n  inputs:\n    mavenPomFile: 'visual_tests\/pom.xml'\n    goals: 'test'\n    options: 'clean test'\n    publishJUnitResults: true\n    testResultsFiles: '**\/surefire-reports\/TEST-*.xml'\n    javaHomeOption: 'JDKVersion'\n    mavenVersionOption: 'Default'\n    mavenAuthenticateFeed: false\n    effectivePomSkip: false\n    sonarQubeRunAnalysis: false\n  env:\n    APPLITOOLS_API_KEY: $(APPLITOOLS_API_KEY)\n    RUNWHERE: pipeline\n    TEST_START_PAGE: https:\/\/$(hostname)\n  displayName: \"Visual Testing\"\n<\/code><\/pre>\n<p>Notice at the very bottom, in the last task, I have 3 required environment variables set:<\/p>\n<ul>\n<li>APPLITOOLS_API_KEY<\/li>\n<li>RUNWHERE<\/li>\n<li>TEST_START_PAGE<\/li>\n<\/ul>\n<p>In this kind of environment (Azure DevOps, GitHub Actions, or virutally any build server that is not container based), the <code>RUNWHERE<\/code> environment variable will be set to <code>pipeline<\/code>, which is the reference term we reviewed earlier in this post.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/10\/Screen-Shot-2020-03-11-at-2.12.48-PM-1.png\"><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/10\/Screen-Shot-2020-03-11-at-2.12.48-PM-1.png\" alt=\"Image Screen Shot 2020 03 11 at 2 12 48 PM\" width=\"969\" height=\"857\" class=\"alignnone size-full wp-image-60097\" srcset=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/10\/Screen-Shot-2020-03-11-at-2.12.48-PM-1.png 969w, https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/10\/Screen-Shot-2020-03-11-at-2.12.48-PM-1-300x265.png 300w, https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/10\/Screen-Shot-2020-03-11-at-2.12.48-PM-1-768x679.png 768w\" sizes=\"(max-width: 969px) 100vw, 969px\" \/><\/a><\/p>\n<h3>GitHub Actions Example<\/h3>\n<p>The below code is a little different than the Azure DevOps snippet above, particularly because there isn&#8217;t an Applitools Extension <em>yet<\/em>. In the meantime, we can capture the necessary environment variables the same way we usually would in bash, and then Actions allows us to use <a href=\"https:\/\/help.github.com\/en\/actions\/reference\/development-tools-for-github-actions#set-an-environment-variable-set-env?WT.mc_id=devops-9717-jessde\">::set-env<\/a> to persist them to future steps.<\/p>\n<p>You also might notice there is an <em>extra<\/em> step for java &#8211; this is just so GitHub Actions uses the proper version of java for your project; in my case and my code, I require Java 13. The below snippet is an actual action, not just bash code, and it even creates a settings.xml file for us with our GitHub information securely passed.<\/p>\n<pre class=\"prettyprint\">- name: http check\n   run: |\n    attempt_counter=0\n    max_attempts=5\n    until $(curl --output \/dev\/null --silent --head --fail https:\/\/$hostname\/); do\n        if [ ${attempt_counter} -eq ${max_attempts} ];then\n        echo \"Max attempts reached\"\n        exit 1\n        fi\n        printf '.'\n        attempt_counter=$(($attempt_counter+1))\n        sleep 20\n    done\n    \n- name: Applitools Build Task\n  run: |\n    # set batch ID\n    APPLITOOLS_BATCH_ID=${{ github.run_id }}\n    echo \"::set-env name=APPLITOOLS_BATCH_ID::$APPLITOOLS_BATCH_ID\"\n    echo \"Setting environment variable APPLITOOLS_BATCH_ID: $APPLITOOLS_BATCH_ID\"\n    \n    # set batch name\n    APPLITOOLS_BATCH_NAME=\"GitHub Actions \/ ${{ github.workflow }}\"\n    echo \"::set-env name=APPLITOOLS_BATCH_NAME::$APPLITOOLS_BATCH_NAME\"\n    echo \"Setting environment variable APPLITOOLS_BATCH_NAME: $APPLITOOLS_BATCH_NAME\"\n    \n    # set server url\n    APPLITOOLS_SERVER_URL=\"https:\/\/eyes.applitools.com\"\n    echo \"::set-env name=APPLITOOLS_SERVER_URL::$APPLITOOLS_SERVER_URL\"\n    echo \"Setting environment variable APPLITOOLS_SERVER_URL: $APPLITOOLS_SERVER_URL\"\n    \n    # set batch sequence\n    APPLITOOLS_BATCH_SEQUENCE=\"GitHub Actions \/ ${{ github.workflow }}\"\n    echo \"::set-env name=APPLITOOLS_BATCH_SEQUENCE::$APPLITOOLS_BATCH_SEQUENCE\"\n    echo \"Setting environment variable APPLITOOLS_BATCH_SEQUENCE: $APPLITOOLS_BATCH_SEQUENCE\"\n    \n    # set test start page\n    TEST_START_PAGE=https:\/\/$hostname\n    echo \"::set-env name=TEST_START_PAGE::$TEST_START_PAGE\"\n    echo \"Setting environment variable TEST_START_PAGE_TEST: $TEST_START_PAGE\"\n\n- name: \"Set JDK version\"\n  uses: actions\/setup-java@v1\n  with:\n    java-version: '13'\n    architecture: x64\n    jdkFile: \/usr\/lib\/jvm\/java-13-oracle\n\n- name: \"Visual Testing\"\n  env:\n      APPLITOOLS_API_KEY: ${{ secrets.APPLITOOLS_API_KEY }}\n      RUNWHERE: pipeline\n  run: mvn -f visual_tests\/pom.xml clean test<\/pre>\n<p>Notice again, at the very bottom in the last task, I still have my required environment variables set; only I opted to export the <code>TEST_START_PAGE<\/code> as part of my Applitools bash test. One of the environment variables (<code>APPLITOOLS_API_KEY<\/code>) is a secret encrypted variable; to learn how to create GitHub Actions secrets, <a href=\"https:\/\/help.github.com\/en\/actions\/configuring-and-managing-workflows\/creating-and-storing-encrypted-secrets?WT.mc_id=devops-9717-jessde\">click here<\/a>.<\/p>\n<p>Once you have your tasks added and environment variables set, you&#8217;re ready to run your pipeline!<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/10\/Screen-Shot-2020-03-11-at-2.21.02-PM.png\"><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/10\/Screen-Shot-2020-03-11-at-2.21.02-PM.png\" alt=\"Image Screen Shot 2020 03 11 at 2 21 02 PM\" width=\"1259\" height=\"826\" class=\"alignnone size-full wp-image-60098\" srcset=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/10\/Screen-Shot-2020-03-11-at-2.21.02-PM.png 1259w, https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/10\/Screen-Shot-2020-03-11-at-2.21.02-PM-300x197.png 300w, https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/10\/Screen-Shot-2020-03-11-at-2.21.02-PM-1024x672.png 1024w, https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2020\/10\/Screen-Shot-2020-03-11-at-2.21.02-PM-768x504.png 768w\" sizes=\"(max-width: 1259px) 100vw, 1259px\" \/><\/a><\/p>\n<p>Stay tuned for the final post in this series where we walk through how to accomplish the same benefits of visual testing in a container based pipeline!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Join us for the second post in this blog series, &#8220;Things to consider when running visual tests in CI\/CD pipelines&#8221;, where we focus on the pipeline based CI\/CD method!<\/p>\n","protected":false},"author":43194,"featured_media":60103,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[226,252],"tags":[],"class_list":["post-60092","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ci","category-testing"],"acf":[],"blog_post_summary":"<p>Join us for the second post in this blog series, &#8220;Things to consider when running visual tests in CI\/CD pipelines&#8221;, where we focus on the pipeline based CI\/CD method!<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/60092","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/43194"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=60092"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/60092\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/60103"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=60092"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=60092"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=60092"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}