This is that last post of a 4 part series on Coded UI Tests. We will wrap up our example with how to wire up our Coded UI Tests to CruiseControl.NET (our CI Server). Here's where we've come:
Part 1: How to Create a Coded UI Test
Part 2: Code Organization: Test Lists, Test Categories, Oh my! – Coded UI Test Organization
Part 3: Running Coded UI Tests From Command Line
This blog will explain how to run Coded UI Tests from CruiseControl.NET.
Overview Steps
- Create a batch file to call the GUI Tests
- CruiseControl.NET Configuration
- Install “MSTest Reports” package
- Modify the dashboard.config file
- Modify the ccnet.config file and add a project
- Modify the MSTestReport2008.xsl file
- Run the project in CCNET
- Tips and Tricks
Create a Batch File to Call the Coded UI Tests
First, create a batch file that will run your Coded UI Tests. I've blogged about how you can do that here <LINK>. Note that you if you choose to run multiple tests from one command line prompt, the order in which they are ran is not guaranteed. I've worked around this by calling 3 commands, in order, with 3 different result files. If you don't save them to different files, the second command will error out because the file already exists. Since we have 3 different result files, we'll use CC.NET to merge them all into one report. Here's our sample bat file:
rd /S /Q "C:\build\reports"
mkdir "C:\build\reports"
cd C:\Build\Working\
"C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe" /testmetadata:"C:\Build\Working\CalcCodedUITest.vsmdi" /testlist:List_LoadCalc /resultsfile:"C:\Build\Reports\List_LoadCalcResults.trx"
"C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe" /testmetadata:"C:\Build\Working\CalcCodedUITest.vsmdi" /testlist:PerfCalcResults /resultsfile:"C:\Build\Reports\PerfCalcResults.trx"
"C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe" /testmetadata:"C:\Build\Working\CalcCodedUITest.vsmdi" /testlist:ShutDownCalcResults /resultsfile:"C:\Build\Reports\ShutDownCalcResults.trx"
The interesting things in this code is we delete the folder the reports are created in (C:\buid\reports) since they are merged in CCNet and we change the directory to the directory of the solution (C:\build\working) and then run 3 seperate test lists.
CruiseControl.NET Configuration
Install the MSTest Reports Package
-Assuming you have CC.NET installed, go into CCNet's Administer Dashboard and install the "MSTest Reports" package. This package will enable you to view the test results through CC.NET.
Modify the dashboard.config File
In order to view the results, you need to tweak the dashboard.config file (default location: C:\Program Files\CruiseControl.NET\webdashboard) first and change the xsl file from MsTestSummary.xsl to MsTestSummary2008.xsl in the <buildPlugins> tag so it looks like the following:
<buildPlugins>
<buildReportBuildPlugin>
<xslFileNames>
<xslFile>xsl\header.xsl</xslFile>
<xslFile>xsl\modifications.xsl</xslFile>
<xslFile>xsl\MsTestSummary2008.xsl</xslFile>
</xslFileNames>
</buildReportBuildPlugin>
<buildLogBuildPlugin />
<xslReportBuildPlugin description="MSTest Report" actionName="MSTestBuildReport" xslFileName="xsl\MsTestReport2008.xsl"></xslReportBuildPlugin>
</buildPlugins>
Do an IISReset. IISReset is necessary for CC.NET to pick up the changes.
Modify the CCNet.config file
Next, you'll need to add the project to the ccnet.config file (default location: C:\Program Files\CruiseControl.NET\server\ccnet.config) that will execute an "exec" task that will call your batch file to run the Coded UI tests. More examples can be found using CruiseControl.NET documentation
First, add the queue name like so:
<queue name="alpha" />
Then add the project node with a name, etc, like so:
<project queue="alpha" queuePriority="1">
<name>Coded UI Command Line</name>
<tasks>
<exec executable="C:\CodedUIFiles\CalcCodedUITests.bat" />
</tasks>
<publishers>
<merge>
<files>
<file>c:\Build\Reports\LoadCalcResults.trx</file>
<file>c:\Build\Reports\PerformCalcResults.trx</file>
<file>c:\Build\Reports\CloseCalcResults.trx</file>
</files>
</merge>
</publishers>
</project>
The interesting piece is that we are merging 3 ".trx" files. We needed to ensure the order in which the test lists are ran and to ensure the order, in the bat file, we have 3 command lines executing tests, one after the other, to different results files. If we had set it to the same results file, the system will error out stating that the file already exists. These file names correspond to the file names that are created in the bat file when the tests run and get deleted before every run. Since the files are merged into the CCNet report file, there's no need to keep them twice.
Modify the MSTestReport2008.xsl File
A side effect to running 3 command line tests is in the reporting, it will only show the same number of tests that are in the last group of tests ran. In order to sum up all the tests ran, we need to modify the MsTestReport2008.xsl (C:\Program Files\CruiseControl.NET\webdashboard\xsl\MSTestReport2008.xsl). Modify the file and add the sum commands like so:
<xsl:variable name="pass_count" select="sum(/cruisecontrol/build/*[local-name()='TestRun']/*[local-name()='ResultSummary']/*[local-name()='Counters']/@passed)"/>
<xsl:variable name="inconclusive_count" select="sum(/cruisecontrol/build/*[local-name()='TestRun']/*[local-name()='ResultSummary']/*[local-name()='Counters']/@inconclusive)"/>
<xsl:variable name="failed_count" select="sum(/cruisecontrol/build/*[local-name()='TestRun']/*[local-name()='ResultSummary']/*[local-name()='Counters']/@failed)"/>
<xsl:variable name="total_count" select="sum(/cruisecontrol/build/*[local-name()='TestRun']/*[local-name()='ResultSummary']/*[local-name()='Counters']/@total)"/>
Save the file and do an IISReset for CCNet to pick up the changes.
Run the Project in CCNET
Go to CCNet and click on "Force" to force the Coded UI tests to run.
Once the tests are finished, click on the name of the project -> "Latest Build" on the left hand side -> "MSTest Report"
Tips and Tricks
Here are some tips and tricks in running Coded UI tests from a WinForms application:
- Due to security issues, CCNet cannot run as a service and run Coded UI Tests. Turn the service off and run the CCNet application as an Administrator.
- The Coded UI Tests can be kicked off from any machine that can access the CCNet web page. However, the computer's desktop that CCNet is installed on will need to be maximized and unlocked (you can operate this from an RDC session) in order to run tests. If the CCNet computer's desktop is locked, minimized or set to a different screen resolution than when i twas recorded, the tests will fail and throw a "can't interact with desktop" error.
- Sometimes you can get a false positive and CCNet will say the last build status will say "Success". This will not be necessarily the case for all tests. After the Coded UI tests are ran, make sure you view the MSTest Report to view which tests passed or failed.
- Ideally, set up CCNET to pull down the code from your source control, compile it, then run the Coded UI tests from the bat file.
- Structure your bat file to run test lists, not individual tests. The advantage to this is that when new tests are added to the list, you won't need to modify the bat file, it will just be included. Organizing in test lists also allow for horizontal scalability. For example, say you have a large application that runs on multiple back ends, you could setup another CI Server on another VM, setup a new bat file just with a specific back end of tests, and let that VM run tests at the same time.
- Coded UI Tests are slow by nature so over time you'll want to set up a schedule trigger in CCNet to kick this off at a certain time, for example, as an over night process. I would also recommend a dedicated computer or VM to perform these tests.
Hope this has helped down your path of automated Coded UI tests.
