Developer Zone

Advanced Software Development with MATLAB

You’ve Got Mail!

Posted by Andy Campbell,

1 views (last 30 days)  |   Likes   |   10 comments
Today I wanted to share some fun I've been having with customizing the emails sent from my Jenkins builds. This was inspired by Narendra's comment on a previous post. So, getting to it, I took a quick look after reading the comment and found the Email-ext plugin that enables customization of the emails that are sent to administrators and code committers (offenders?) for failing or unstable builds. Out of the box, this plugin sends an email that lists the failing tests that were encountered. While this is nice, it doesn't pass muster in my book. Why?
  1. It is very much catered to JUnit test results by default. These aren't JUnit results #forgoodnesssakes! We are talking MATLAB code here! I want the CI system configuration to be production grade just like the code it is integrating, which means we don't accept calling MATLAB test results JUnit test results!
  2. Showing the failed tests is nice but I also want to see the diagnostics. If we have a chance to know immediately what the problem is then I don't want to have to click through to have to figure this out. Having the diagnostics in the email might mean I know exactly why the build fails when I get the email on my phone while riding the T on my way home. The remaining ride then has the benefit of valuable brain cycles already figuring out the solution. If I have to click through? No dice.
Make it MATLAB OK, my strategy here is to pull the basic format from the default html email template because the customization API involves writing jelly scripts and I bet not many people have extensive experience writing in jelly (I certainly haven't). You can see the default jelly script for an html email is listed here, so for a first cut lets take a simplified version of that and make it our own. If you analyze the default html template you can see the jelly script has this basic format: 
<j:jelly xmlns:j="jelly:core" xmlns:st="jelly:stapler" xmlns:d="jelly:define">

<STYLE>
<!-- Define CSS Styling -->
</STYLE>

<BODY>
<!-- Produce html content for email -->
</BODY>

</j:jelly>
Within the main script body there is a node for styling and a node for the main body. For now lets just keep the styling that matches the default template, with one notable difference in that we define another class to account for filtered tests. 
  <STYLE>
    BODY, TABLE, TD, TH, P {
    font-family:Verdana,Helvetica,sans serif;
    font-size:11px;
    color:black;
    }
    h1 { color:black; }
    h2 { color:black; }
    h3 { color:black; }
    TD.bg1 { color:white; background-color:#0000C0; font-size:120% }
    TD.bg2 { color:white; background-color:#4040FF; font-size:110% }
    TD.bg3 { color:white; background-color:#8080FF; }
    TD.test_passed { color:blue; }
    TD.test_failed { color:red; }
    TD.test_filtered { color:orange; }
    TD.output { font-family:Courier New; }
  </STYLE>
For the body content why don't we prune out what we aren't working with and just keep what we are interested in at the moment. So I'll keep the general job information, the changeset information from your SCM system, and the artifacts: 
<BODY>
<j:set var="spc" value="&amp;nbsp;&amp;nbsp;" />


<!-- GENERAL INFO -->

<TABLE>
  <TR><TD align="right">
    <j:choose>
      <j:when test="${build.result=='SUCCESS'}">
        <IMG SRC="${rooturl}static/e59dfe28/images/32x32/blue.gif" />
      </j:when>
	  <j:when test="${build.result=='FAILURE'}">
        <IMG SRC="${rooturl}static/e59dfe28/images/32x32/red.gif" />
      </j:when>
      <j:otherwise>
        <IMG SRC="${rooturl}static/e59dfe28/images/32x32/yellow.gif" />
      </j:otherwise>
    </j:choose>
  </TD><TD valign="center"><B style="font-size: 200%;">BUILD ${build.result}</B></TD></TR>
  <TR><TD>Build URL</TD><TD><A href="${rooturl}${build.url}">${rooturl}${build.url}</A></TD></TR>
  <TR><TD>Project:</TD><TD>${project.name}</TD></TR>
  <TR><TD>Date of build:</TD><TD>${it.timestampString}</TD></TR>
  <TR><TD>Build duration:</TD><TD>${build.durationString}</TD></TR>
</TABLE>
<BR/>


<!-- CHANGE SET -->

<j:set var="changeSet" value="${build.changeSet}" />
<j:if test="${changeSet!=null}">
  <j:set var="hadChanges" value="false" />
  <TABLE width="100%">
    <TR><TD class="bg1" colspan="2"><B>CHANGES</B></TD></TR>
    <j:forEach var="cs" items="${changeSet}" varStatus="loop">
      <j:set var="hadChanges" value="true" />
      <j:set var="aUser" value="${cs.hudsonUser}"/>
      <TR>
        <TD colspan="2" class="bg2">${spc}Revision <B>${cs.commitId?:cs.revision?:cs.changeNumber}</B> by
          <B>${aUser!=null?aUser.displayName:cs.author.displayName}: </B>
          <B>(${cs.msgAnnotated})</B>
         </TD>
      </TR>
      <j:forEach var="p" items="${cs.affectedFiles}">
        <TR>
          <TD width="10%">${spc}${p.editType.name}</TD>
          <TD>${p.path}</TD>
        </TR>
      </j:forEach>
    </j:forEach>
    <j:if test="${!hadChanges}">
      <TR><TD colspan="2">No Changes</TD></TR>
    </j:if>
  </TABLE>
<BR/>
</j:if>


<!-- ARTIFACTS -->

<j:set var="artifacts" value="${build.artifacts}" />
<j:if test="${artifacts!=null and artifacts.size()&gt;0}">
  <TABLE width="100%">
    <TR><TD class="bg1"><B>BUILD ARTIFACTS</B></TD></TR>
    <TR>
      <TD>
        <j:forEach var="f" items="${artifacts}">
      	  <li>
      	    <a href="${rooturl}${build.url}artifact/${f}">${f}</a>
      	  </li>
      	</j:forEach>
      </TD>
    </TR>
  </TABLE>
<BR/>  
</j:if>
Still with me? OK good this next part gets interesting because I want to see the test results in a similar way to how MATLAB presents them to me as an array of TestResults. So rather than looping over all of the packages, we are going to print the total number of passed/failed/filtered tests and then create a section for failed tests and a section for filtered tests. 
<!-- Test reporting TEMPLATE (references to JUnit refer to the JUnit style XML Output produced by the MATLAB TestRunner) --> 
<j:set var="resultList" value="${it.JUnitTestResult}" />
<j:if test="${resultList.isEmpty()!=true}">
  <TABLE width="100%">
    <j:forEach var="testResult" items="${resultList}">
      <j:set var="failCount" value="${failCount + testResult.getFailCount()}"/>
      <j:set var="skipCount" value="${skipCount + testResult.getSkipCount()}"/>
      <j:set var="passCount" value="${passCount + testResult.getPassCount()}"/>
    </j:forEach>
    <TR><TD class="bg1" colspan="2"><B>MATLAB Test Suite Summary:</B> ${passCount+failCount+skipCount} Ran, ${failCount} Failed, ${passCount} Passed, ${skipCount} Filtered</TD></TR>

    <TR><TD class="bg1" colspan="2"><B>Failed Tests</B></TD></TR>
    <j:forEach var="testResult" items="${resultList}">
      <j:forEach var="packageResult" items="${testResult.getChildren()}">
        <j:forEach var="failed_test" items="${packageResult.getFailedTests()}">
          <TR bgcolor="white"><TD class="test_failed" colspan="2"><B><li>Failed: ${failed_test.getFullName()} </li></B></TD></TR>
        </j:forEach>
      </j:forEach> 
    </j:forEach> 

    <TR><TD class="bg1" colspan="2"><B>Filtered Tests</B></TD></TR>
    <j:forEach var="testResult" items="${resultList}">
      <j:forEach var="packageResult" items="${testResult.getChildren()}">
        <j:forEach var="filtered_test" items="${packageResult.getSkippedTests()}">
          <TR bgcolor="white"><TD class="test_filtered" colspan="2"><B><li>Filtered: ${filtered_test.getFullName()} </li></B></TD></TR>
        </j:forEach>  
      </j:forEach>
    </j:forEach>
  </TABLE>	
  <BR/>
</j:if>
Note that what we have done here is analyzed information in the jenkins TestResult instances that contain the information included in the JUnit style xml produced by MATLAB's XMLPLugin (remember?) With this information we are looping over these results three times, once to count the totals, once to print the failed tests, and once to print the filtered tests. Now we can show the log tail for posterity and close out the BODY and jelly tags. 
<!-- LOG OUTPUT -->

<TABLE width="100%" cellpadding="0" cellspacing="0">
<TR><TD class="bg1"><B>Log Tail (Full Log Attached)</B></TD></TR>
<j:forEach var="line" items="${build.getLog(100)}"><TR><TD class="console">${line}</TD></TR></j:forEach>
</TABLE>
<BR/>

</BODY>
</j:jelly>
Does it work? Well not yet. First we need to save this jelly script in a location that allows us to configure Jenkins to use this script when creating our emails. To do this, you need to save the script into the $JENKINS_HOME/email-templates/ folder in order to be picked up by the plugin. If you are not the jenkins administrator this may take some cajoling of said administrator. Once that cajoling is successful however, you can configure the plugin to use this script on a project by project basis. For me this looks like this: With this script applied, I now get this email from Jenkins: Woot woot! That email speaks MATLAB! Next step - include some diagnostics. What's the diagnosis, doc? This is actually quite easy with a simple modification to the failures portion of the jelly script. We just need to add another table row as preformatted text along with a call to getErrorStackTrace, which is where MATLAB's diagnostics are placed. 
<TR><TD class="bg1" colspan="2"><B>Failed Tests</B></TD></TR>
<j:forEach var="testResult" items="${resultList}">
  <j:forEach var="packageResult" items="${testResult.getChildren()}">
    <j:forEach var="failed_test" items="${packageResult.getFailedTests()}">
      <TR bgcolor="white"><TD class="test_failed" colspan="2"><B><li>Failed: ${failed_test.getFullName()} </li></B></TD></TR>
      <TR><TD class="console"><PRE>${failed_test.getErrorStackTrace()}</PRE></TD></TR>
    </j:forEach>
  </j:forEach> 
</j:forEach>
With this simple change Jenkins now includes the diagnostic information in the email as well. The Best of Both Worlds OK, I need to do one more final tweak (I promise). While I am fan of including the diagnostics in the email, it almost seems like it is too much information at once. What I would really like is the first output we produced that could show the diagnostics one at a time for each test failure I look at. Can it be done? Well kinda sorta. Remember we are dealing with email content within email clients, not full fledged browsers rendering dynamic html and JavaScript. This would typically be easy with JavaScript, but unfortunately that's unsupported for email content. We need some form of dynamic content that reacts to mouse clicks in static html content. Mouse clicks, or mouse hovers . Alright, the cat is out of the bag. This can be done for the subset of email clients that support CSS hover selectors! For others, you can still send the full email content you just won't be able to see the dynamic behavior. My client happens to support the hover CSS selector so it works great for me. Here's what we need to do. First update the style section of the document with a few more style classes and the hover behavior which sets the font size of the diagnostic to zero percent when we are not hovering over the test name and sets the size to 100% when we are. This looks like the following 
<STYLE>
  BODY, TABLE, TD, TH, P {
  font-family:Verdana,Helvetica,sans serif;
  font-size:11px;
  color:black;
  }
  h1 { color:black; }
  h2 { color:black; }
  h3 { color:black; }
  TD.bg1 { color:white; background-color:#0000C0; font-size:120% }
  TD.bg2 { color:white; background-color:#4040FF; font-size:110% }
  TD.bg3 { color:white; background-color:#8080FF; }
  TD.test_passed { color:blue; }
  TD.test_failed { color:red; }
  TD.test_filtered { color:orange; }
  TD.output { font-family:Courier New;font-size:0%;}
  TD.build_log { font-family:Courier New;}
  TR.failed_row:hover {background-color:lightgray;} 
  TR.failed_row:hover + TR.output_row TD.output{font-size:100%;} 
</STYLE>
Here we add a few classes for table rows, and mark the test result row as light gray when we hover over it. At the same time, we also change the styling of the output row immediately following the row being hovered over to be 100%. Trickery! This also requires a few changes to the test results portion of the jelly script as well, mainly just to opt into these different style classes. 
<TR><TD class="bg1" colspan="2"><B>Failed Tests</B></TD></TR>
<j:forEach var="testResult" items="${resultList}">
  <j:forEach var="packageResult" items="${testResult.getChildren()}">
    <j:forEach var="failed_test" items="${packageResult.getFailedTests()}">
      <TR class="failed_row" bgcolor="white"><TD class="test_failed" colspan="2"><B><li>Failed: ${failed_test.getFullName()} </li></B></TD></TR>
      <TR class="output_row"><TD class="output"><PRE>${failed_test.getErrorStackTrace()}</PRE></TD></TR>
    </j:forEach>
  </j:forEach> 
</j:forEach>
How'd we do? Well at least in my email client, we have a nice concise email that gives me my full diagnostic information on a per test basis when I hover over a test of interest. Here is the final jelly script. Please take this and make it better and let me know how you fare in the comments: 
<j:jelly xmlns:j="jelly:core" xmlns:st="jelly:stapler" xmlns:d="jelly:define">
  
  <STYLE>
    BODY, TABLE, TD, TH, P {
    font-family:Verdana,Helvetica,sans serif;
    font-size:11px;
    color:black;
    }
    h1 { color:black; }
    h2 { color:black; }
    h3 { color:black; }
    TD.bg1 { color:white; background-color:#0000C0; font-size:120% }
    TD.bg2 { color:white; background-color:#4040FF; font-size:110% }
    TD.bg3 { color:white; background-color:#8080FF; }
    TD.test_passed { color:blue; }
    TD.test_failed { color:red; }
    TD.test_filtered { color:orange; }
    TD.output { font-family:Courier New;font-size:0%;}        
    TD.build_log { font-family:Courier New;}        
    TR.failed_row:hover {background-color:lightgray;}           
    TR.failed_row:hover + TR.output_row TD.output{font-size:100%;}                
  </STYLE>
  <BODY>
    <j:set var="spc" value="&amp;nbsp;&amp;nbsp;" />
    
    
    <!-- GENERAL INFO -->
    
    <TABLE>
      <TR><TD align="right">
          <j:choose>
            <j:when test="${build.result=='SUCCESS'}">
              <IMG SRC="${rooturl}static/e59dfe28/images/32x32/blue.gif" />
            </j:when>
            <j:when test="${build.result=='FAILURE'}">
              <IMG SRC="${rooturl}static/e59dfe28/images/32x32/red.gif" />
            </j:when>
            <j:otherwise>
              <IMG SRC="${rooturl}static/e59dfe28/images/32x32/yellow.gif" />
            </j:otherwise>
          </j:choose>
      </TD><TD valign="center"><B style="font-size: 200%;">BUILD ${build.result}</B></TD></TR>
      <TR><TD>Build URL</TD><TD><A href="${rooturl}${build.url}">${rooturl}${build.url}</A></TD></TR>
      <TR><TD>Project:</TD><TD>${project.name}</TD></TR>
      <TR><TD>Date of build:</TD><TD>${it.timestampString}</TD></TR>
      <TR><TD>Build duration:</TD><TD>${build.durationString}</TD></TR>
    </TABLE>
    <BR/>
    
    
    <!-- CHANGE SET -->
    
    <j:set var="changeSet" value="${build.changeSet}" />
    <j:if test="${changeSet!=null}">
      <j:set var="hadChanges" value="false" />
      <TABLE width="100%">
        <TR><TD class="bg1" colspan="2"><B>Changes</B></TD></TR>
        <j:forEach var="cs" items="${changeSet}" varStatus="loop">
          <j:set var="hadChanges" value="true" />
          <j:set var="aUser" value="${cs.hudsonUser}"/>
          <TR>
            <TD colspan="2" class="bg2">${spc}Revision <B>${cs.commitId?:cs.revision?:cs.changeNumber}</B> by
              <B>${aUser!=null?aUser.displayName:cs.author.displayName}: </B>
              <B>(${cs.msgAnnotated})</B>
            </TD>
          </TR>
          <j:forEach var="p" items="${cs.affectedFiles}">
            <TR>
              <TD width="10%">${spc}${p.editType.name}</TD>
              <TD>${p.path}</TD>
            </TR>
          </j:forEach>
        </j:forEach>
        <j:if test="${!hadChanges}">
          <TR><TD colspan="2">No Changes</TD></TR>
        </j:if>
      </TABLE>
      <BR/>
    </j:if>
    
    
    <!-- ARTIFACTS -->
    
    <j:set var="artifacts" value="${build.artifacts}" />
    <j:if test="${artifacts!=null and artifacts.size()&gt;0}">
      <TABLE width="100%">
        <TR><TD class="bg1"><B>Build Artifacts</B></TD></TR>
        <TR>
          <TD>
            <j:forEach var="f" items="${artifacts}">
              <li>
                <a href="${rooturl}${build.url}artifact/${f}">${f}</a>
              </li>
            </j:forEach>
          </TD>
        </TR>
      </TABLE>
      <BR/>  
    </j:if>
    
    <!-- Test reporting TEMPLATE (references to JUnit refer to the JUnit style XML Output produced by the MATLAB TestRunner) -->     
    <j:set var="resultList" value="${it.JUnitTestResult}" />
    <j:if test="${resultList.isEmpty()!=true}">
      <TABLE width="100%">
        <j:forEach var="testResult" items="${resultList}">
          <j:set var="failCount" value="${failCount + testResult.getFailCount()}"/>
          <j:set var="skipCount" value="${skipCount + testResult.getSkipCount()}"/>
          <j:set var="passCount" value="${passCount + testResult.getPassCount()}"/>
        </j:forEach>
        <TR><TD class="bg1" colspan="2"><B>MATLAB Test Suite Summary:</B> ${passCount+failCount+skipCount} Ran, ${failCount} Failed, ${passCount} Passed, ${skipCount} Filtered</TD></TR>
        
        <TR><TD class="bg1" colspan="2"><B>Failed Tests</B></TD></TR>
        <j:forEach var="testResult" items="${resultList}">
          <j:forEach var="packageResult" items="${testResult.getChildren()}">
            <j:forEach var="failed_test" items="${packageResult.getFailedTests()}">
              <TR class="failed_row" bgcolor="white"><TD class="test_failed" colspan="2"><B><li>Failed: ${failed_test.getFullName()} </li></B></TD></TR>
              <TR class="output_row"><TD class="output"><PRE>${failed_test.getErrorStackTrace()}</PRE></TD></TR>
            </j:forEach>
          </j:forEach> 
        </j:forEach>
        
        <TR><TD class="bg1" colspan="2"><B>Filtered Tests</B></TD></TR>
        <j:forEach var="testResult" items="${resultList}">
          <j:forEach var="packageResult" items="${testResult.getChildren()}">
            <j:forEach var="filtered_test" items="${packageResult.getSkippedTests()}">
              <TR bgcolor="white"><TD class="test_filtered" colspan="2"><B><li>Filtered: ${filtered_test.getFullName()} </li></B></TD></TR>
            </j:forEach>  
          </j:forEach>
        </j:forEach>
      </TABLE>	
      <BR/>
    </j:if>

    <!-- CONSOLE OUTPUT -->
      <TABLE width="100%" cellpadding="0" cellspacing="0">
        <TR><TD class="bg1"><B>Build Log Tail (Full Log Attached)</B></TD></TR>
        <j:forEach var="line" items="${build.getLog(100)}"><TR><TD class="build_log">${line}</TD></TR></j:forEach>
      </TABLE>
      <BR/>
    
  </BODY>
</j:jelly>

Published with MATLAB® R2015b

|
Category: 
Continuous Integration
  • print

See Also

Comments

To leave a comment, please click here to sign in to your MathWorks Account or create a new one.