Chris Elleman Technical Stuff

16 May/10 Off

Using NewsGroups/Usenet

Update 21/05/2010 - it appears that Newzbin has gone offline due to legal action, a suggested alternative on the linked blog was NZBmatrix.

Update 28/05/2010 - Newzbin is back online, hurray!.

I often get asked about the best way to obtain files shared across the Internet, most people use Bit-Torrent, but to be honest its too distributed for my liking, and the sites keep getting shutdown. When I worked for the ISP, we used to use just FTP, but then we had a server where all of the content was regularly updated.

Nowadays, the best way (IMHO) is to use the News-Groups (or Usenet as it was originally known), to use these you will need the following:

  • A usenet provider, who allows access to the binary newsgroups, a good one is Usenetserver - they do charge ($15/month, cheaper if you bulk-buy), but they allow you to max out your broadband easily.
  • A Usenet indexing site, there are several free and paid for, my preference was for Newzbin but they have been fighting a legal battle, so a good free one is Nzbindex
  • A Usenet client (supports NZB files) which downloads the files, error checks and then decompresses them all in one go, the best tool for this is Ninan as it comes in Windows and Linux versions.

Setup: the basic set-up involves creating your accounts on both Usenetserver, and the indexing site (if using Newzbin), you then enter these details into Ninan's config. You then start-up the Ninan server which provides you with a web interface, so that you can search for files and paste the Post ID into Ninan, then it will download the files for you.

How it works: the way the system works is quite straightforward, the source file(s) are zipped up and broken down into a number of compressed files (using WinRar/.rar or WinZip/.zip), a set of parity (PAR2) files are then generated, which can be used later for error correction; these files are then individually uploaded into the usenet and a post id is assigned to each one, the uploader/poster records these post ids in an xml file called an NZB file. This NZB file is then uploaded to an indexing site, which another person/downloader can then come along and download the NZB, then use their Usenet client (Ninan) to download all of the relevant parts and re-constuct the original file. Simple.

If any of the files gets corrupted during either the upload or the download, the PAR2 files can be used to re-construct the original file - Ninan includes this capability and checks automatically.

Disclaimer: you should only use this guide/software for lawful purposes.

16 May/10 Off

Efergy e2: Monitoring your homes energy usage

My house has a problem, we spend an awful lot on electricity; currently we are paying £97/month, and I have no idea where its going. Its true we have a lot of computers in the house, along with the new LED TV we bought last Christmas, but still, we can't be using that much.To help, I have invested in a device to help me track usage over time, the Efergy e2, £43 from thegreenstop, I've had an energy monitor before which has just sat on the mantelpiece showing us the current energy usage, when we are looking at it; but this device is different, it stores the readings in its memory and then allows you to download them to your PC so that you can see how much energy you are using, and how much you are spending. Efergy e2 e-link software

For each hour of the day, you can see what the average usage is, and it allows you to view historical information as well, genius! I've already found that when I first installed it, even when we were sleeping the house was using 400watts of electricity, costing around £0.05p/hour, doesn't sound much, but that cost alone is £438/year. So I've been round, unplugged some kit we weren't using (old PVR, XBox etc), last night the lowest reading was 185watts, over half - so I think we are well on the way.

16 May/10 Off

Apache Tomcat 5.5 Stale Connections to Oracle DB

The Problem: the media company was having a problem with its Tomcat configuration, whereby, DB connections were going stale. The systems involved were pretty standard, an Oracle 10g Database and Tomcat DBCP all running on Solaris 10 x86. The problem was that Tomcat was spinning up a set of datasource pools per web context which meant that there were a lot of idle connections which were never being used and so never re-cylced, leading to stale connections on the database; also, with the maxwait being set to never expire, this also contributed to the stale connections

The current datasource configuration: an example (context.xml):

<Resource name="jdbc/ECE_UPDATE_DS"
 auth="Container"
 type="javax.sql.DataSource"
 driverClassName="oracle.jdbc.OracleDriver"
 url="jdbc:oracle:thin:@(DESCRIPTION = (LOAD_BALANCE = OFF) (ADDRESS = (PROTOCOL=TCP)(HOST=dtodbgz80)(PORT=1521)) (ADDRESS = (PROTOCOL=TCP)(HOST=dtodbgz80)(PORT=1521)) (CONNECT_DATA = (SERVICE_NAME = TS11)))"
 connectionProperties="{ValidateConnection=true,ConnectionWaitTimeout=10}"
 username="uat"
 password="uat"
 validationQuery="select 1 from dual"
 testOnBorrow="true"
 poolPreparedStatements="true"
 maxOpenPreparedStatements="10"
 initialSize="2"
 minIdle="2"
 maxActive="5"
 maxIdle="5"
 maxWait="-1"
 removeAbandonedTimeout="60"
 removeAbandoned="true"
 logAbandoned="true"/>

The Solution: this was to change the datasource configuration, so that one a single set were spun up per tomcat server, and then each application context would get a reference so that they could use to the "master" pool; changing the maxwait to a resonable value of 5 mins (30000 milliseconds) also helpd to remove the stale connections and let the DBCP pool manage itself more effecitivly:

The new datasource configuration:

server.xml:

<Resource name="jdbc/ECE_UPDATE_DS"
 auth="Container" type="javax.sql.DataSource"
 driverClassName="oracle.jdbc.OracleDriver"
 url="jdbc:oracle:thin:@(DESCRIPTION = (LOAD_BALANCE = OFF) (ADDRESS = (PROTOCOL=TCP)(HOST=dtodbgz80)(PORT=1521)) (ADDRESS = (PROTOCOL=TCP)(HOST=dtodbgz80)(PORT=1521)) (CONNECT_DATA = (SERVICE_NAME = TS11)))"
 connectionProperties="{ValidateConnection=true,ConnectionWaitTimeout=10}"
 username="uat"
 password="uat"
 validationQuery="select 1 from dual"
 testOnBorrow="true"
 poolPreparedStatements="true"
 maxOpenPreparedStatements="10"
 initialSize="2"
 minIdle="2"
 maxActive="5"
 maxIdle="5"
 maxWait="30000"
 removeAbandonedTimeout="60"
 removeAbandoned="true"
 logAbandoned="true"/>

 context.xml:

<ResourceLink name="jdbc/ECE_UPDATE_DS"
    global="jdbc/ECE_UPDATE_DS"
    type="javax.sql.DataSource"/>

 

References: http://tomcat.apache.org/tomcat-5.5-doc/jndi-datasource-examples-howto.html

15 May/10 Off

Filtering transformed XML when displaying HTML

The problem: at the Media Company my team has a number of systems which developers/testers/etc require accounts on; when a new person starts we have to manually provision accounts on each system which is a bit of a pain, especially when you are working on lots of projects and can get distracted. I wanted to put together a webpage which automatically queried each system and then consolidated a view of users accounts on the various systems, so that almost in real-time you could see their accounts being provisioned.

The Systems: we are using the following systems, with the APIs used to query them in brackets:

  • Anthill Pro - automatated Build/Release tool (Oracle DB Query)
  • Atlassin Jira - Bug tracking tools (MySQL DB Query)
  • VersionOne - Sprint planning/tracking tool (Rest API)
  • SubVersion - Code respository (.htpasswd Query)
  • TestRun - test script tracking tool (MySQL DB Query)

How it was built: rather than have a dynamic webpage query each system in real-time, I thought it would be better to gather the information from each system and produce a master XML file which has the detail for all of the systems, this also means that any user load is absorted by the web-server. I choose to use a perl script to make various requests using the APIs of the different systems to produce the resultant XML, an example of which is below:

<?xml version="1.0" encoding="UTF-8"?>
<accounts>
 <user username="celleman">
 <AD_dn>CN=Elleman\, Christopher,OU=Users,OU=Technology,OU=Departments,DC=....</AD_dn>
 <AD_email>Christopher.Elleman@...</AD_email>
 <AD_name>Elleman, Christopher</AD_name>
 <AD_sAMAccountName>CElleman</AD_sAMAccountName>
 <AD_title>System Engineer</AD_title>
 <ahpro_role>System Admin</ahpro_role>
 <ahpro_user_id>412</ahpro_user_id>
 <email>christopher.elleman@...</email>
 <jira_account>celleman</jira_account>
 <jira_user_id>10010</jira_user_id>
 <svn_roles>solengg-admins build-admins</svn_roles>
 <testrun_admin>0</testrun_admin>
 <testrun_userid>1</testrun_userid>
 <username>celleman</username>
 <v1_member_id>0</v1_member_id>
 <v1_role>System Admin</v1_role>
 <v1_username>..\celleman</v1_username>
 </user>
 <summary>
  <elapsedSeconds>3</elapsedSeconds>
  <generated>Sat May 15 22:10:03 BST 2010</generated>
 </summary>
</accounts>

The Transformation: The XSL is shown below, this converts the XML into HTML:

<?xml version="1.0"?>
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="html" />

<xsl:template match="/">
 <table>
 <tr bgcolor="#66CCFF">
  <th>Username</th>
  <th>Name</th>
  <th>Title</th>
  <th>Email</th>
  <th>Anthill Pro</th>
  <th width="50">Jira</th>
  <th width="50">SVN</th>
  <th>TestRun</th>
  <th>VersionOne</th>
 </tr>
 <xsl:for-each select="accounts/user">
 <tr>
  <td><xsl:value-of select="@username"/></td>
  <td><xsl:value-of select="AD_name"/></td>
  <td><xsl:value-of select="AD_title"/></td>
  <td><a href="mailto:{email}"><xsl:value-of select="email"/></a></td>
  <xsl:choose>
   <xsl:when test="ahpro_role">
    <td><xsl:value-of select="ahpro_role"/></td>
   </xsl:when>
   <xsl:otherwise>
    <td style="background-color: red"></td>
   </xsl:otherwise>
  </xsl:choose>
  <xsl:choose>
   <xsl:when test="jira_user_id">
    <td><xsl:value-of select="jira_user_id"/></td>
   </xsl:when>
   <xsl:otherwise>
    <td style="background-color: red"></td>
   </xsl:otherwise>
  </xsl:choose>
  <xsl:choose>
   <xsl:when test="svn_roles">
    <td></td>
   </xsl:when>
   <xsl:otherwise>
    <td style="background-color: red"></td>
   </xsl:otherwise>
  </xsl:choose>
  <xsl:choose>
   <xsl:when test="testrun_admin">
    <td></td>
   </xsl:when>
   <xsl:otherwise>
    <td style="background-color: red"></td>
   </xsl:otherwise>
  </xsl:choose>
  <xsl:choose>
   <xsl:when test="v1_role">
    <td><xsl:value-of select="v1_role"/></td>
   </xsl:when>
   <xsl:otherwise>
    <td style="background-color: red"></td>
   </xsl:otherwise>
  </xsl:choose>
 </tr>
 </xsl:for-each>
 </table>
 <p>Generated: <xsl:value-of select="accounts/summary/generated"/></p>
</xsl:template>
</xsl:stylesheet>

The HTML + Javascript: in order to combine the XML and XSL at runtime, and also import the CSS, the html including the inline JavaScript is below, the example URLs are:

http://accounts.html

and, with filtering:

http://accounts.html?username=celleman


<html>
<head>
<link rel="stylesheet" href="style.css" type="text/css"/>
<title>Sol Eng Tools Accounts</title>
<meta http-equiv="Refresh" content="1800">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="Author" content="Chris Elleman" />
<script type="text/javascript" src="http://www.google.com/jsapi"></script>
<script>
google.load("jquery", "1.3.1");
function getURLParam( name )
{
  name = name.replace(/[\[]/,"\\\[").replace(/[\]]/,"\\\]");
  var regexS = "[\\?&]"+name+"=([^&#]*)";
  var regex = new RegExp( regexS );
  var results = regex.exec( window.location.href );
  if( results == null )
    return "";
  else
    return results[1];
}
function loadXMLDoc(fname)
{
  var xmlDoc;
  // code for IE
  if (window.ActiveXObject)
  {
    xmlDoc=new ActiveXObject("Microsoft.XMLDOM");
  }
  // code for Mozilla, Firefox, Opera, etc.
  else if (document.implementation
  && document.implementation.createDocument)
  {
    xmlDoc=document.implementation.createDocument("","",null);
   }
  else
  {
    alert('Your browser cannot handle this script');
  }
xmlDoc.async=false;
xmlDoc.load(fname);
return(xmlDoc);
}
function displayResult()
{
xml=loadXMLDoc("accounts.xml");
xsl=loadXMLDoc("accounts.xsl");
// code for IE
if (window.ActiveXObject)
  {
    ex=xml.transformNode(xsl);
    document.getElementById("example").innerHTML=ex;
  }
  // code for Mozilla, Firefox, Opera, etc.
  else if (document.implementation
  && document.implementation.createDocument)
  {
    xsltProcessor=new XSLTProcessor();
    xsltProcessor.importStylesheet(xsl);
    resultDocument = xsltProcessor.transformToFragment(xml,document);
    document.getElementById("example").appendChild(resultDocument);
  }
  ShowSelectedUser();
}
function ShowSelectedUser() {
 var users = getURLParam("username");
 if(users.length < 1) {
        return;
 }
 // Hide all rows
 var debug=document.getElementById("debug");
 $(".accounts_main_table").find("tr").hide().eq(0).show();
 // Un-Hide rows in the username variable
 var usersArray = users.split(",");
 for ( var i=0; i < usersArray.length; ++i ){
        var currentUser = usersArray[i];
   debug.innerHTML += "user: "+currentUser+"<br/>\n";
        $(".accounts_main_table").find("tr").each(function(index){
   if($(this).find("td").eq(0).text()==currentUser){
    $(this).show();
   }
 });
 }
}
</script>
</head>
<body onLoad="displayResult()">
<div id="debug" style="display: none;">
<b>DEBUG:</b><br/>
</div>
<div id="example">
</div>
</body>
</html>

Ok, so the un-hide part of the javascript is not the most efficient, but hey, it works!

14 May/10 Off

Hello World!

Isn't that how everything starts...welcome to my website, this is to replace my old website with something relevant to what I'm doing now...See  About Me…Briefly for more information.