Latent bug syndrome

Today I have fallen foul of what I call the "Latent Bug Syndrome". This will hopefully help people understand just how complex software engineering is, and why bugs occur in software despite the best efforts of all involved. Let me explain.

Let's say you have a system like the following.

where System A represents a fairly complex, fully featured, fully tested product that exposes cerrtain functionality to the end users. Now System A can be operating 100% perfectly (or at least for the overwhelming majority of scenarios that are encompassed by the exposed feature set), however there still may be a "Latent Bug" hidden deep within the underlying logic of this system that is never exposed to the end user because the exposed feature set never actually allows the system to get into the state that would cause this bug to exhibit. As a really contrived example, let's take the classic divide by zero problem

 

public stat double DoCalc(double val)

{

return 42 / val;

}

This function operates as expected under all conditions except where val = 0, in which case it throws an exception. If the exposed feature set only alows the user to select numbers from a drop down box that contains the numbers 1, 2, and 3, then every possible test that a tester can do will NEVER produce an error. Keep in mind this is possibly the simplest example one can find, in real life, the scenarios are far more convoluted. The wrong assumption from this is that System A has no bugs.

Now lets suppose that System A is augmented with System B as shown below.

Now System B has the potenial of producing conditions in System A that not only were never tested, but were never even thought of by the original developers and testers of System A, and in my contrived example, a user input into system B may very well cause the value of the val parameter to be passed 0. The thing is that the developers (and testers) of System B are usually different to the developers (and testers) of System A, and often don't have access to the source code, or even if they do, don't have time to follow every code path through the interactions with System A accounting for every possible state that System A could possibly be in to find such a bug.

If the testers of System A are doing a good job, they may actually pick up the bug, then of course the blame game starts. Who is responsible for this bug, who's going to fix it why wasn't this bug picked up before etc....

So this happened to me today. Fortunately the bug never made it to a production system (and no it wasn't the divide by zero bug suggested above, it was infinitely more intricate), but it did make me aware that I had made some assumptions about System A that turned out to be less than 100% accurate.

 

So how can one guard against The Latent Bug Syndrome.

 

System A

• Thorough unit tests of all public (and some times even private) methods exposed by system A will catch a vast number of these bugs.
• Better documentation of publicly exposed methods to help developers integrating with your system understand what assumptions you are making about the input to a method, and what state you are expecting the system to be in.
• Test Driven Development could actually go a long way to irradicating this all together because in TDD you only write code to satisfy the tests nothing less, nothing more, so you don't get the situation where there is code that under ALL conditions tested is never executed. However, I have yet to see a company embrace TDD to that extent as the sheer time involved is usually far too much to justify it commercially.

System B

• Developers and testers of system B need to have a good understanding of how system A works, what it was intended to do, and how they are changing the way in which System A is being called. Understanding this may help you know where problem are likely to occur.
• Know what your assumptions about System A are... and constantly question them.
• Again unit tests are your friend.
• wrapping your interaction with System A is also a good way of being able to respond to changes in System A if they occur.
• Thorough end user testing (there is NO substitute).

VS2005 Project Templates Failing

I came across a problem recently where I went to create a standard Console Application project in Visual Studio 2005 and to my alarm I got a message saying

 

---------------------------
Microsoft Visual Studio
---------------------------
Could not find file 'C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\ProjectTemplatesCache\CSharp\Windows\1033\ConsoleApplication.zip\ConsooleApplication.csproj'

---------------------------
OK
---------------------------

 

This took me back a bit, as creating a Console application is one of the most basic tasks you would ever want to do in Visual Studio 2005. my next thought (after trying exactly the same thing 5 or 6 times) was to run visual studio with elevated priviledges, but alas this still didn't fix the problem. I checked and sure enough I could still create a Windows application no problems (with or without elevating my priviledges), so I just put it down to some kind of vista weirdness and continued on (using Winforms apps instead of console apps whenever I wanted to create a quick test). Today, however, I discovered that I could no longer create Web Service Applications as they crashed with a very similar error, and with a presentation on setting up your development environment on Windows Vista looming, I decided that I had better get to the bottom of the problem.... and so I did.

 

It turned out to be the Guidance Automation Toolkit and Guidance Automation Extensions packages I recently installed that for some reason had done some damage to the Project Template System in Visual Studio. now I guess fair is fair, both of these packages are in beta, and there are some known issues around running them on Vista (unfortunately I was not aware of these issues at install time). One of the big issues with the Guidance Automation Extensions is around uninstalling it. You get so far, and then a dialog pops up, no title, no message, just an OK button, and it stops uninstalling. fortunately I found a blog post by one Greg Duncan on how to uninstall the Guidance Automation Extensions.

 

I can now happily create console and web service applications again, but more importantly stand up in front of the Victoria .Net SIG next tuesday and talk confidently about running Visual Studio 2005 on Vista.

Speaking at Victoria .Net SIG

Just a quick note to say that I'll be speaking at the next Victoria.Net SIG meeting on the 12th of June at Microsoft building, Level 5, 4 Freshwater Place, Southbank. Turn up for Pizza and soft drink at 5:30 pm.

 

The topic of my discussion will be setting up your development environment on Windows Vista. Hope to see you there.

IT a series of stops and starts

Is it just me, or does anyone else in the industry feel that working in the IT industry is a series of stops and starts. What I mean by this is that often when i start on something that I think is going to (or at least should be) faily straight forward, quite often, very early on in the peice I come across a problem that consumes way more time than I'd expected. For example, today, I was asked to fix a bug that was occuring on a project that I was involved with no less than a few weeks ago. I know the codebase had changed significantly, as a merge with the main branch had been done. What I didn't expect was that I'd spend the first 3 and a half hours tryng to get my machine to a point where I could compile and then run the project. Now part of the problem was my grappling with the back up and restore features of SQL Server, and one could argue that I potentially should understand that better (and now do), but the rest of the time was grappling with issues that we have been having with bugs inTFS that cause issues when trying to Get the latest versin of all files in large projects. Then there are aother times like right now where I am trying to do a few things on my PC, and even though I have a fairly new DELL D820 laptop with 2GB of RAM, and 4GB of ready boost, the hard drive is churning continuously (not sure why), and it takes up to 10 seconds to switch between tasks. This may have something to do with the fact that about 20 minutes ago I had to actually reset my computer after it crashed while trying to use Explorer. I am fast approaching "a sad realisation" that although windows Vista may be more secure, it is not in any way speedy, or stable. I really think Microsoft rushed it out the door, and that the end user is paying the price. It's times like these when I start to think about work in other sectors, hmmmm.... maybe I'll start up a Tango dance studio.

Tango performance

Just to show how behind the times I am, here is my first ever appearance on You Tube. This is a Tango performance Niki and I did last weekend at our friend Yeow's birthday party. Niki and I have been dancing Argentinean Tango for 6 years now, and it is becoming an all consuming passion.

 

 

Technorati tags: Tango

It's had its fair chance, time to switch back

That's it I've had it with IE7, I used to use firefox for the vast amount of my work until IE7 came out, and in the spirit of giving IE7 a fair trial, I decided to use it as my default browser. I started this back when I was still running Windows XP, and I was then using IE7 Beta 2, and the odd crash was acceptable for a beta product. I liked the tabbed browsing, and thought theat they actually did some things slightly better than firefox. I was impressed with the efforts made in standards support, and for this reason, I had persisted with it, even though the rendering engine is significantly slower (up to 5 times slower in some cases). When I started running Visat in november last year, I noticed that from time to time IE would crash (even though it was now out of beta), but I thought "patience... Microsoft are fairly good at stabilizing their products.... eventually, and the release of Vista was a rather rushed one, just so they could get it out the door before the end of the year, so I can expect that there may be some bugs that they find and fix along the way". Well, it's now mid April, and Internet Explorer is still crashing quite frequently, and today as I was browsing some Micosoft web sites, it crashed again. This is the final straw, I think they've had long enough to fix these problems. I've just switched back to firefox as my default browser.

Blog posts as an indication of my state of mind

When I started blogging way back at the start of 2004 (see http://scottbaldwin.blogspot.com/2004_03_01_archive.html, (I'd like to say before blogging became popular, but that would definitely be a lie) I was in a very stressful job that I didn't really enjoy, it was taking up far too much of my time, and was making me feel quite frustrated with the IT industry to the point where I was almost ready to leave the field for good. So even though I found out about blogging through my work in the IT field, the first blog I started was nothing to do with IT at all, in fact musings of a morbid mind was where I intended to post all of my political rantings and ravings, my general thoughts on topics (generally outside of IT), family and travel photos etc... , and just a general writing exersize for me. I started my geek blod Daily Dribblings of a Demented Developer shortly after (May 2004), but it was always the poor cousin of my musings (btw no prizes for guessig that alliteration is one of my favourite literary devices). I started work for Readify at the start of 2005, and since then, my writings on my Geek blog have steadily increased, and my writings on my musings blog have decreased, to the point where I noticed today that my geek blog has almost caught up, and what's more surprising the blog articles I've got in the pipeline are almost ALL for my Geek blog.

Its amazing the amount of difference working for a good company can make, there are still aspects of the IT industry I'm not enraptured about, and I still get stressed from time to time when things don't work the way I expect them to, but I have come to realise that there is a lot I do enjoy about IT.

Fixing the WCF - SSIS Web Service Task problem the easy way.

Literally minutes after publishing my post on calling a WCF Web Service from the SSIS Web Service task, I checked back with the forum post that I'd mentioned in that blogpost http://forums.microsoft.com/TechNet/ShowPost.aspx?PostID=1023821&SiteID=17 only to discover that Uwe Heinkel had given me the peice of information I was missing in the first place. So the crux is I don't have to hand craft the dtsx xml file to get it to play ball.

 

I simply remove the following line from the wsdl file

<xsd:import schemaLocation="http://localhost:50344/MyWebService/Service.svc?xsd=xsd2" namespace="http://schemas.datacontract.org/2004/07/"/>

 and it all just works as expected.

 

damn.... I was having soooo much fun wading around in the dtsx xml file.

Calling a WCF Web Service from the SSIS Web Service Task

I have recently been doing some work using SSIS, and this is really the first time I've used for anything serious (ie beyond doing the tutorials that come with SQL Server 2005). I have also recently been learning WCF and as we needed to create some Web Services to deliver the information to SSIS, I decided that this would be the perfect opportunity to put my newly acquired WCF skills to good use. Now I always like to start things out very simply, which in this case meant prototyping the SSIS Web Service Task with the most basic of WCF Web Services I could possibly create, thinking once I get this working, I can then move on to the actual task that I want to achieve with confidence... unfortunately this is where my whole plan started to unravel. For those following at home, here are the steps I took.

1. Create a new WCF Web Service in Visual Studio 2005 by selecting File->New->Web Site, then choosing a WCF Web Service. (Note .Net 3.0 and related Visual Studio extensions are required for this. Also choosing a File based or IIS based web service shouldn't matter).

2. Open the web.config file change the binding to basicHttpBinding (although the same end result is still achieved with the default wsHttpBinding).

3. Expose the metadata over httpget by adding the following line to the service behavior in the config fil

<serviceMetadata httpGetEnabled="true" />

4. run the service.

5. Create a new SSIS project in Visual Studio by selecting File->New->Project, then choosing Business Intelligence->Integration Services from the menu.

6. Add an http connection that points to the wsdl file for your service, and a Web Service Task to the Control Flow that uses the http connection.

7. Edit the Web Service Task, and download the wsdl exposed by your service, and save it locally to a file by using the OverwriteWSDLFile = true on the General page.

8. Select Input, and then select 'MyService' from the drop down.

 

At this point I recieved an error message.

 

Item has already been added. Key in Dictionary: ‘DataContract1’ key being added : ‘DataContract1’

It then would NOT populate the Method DropDown, so I was unable to select a Service Method to call, and because you are confined to selecting from the drop down, there is no possible way to make any of the Web Service method calls.

 

Now I'm the superstitious kind when it comes to writing software, my firm belief is that if you can't get the most basics Web Service working, then you may as well not try to get your complex web service that wants to pass around meaningful data, working, you'll end up in tears.

A google search on the problem yielded the following forum post http://forums.microsoft.com/TechNet/ShowPost.aspx?PostID=1023821&SiteID=17 which was helpful in revealing the underlying problem. The problem is with the generated wsdl and corresponding schemas, not sure exactly what is wrong with them, but for some reason, the UI in Visual Studio doesn't like the way WCF publishes its metadata, in particular the way it published metadata about its DataContracts (complex types). So I thought I'd make the default WebService that WCF generates even simpler by removing any trace of the DataContract. This worked fine, I could call a WCF WebService from within SSIS no problems... with the restriction that I could only pass primitive types to/from the web service. now this might be fine for your purposes, but I required something a little more powerful, so I decided to persist with figuring out the solution.

 

I tried a number of things with the generated schemas, but I think in the end I'd exhausted my limited understanding of wsdl and xsd, and was no further ahead. Next I remembered a very simple statement by a fellow collegue of mine Andrew Ball who was at the time giving me some SSIS advice. He said "... under the hood an SSIS package (dtsx file) is just a big xml file". Armed with this I began to see the issue as more of a tooling issue, and thought that maybe the issue is purely with the designer. So I cracked open a sample that Andrew had sent me and found where the various Web Service properties were being set. I also cracked open good old reflector and took a look at the way in which Complex Types were handled in the Microsoft.SqlServer.WebServiceTask.dll Which is responsible for the Web Service Task in SSIS. about half an hour later (give or take 10 mins) I had the following XML

<WSTask:MethodInfo WSTask:MethodName="MyOperation2" WSTask:MessageName="">

<WSTask:Documentation>

Basic Hello world service call that uses a DataContract

</WSTask:Documentation>

<WSTask:ParamInfo WSTask:Name="dataContractValue" WSTask:Datatype="DataContract1" WSTask:ParamType="Complex" WSTask:SeqNumber="0">

<WSTask:ComplexValue>

<WSTask:ComplexProperty WSTask:Name="FirstName" WSTask:Datatype="string" WSTask:ParamType="Primitive">

<WSTask:PrimitiveValue>Scott</WSTask:PrimitiveValue>

</WSTask:ComplexProperty WSTask:Name>

<WSTask:ComplexProperty WSTask:Name="LastName" WSTask:Datatype="string" WSTask:ParamType="Primitive">

<WSTask:PrimitiveValue>Baldwin</WSTask:PrimitiveValue>

</WSTask:ComplexProperty WSTask:Name>

</WSTask:ComplexValue>

</WSTask:ParamInfo>

 

Which when placed inside a web service task element, can be used to successfully call the web service with the DataContract.

 

So my conclusion is that there is a bug in the SSIS Web Service task UI that doesn't handle populatinf the Method drop down correctly (I'd trust WCF before I'd trust SSIS), but this is merely a bug with the UI, not with the way in which the Web Service task actually goes about calling the web service.

 

 

Added after post (literally minutes after): For the easy way to solve this problem see http://sjbdeveloper.blogspot.com/2007/03/fixing-wcf-ssis-web-service-task.html

Flickr becomes alive

Just found a really neat Plugin for Windows live writer called flickr4writer that allows you to add images from your Flickr photos by simply clicking "Insert->Flickr image...", so to celebrate here is one of my flickr photos.

 

Molly helps Microsoft with Standards

This is huge... I've been up against some major deadlines so I haven't been keeping up with all my blog reading, but today I decided that before I go home I was just going to catch up on a few of the more important ones, and so I was reading the IEBlog when I stubled across this article, the basic crux of which is that Molly E. Holzschlag has "signed on with the Internet Explorer team on a contract basis to work on standards and interoperability issues".

 

I have actually heard Molly speak at the Web Directions South conference last year, and she is a really inspiring speaker, and someone who tirelessly advocates for better adoption of Web Standards. It'll be interesting to follow her progress at The Daily Molly.

Published articles for The Readify Tech news letter

Just thought I'd mention that I now have a published two part article on the Readify tech news letter on SQL Server Express Edition.

 

You can view the articles at

SQL Server Express (Part One) - Another reason to move your user applications onto a stable platform

and

SQL Server Express (Part Two) - How User Instances Work

It finally dawned on me why I don't like the name "ASP.Net AJAX"

Microsoft renamed their AJAX offering from ATLAS to ASP.Net Ajax around November last year. Now I personally never quite understood what was so bad about ATLAS, but I do realize that as a general rule, product code names generally have to change at some point before the release to the main stream, (Longhorn -> Vista, etc...), but I must confess that quite often Microsoft completely screw it up.

 

I guess there is significant pressure from various departments, notably marketing and branding who want to maintain a consistent theme, and have the morbid fascination with trying to imbibe descriptive meaning into product names, and also from the legal department who are desperately trying to avoid trademark infringements. This is why I didn't blog about the name change when I first learned of it, but I always felt a huge sense of disappointment about the change, but couldn't quite put my finger on why I was so disappointed... until this week.

 

Beyond the obvious lack of imagination displayed taking the two core technologies (ASP.Net and AJAX), and just combining them (which in itself causes some confusion), I realised something this week when I was attempting to debug an issue with the library. As a developer one of our most important tools is our favourite Internet search engine. Whenever we come across a problem say with a CSS styling that isn't displaying correctly in a browser we just go to our favourite search engine and type in a question, for example "CSS min-width Internet Explorer". In Google the first page of the previous search returns all types of useful hits, including information regarding the fact that min-width is not supported in Internet Explorer 6.0 and a number of ways you can work around this. This is very powerful and allows us to do our jobs a lot quicker.

 

I discovered when trying to debug a problem with ASP.Net AJAX that this using ASP.Net and AJAX is not only longer to type than ATLAS, but returns all kinds of results that you wouldn't really be interested in. If you just type in ASP.Net AJAX into Google, the first page returns (as you'd expect), the official home page (ajax.asp.net), and a few other relevant topics, but then it starts to get more off-topic, like other component vendors offering AJAX style controls in ASP.Net such as ajaxium, ajaxpro, etc..., this is then further confounded if you need more search terms. The search is helped marginally by quoting "ASP.Net AJAX", but still returns that ajaxium website in the first page of results.

 

If Microsoft had've done what the trend is in the rest of the industry and actually used some imagination to come up with a short sharp name that was unique in the industry space, it could've made all our lives a little bit easier.

Dell Quickset disabling Internal network card

I had an interesting network problem today. I arrived at work, plugged my laptop into the power, and then into the network cable, switched it on only to find that I had no network connectivity. After rebooting twice, and discussing it with my colleague sitting next to me who has pretty much an identical laptop, connected to the same router, with no such networking problems, I did a bit of poking around.

I have a latitude D820 running Windows Vista, and the Dell Quickset power utility. One feature of quickset is that it disables the internal network card if you are not using it and you are on battery power, to get a few more minutes of battery life. It turns out that it must be a bit buggy as this turned out to be my problem. It had disabled my network card from the night before, and admittedly I had suspended my session, but then this morning, even after plugging into the power at work, and after rebooting my system twice, my network card still failed to respond. I fixed it by explicitly setting in the quickset UI to "always activate on battery".

IE Developer toolbar rocks

Just downloaded the latest beta of the IE Developer toolbar, and I must say it seriously rocks....

 

I've been using previous versions for a while now, and have found it extremely helpful in debugging styling problems, but this version goes to the next level. I haven't had a good chance to put it through it's paces yet, but I'm sure the "Style Trace" functionality will prove indispensable. It'll be interesting to see what will be in the final version of this product, they may even start to challenge firefox as the "Developers preferred browser", although I think they still have a way to go there.

Web Directions North gets the nod of approval from Microsoft Luminaries

Just read a post on PeterL's blog, about Web Directions North. I went to the Web Directions South conference myself. Here's my wrap up of the Aussie version, although I must admit that my post doesn't do it justice, the event is well worth attending, and will challenge the way you think about the web.

 

Warning: if you do anything of note during a session, you'll probably see yourself used as an example of the power of web 2.0 as the speaker pulls down the flickr image of you minutes after the event.

 

tags: wd06

Back and Live

After much frustration with Windows Live Writer (Beta) and Blogger (Beta) having communications issues, I am now back blogging with Windows Live Writer.

 

Gotta love trying to get two beta products communicating.

 

 

Technorati tags: blogger, Windows Live

As Kermit would say ... "It's not easy being Green"

A fellow collegue of mine Darren Neimke has posted a rather interesting post on his blog, which has got me thinking. Now I have to admit, I do consider myself to be a "Greenie", however, far from having "jumped on the bandwagon, I have been a member of The Australian Greens now for well over 3 years, and in fact I can trace my pilgramage from the high slopes of comfortable numbness about environmental issues into the radical valley of the shadow of Greenie Activism to a course I took back at uni, almost 11 years ago. A philosophy course aimed at Engineers called Technology and Human Values in which we studied "systems engineering" approaches to the ways technologies are used and abused both here and in developing countries.

Anyway, my main reason for writing this blog article is to suggest to Darren, that I feel has has understated some of the issues slightly. I am ussuming that some of the understatements were intentional and for effect like " .... Or some ice melting in Antarctica." trivialising the disapearance of entire glaciers, however, I the one I really want to concerntrate on is the issue of research and the problems of people reading something and just believing it. From his post

Research is a skill that is being diminished by the one-click world of Google and the new media. People hear something and believe it to be true without even having the faintest clue of how they would ratify such assertions.

I think in this statement are a number of problems all rolled into one.
Firstly, yes we live in a society based on instant gratification. Google is a great tool, that can be used to justify any point of view you want to throw at it.
Secondly, I think it is quite difficult to "ratify" certain facts. The problem comes from the nature of the beast that is science. To really fully understand the latest scientific findings, requires you to pretty much be at the forefront of scientific reseach in that particular area, and this requires not only having a PHD, but being actively involved in hands on research. The problem is that even scientists at this level can't always seem to agree, and the way in which research projects get funding often requires researchers to make bold (sometimes even rediculous) statements to gain the attention of the various institutions that are willing to put up the money to fund this research. How is your average punter expected to be able to know what the truth is even if they have tracked down the information from people considered leaders in their field.

Not wanting to be all doom and gloom, I think that there are solutions emerging to these problems.

If I may use a buzz-word, with the web 2.0 paradigm, we are starting to understand the "wisdom of the crowd", in fact google has used this in its search algorithms for a while, but there are even more powerful tools that enable collaboration on an unprecidented scale, sites such as wikipedia are classic examples of this.

With respects to the scientific community, by its very nature, it is a dynamic exchange of ideas and theories. unlike other debates we encounter, such as religion, politics etc... there are not the same intitutionalised barriers to callenging the status quo. If a scientist or group of scientists have a theory, this will be critiqued by many other scientists who will then be able to repeat either the same experiment and validate the findings, or disprove them. Even if a scientist or group of scientist start to hold some political sway, and spread inaccurate theories for a time, eventually someone will be able to challenge the theory, and a better theory will prevail.

So where does that leave us with the environment debate. Well, my personal feeling is that the premise that us 6 billion humans are having a noticable impact on the world we live in has been kicking around for long enough now, and the vast majority of scientists who are working in the field seem to agree on some of the big ticket items that constitute climate change. What the results might be are admitedly speculation. Educated speculation, but speculation none the less. However, I think that we can no longer use the ignorance is bliss approach to keeping the status quo. We have to realise that systematic change is essential if we as a species want to continue to maintain the quality of life that we currently do. Even more so if we want to take any significant amount of non-human species into future. This then becomes a battle that sometimes needs to be fought on a political level, and sometimes on a personal level. So Darren, next time I'm bending your ear about how stupid it is to be logging our water catchments for wood chips in the middle of a drought, and you slip seemlessly into that glazed over look, at least you'll know where I'm comming from.

Technorati and claiming your blog

I started playing around with Technorati the other day and decided to "Claim my blogs", just to see if anyone else was linking to me. I have two blogs, this one which is my geek blog, and another one that I reserve for my thoughts on politics/art/etc... anything that's not technology related. To my surprise I found quite a few people link to this blog, but to mt disappointment, found that I had absolutely NO links to my other blog. I guess that's no surprise really considering I have never been really active in promoting my other blog. So I thought I'd actually attempt to add some technorati goodness by putting in a shameless plug on my popular blog for my less popular blog.

Musings of a Morbid Mind.

No prizes for guessing that aliteration is one of my favourite literary devices.

Development V's Production Environment

An interesting post on software environmental issues by Mitch Denny caught my attention today, not only because he sent a link around on our internal tech list, but because it's something that I think continually trips up a lot of developers (myself included). Mitch basically argues (and I agree) that development should be done in the same basic environment as the production system, he plays this off against infrastructure people who tend to want to keep a gap between production and development for safety reasons, I mean wo knows what wholes these crazy developers are going to poke in our top-notch security infrastructure.

Most of my experience has been in the shrink wrapped software industry ans as such, there is little point trying to match development and production environments, so in this case I suggest develop on the environment that makes the developers most productive, but then make sure you test, using whatever virtualisation technology you prefer, on ALL platforms in your supported platform matrix.

With enterprise development however, I would say that one should develop in an environment as close as is practicable to the production environment. It never ceases to amaze me the amount of problems that arise because of differences between the development and production environments. Have you ever heard a developer say "gee, it doesn't do that on my machine". The closer you make development environments to the production environment, the less you'll hear this anoying phrase.

The flip side of the coin is that developers need to understand fairly well the differences between their development environment and their production environment. One thing that has bitten me in the past, and a lot of other developers as well is the difference between IIS and the Visual Studio Development Web Server (The web server formerly known as Cassini). There are a number of differneces, and also a number of reasons that developers like to use it over IIS. One major difference which always seems to trip me up is that Cassini passes every requested file through the ASP.Net pipeline, as opposed to IIS which will serve a lot of files normally (ie *.css, *.jpg, *.gif, *.png, *.html, etc...), and only pass the ASP.Net specific files (*.aspx, *.asmx, *.ashx etc...) through to the ASP.Net pipeline. When you first start a project you don't really notice any difference until you think to yourself, "gee, it would be nice to add forms authentication now", and so you put in the standard <deny user="?" /> element and all of a sudden your login page has no graphics and doesn't look anything like how you designed it. This is because Cassini is sending all of you image and css files through the ASP.Net pipeline and they are getting blocked by the Security module because you are not authenticated... yet.

Tags: ASP.Net, IIS

Visual Studio 2005 misbehaving after Automatic updates

I have spent the morning struggling with Visual Studio. It has been crashing non-stop, but I have got to the bottom of the problem, and am happy to share with the rest of the world.

Visual Studio was happily starting and loading my solution, but whenever I clicked on the test menu and attempted to do something like open the TestView window, VS was crashing. If I tried to open a localtestrun.testrunconfig file it would crash with the following error "Could not open assembly System.Data version 2.0.0.0 .... The system cannot find specified path".

I won't bore you with the process I had to go through to find the resolution, but the problem turned out to be that this morning My machine installed some updates from Windows Automatic Updates. Most of these updates were succesful, but one of them, namely "Security Update for Microsoft .NET Framework, Version 2.0 (KB922770)" had failed with the following error "Error Code: 0x643". after doing a search on this particular error I found the following article http://support.microsoft.com/kb/923100. However, I did not need to follow the resolution outlined there, I simply went to the Windows Update site and manually installed the update.

I hope this helps someone out before they have to spend half a day like I did trying to track it down.

PopupControlExtender inside an EditTemplate

I found a problem today with the PopupControlExtender (Part of the ATLAS Control ToolKit), that occurs when it is embedded inside an EditTemplate for a GridView Control. I suspect the same problem would occur inside any template field, and potentially for other ATLAS Control Toolkit controls.

I wanted to edit a date field inside my EditTemplate, and so I set about implementing something not too dissimlar to the example for the PopupControlExtender on the Atlas Control Toolkit website. Firstly I prototyped it on a page all on its own, no problems everything worked as expected. Then I attempted to place the same code inside my EditTemplate field. At this point when I attempted to go into edit mode I recieved an error message "Assertion Failed unrecognized tag atlascontroltoolkit:popupControlBehavior". Obviously the necessary javascript libraries were not being downloaded when the page rendered. I assumed this was because of the dynamic way in which the EditTemplate renders.

To fix this problem, I simply placed an empty PopupControlExtender onto the design surface of my page. This ensures the necessary javascript files are downloaded, and when the EditTemplate is rendered, the ATLAS engine knows what you're talking about.

I also encountered a coulpe of bugs with the popup control being wrapped inside an update panel. I managed to fix these by simply downloading the latest CTP of the ATLAS Control Toolkit, (now called the "Microsoft ASP.Net Ajax Toolkit"... sorry guys, this is yet another example of how insisting on meaningful naming of products can really kill any excitement and mistique around it, I'm afraid it just doesn't have the same panache as "Ruby On Rails", and if you really want to speak to the buzzword concsious managers of today, you have to break away from boring naming conventions and inject some creativity into the process).

tags: ASP.Net, AJAX

Reflections on Web Directions

I have to be the slowest blogger in the IT community. Over a week ago I attended the Web Directions conference. The conference was really good, and has inspired me in a few areas. Many other bloggers have had their say on the conference, but I may as well have mine, just for the sake of redundancy.

One of the common themes from both the Accessibility speakers (Gian Sampson-Wild and Derek Featherstone) and the User Experience speaker (Kelly Goto) was user testing, and not just asking the user what they think. Kelly Goto's quote was 'We listen to what the users "didn't say" and observed what they did'. I have also been challenged in the area of accessibility, I think the quote for me came from Derek Featherstone, and that is "The web is Accessible by default, we make it in-accessible". It has inspired me to go have a look at the way I develop, and the bad habits I've gotten into. The truth of the matter is that it is not really that much effort to get into good habbits that make web sites more accessible.

I really liked Jeremy Keith's AJAX sessions. The first session started out a little basic, but become more interesting in the last half. In the second session Jeremy discussed a technique he called "Hijax" which is aimed at ensuring accessibility, and support for down level browsers. I am looking forward to seeing how well I can apply his techniques using ATLAS. While on the topic of libraries, Jeremy did make one statement that I'm not sure I can agree with. He said that he didn't believe in using thrid party libraries for doing AJAX, firstly becasue AJAX wasn't that complicated and secondly because if something goes wrong in the library as a developer you'll need to be able to fix it. I almost agree with his first statement, but even still, I am a big fan in NOT re-inventing the wheel. If there's a library that has a great ranking control for example, and they've coded it so that it works across all browsers in your supported browser matrix, then there is a lot of testing and coding that you can potentially avoid. There are always bugs in any software, and AJAX APIs are no exception, the skill of a good developer is to be able to use an API in such a way that they can workaround any bugs in the underlying API, I've lost count how many times I've had to do this myself. Also, as a Winforms developer, I am extremely greatful that I do not have to write Win32 anymore, and I'm sure those of you who have written Win32 would agree with me. Having said that, his "hijax" mechanism of progressive enhancement is really cool, and my current goal is to go through all the ATLAS controls (the API I'm currently using) and see how I can apply this technique to them.

I went along to John Allsops talk on microformats to hopefully pickup anything I'd missed from the first time I heard it, and was inspired all over again.

One minor thing I think the organisors can improve on is something I saw at the Tech-ed conference, the "re-charge desks". The tech-ed organisors had desks with a series of power boards that people could plug their lap tops into between sessions. This would have been really good for me as my laptop is now a year and a half old, and my battery is showing its age, it wasn't even lasting 2 hours.

Worth mentioning:

• Kelly Goto made reference to an example of an interesting User Experience project called Datelens. It is a callendaring visualisation which plugs into outlook (nothing to do with the web at all). I downloaded it and it's quite interesting, definately worth a look. Also it is written in .Net 1.1, which must have been quite an acheivement, and I think with WPF just around the corner, there will be a lot more of these types of user experience applications about.
• The inaugural McFarlane proze for Excellence in standards based web design was awarded to http://www.museum.vic.gov.au/caughtandcoloured/. This site is really worth checking out. It is especially interesting to me because it is written in ASP.Net demonstrating that it is possible to write high quality standards compliant accessible web sites using ASP.Net.

• - The last speaker of the conference was Mark Pesce (inventor of VRML, explained some of his concepts of social software, and described a project he worked on that attempted to aggregate your social behavior patterns and create social network models simply by using the ability of a blue tooth enabled phone. I thought there’d be some people at readify quite interested in this sort of stuff. http://relationalspace.org/

tags: wd06

ATLAS Data Applications and Down-level support

I must confess that aslthough I am excited by the whole AJAX phenonenon, and the rich client experience that it provides for the majority of desktop browsers, I sometimes feel concerned about those that for whatever reason aren't able to experience the full beauty of a javascript enabled client. There are two main groups of people that are missing out

1. Users with certain accessibility needs (often browsers and tools designed for people with disabilities have to run without the javascript support we usually take for granted) and
2. PDA users (This is an increasing market, and to date only 1 browser has implemented any kind of javascript engine for devices, Opera.

So I am always looking for ways to enhance these users experience.

My AJAX toolkit of choice (more through accident than intent) is ATLAS (Microsofts library). ATLAS contains a data enabled control called a ListView which is quite powerful. It works on a client side JavaScript based object model and the best way to get to know it is to read the data section of the ATAS documentation.

 

So the idea is, get a funky ATLAS based website working, and then attempt to create a version of that website as quickly as possible that can be used by people who can't (or won't) use javascript (similar to what google does with gmail).

 

I decided to make the sample as simple as possible without loosing anything along the way. I decided to create a Shopping List, that simply contains a quantity (double) and an item (string).

The bueaty of the ListView in ATLAS is that it can use the System.Component model similar to other databinding in ASP.Net 2.0. So the first step in creating a webservice that derives from Microsoft.Web.DataService (refer to the ATLAS documentation for more details).  I then create my ListView templates and hook it all up using XML-Script, and pretty it upwith any CSS I may care to add. The result can be seen here (Please forgive me I'm not a graphic artist). Once this is done, I needed to create a Non-ATLAS version of the same page. As my webservice already uses the System.Component model, it was extremely easly to add a GridView control, select an Object DataSource and wire it up, and away it goes. All in all it took me abou 10 minutes to create the NON-Atlas page. It would have taken me another 2 minutes to enable things like edit/delete, but for the moment I wanted to keep it the same as the ATLAS version, this may be the subject of a future post. The only significant difference between the two pages is the presence of a select link on the NON-ATLAS enabled version whereas the ATLAS version manages the select operation simply by handling the mouse click.

 

Once you have these two pages it is simply a matter of choosing your favourite way to direct users with different broswser capabilities to the correct site.

ajax gotcha for new players

Being fairly new to the "Brave New World" of ajax, this week I fell into a fairly common trap that gets new players. I studiously looked around for the best practice implementation of making an http request using javascript and came up with the following javascript function which allows the user to make the http request, specify a callback function, and even goes the extra mile of implimenting timeouts (I guess the only other thing I really should do is implement an OnErrorCallback).

function MakeRequest(url, timeout, 
    onCompleteCallback, onTimeoutCallback)
{
    var completeCallback = onCompleteCallback;
    var timeoutCallback = onTimeoutCallback;        
         
    var xmlHttp=createXMLHttpRequest(); 
    
    if (xmlHttp)
    {          
        xmlHttp.onreadystatechange = function()
        {                
            if (xmlHttp.readyState == 4)
            {            
                window.clearTimeout(timeoutId);
                if (xmlHttp.status == 200 || 
                    xmlHttp.status == 304)
                {                
                    completeCallback(
                        xmlHttp.responseText);
                }
            }
        };
                                 
        xmlHttp.open("GET",url,true);
        
        var timeoutId = 
            window.setTimeout(function()
        {        
            if (callInProgress(xmlHttp))
            {
                xmlHttp.abort();
                timeoutCallback('timeout');
            }
        }         , timeout);
        
        xmlHttp.send(null);        
    }
}

function callInProgress(xmlHttp) 
{
    switch ( xmlHttp.readyState ) 
    {
        case 1, 2, 3:
        return true;
        break;

        // Case 4 and 0
        default:
        return false;
        break;
    }
}


function createXMLHttpRequest() {
  if(window.XMLHttpRequest) {
    try { 
      xmlHttpRequest = new XMLHttpRequest();
    } catch(e) { return null; }
  } else if(window.ActiveXObject) {
    try {
      xmlHttpRequest = 
        new ActiveXObject("Msxml2.XMLHTTP");
    } catch(e) {
      try {
        xmlHttpRequest = 
          new ActiveXObject("Microsoft.XMLHTTP");
      } catch (e) { return null; }
    }
  } else return null;
  return xmlHttpRequest;
}

I then wanted to call a .Net HttpHandler, and lets just say for arguments sake that the htp handler looks like this

public class Handler : IHttpHandler
{
    
    public void ProcessRequest (HttpContext context) 
    {
        System.Threading.Thread.Sleep(6000);
        context.Response.ContentType = "text/plain";
        context.Response.Write(Guid.NewGuid().ToString());
    }
 
    public bool IsReusable 
    {
        get 
        {
            return false;
        }
    }       
}

so the call from javascript would simply look something like this

        MakeRequest(
            'Handler.ashx?cookie=abcd&another=xyz', 
            20000, onRequestComplete, 
            onRequestTimeout);       
        
        function onRequestTimeout(response)
        {
            alert('timeout');
        }
        
        function onRequestComplete(response)
        {
            alert(response);
        }

The problem as I found out through good old wikipedia relates to the internet explorers caching of the "GET" http command as described in this article on HMLHttpRequest.
As discussed in the article, there are a number of ways around this, switch to using POST or ensure that you switch the caching off in the http headers. I chose the former.

Automatic updates + Automatic Reboots

Is it only me or does anyone else find this really frustrating.



I got this in the middle of reading my email this morning, and I really didn't want to reboot in 5 minutes. I would have quite hapoily rebooted in half an hour or so when I'd finished what I was doing. The fact that the restart later button was there but disabled just added to the insult.

I think from a layer 8 perspective, Microsoft need to be really careful when forcing reboots. If this happens too frequently, it may be frustrating enough that people actually turn Automatic Updates off, making their systems progressively less secure and vulnerable.

Web Directions

On thursday night I went along to a Web Directions preview night where 2 of the speakers from the upcomming Web Directions conference Ben Barren from gnoos and John Allsop from westciv were giving previews of their presentatoins at the conference. It was a good night, I particularly enjoyed Johns talk on microformats. This is a topic that I have been interested in for a while now, and may have more to say about it later.

Unlike other events I attend, where I know a fair few people, this event I only found one person I'd met before was Nigel Watson, an Architect Advisor from Microsoft. So it was good to make new contacts and expand my circle beyond the sphere of the Melbourne .Net developer and SQL Server sphere. I got to meet Cameron Adams whose book "The Javascript Anthology" I am currently half way through, and also met the people from westciv who write the best damn web standards online training courses in existance.

When I first heard about the web directions conference, I thought that as much as I'd love to go along, I probably wouldn't be able to attend given that I was already attending Tech Ed at the end of August, and I'd be pushing it a little if I asked my employer to pay for another conference and let me have the time off so close after Tech Ed. As the saying goes, "the best layed plans of mice and men gang aft aglay", there was a lucky door prize drawn at the end of the night, which was a free ticket to attend the Web Directions conference. Guess who won........ So now all I have to do is convince my employer to let me have a couple of days off to attend..... that shouldn't be too hard.

Stepping through ATLAS Javascript files

I'm trying to debug an application I'm writing using ATLAS, and I'm wnating to know why my attempts to make browser client magic happen are resulting in java script errors. Fortunately Visual Studio provides a level of debugging for java script files which means that if you're in debug mode when a java script error occurs, you'll automatically be taken to the point in the code where the problem occured and given some useful things like a stack trace and a variable watch window. You can also set break points in your own code, however, if you have registered the scriptsw using an Atlas Script manager as is advised when writing ASP.Net applications


<atlas:ScriptManager ID="ScriptManager1" runat="server" >
<Scripts>
<atlas:ScriptReference ScriptName="AtlasUIDragDrop" />
</Scripts>
</atlas:ScriptManager>


you will not be able step into any of the ATLAS libraries you happen to be using as the debugger complains that the source files are not available. This is because of the way the ATLAS scrip manager embeds the java script.

today I found myself thinking that it would be handy to be able to step into these files to find out what's going on prior to the error.

To solve the problem

1. Copy the necessary java script files from the install location into your project and mark them as "Embedded Resource". In my case Atlas.js and AtlasUIDragDrop.js


2. Comment out or remove the Atlas Script Manager component


3. Add the script files as you would add any normal java script file (ie with the <script> tag)

Once you have done this you should be able to step into the ATLAS java script files to your hearts content, but be warned you might find it a little scarey in there.

The only problem with this is that it won't work for Server Side Atlas controls, such as UpdatePanel or Extender controls, but if you're not using them, or if you can comment them out while you debug stuff, then you're set.

Google Maps and sustainable transport

I have found a great site that is a mashup of google maps. http://www.bikely.com. The idea is you can search the user entered database for the best way to get between various points by bike. The website also allows you to sign in and create your own bike route so you can share your favourite rides with others. You can even put comments at each point that appear as pop-ups on the map. I'll have to start riding my bike more so that I can contribute.

Browser Version hell

Those familiar with the concept of DLL hell and have experienced it first hand often point to web based applications as the savior for such deployment issues, and to an extent they're correct. However, as the push for a richer client experience from web applications grows, and as the technologies that enable it mature, DLL hell is being replaced by Browser version hell. We are all aware that Internet Explorer and the Mozilla based browsers (firefox) have their own peculiarities, but even beyond that are the versions of these browsers. In fact, IE 6 has been particularly slammed for some of its, shall we say "non-standard" behaviour. As a result of this I have been a big fan of firefox that is considerably more standards compliant. This has lead to web designers targeting firefox first, and then tweaking sites as needed for other browsers.

For the past 6 months or so I have been neglecting firefox a little in order to try out the all new IE 7 in its various beta forms. IE 7 is supposed to be far more standards compliant, amongst adding other features users of firefox and other browsers take for granted. So as a result, I haven't updated firefox for a while, and using firefox version 1.0.7 I was testing out a new Atlas based control called the collapsible panel control, and the bad news is, It doesn't work. The area that should display when the panel is expanded is displayed completely blank. Upgrading to version 1.5.04 fixes the problem completely. Now admittedly the Atlas Control Toolkit is very much in a pre-release state, but it does raise a very pertinent question.

Where should web developers draw the line and say I'm not going to support versions of browser X before version Y, and how should web sites indicate that a user should upgrade their browser? Also, how many different versions of browser X should we be realistically expected to test under? This I believe is different to the idea of degrading nicely (ie what should happen if a browser doesn't support Java Script or only supports a limited subset of Java Script). All I can say is that I hope the team at Microsoft fix the problem with the control for firefox Version 1.0.7.

Web Application Projects and SQL Express Data Sources

I have been struggling today with some SQL Express related data issues. I have been trying to use a combination of Web Application Projects and ATLAS, as I am convinced that Microsoft have completely stuffed up with their "Web Site" style projects, I really don't want to use them. So to do this you simply create a new Web Application Project, and then copy all the necessary settings from the sample ATLAS web.config file into your own web.config file, and ALL is fine... that is until you want to do something with SQL Express. Regardless of ATLAS, the standard approach is to add a new item to the project and select a SQL Database. This uses SQL Express in a mode called "User Instances". This is designed to allow you to treat the databases primary file (mydatabase.mdf) as though it is just that, a file (impressive if you know what's really going on under the hood and understand how SQL Server normally works), and this is how all the Microsoft demos go, except that they use Web Sites instead of Web Application Projects. The theory is then to add tables, and create a DataSet off these tables. So I thought it would just be a simple matter of creating a Web Application Project, and then following exactly the same logic. WRONG!!!!!!

My first efforts yielded the following error

Description: An error occurred during the compilation of a resource required to service this request. Please review the following specific error details and modify your source code appropriately.

Compiler Error Message: CS0122: MyWebApplicationProject.Properties.Settings' is inaccessible due to its protection level

This is because the default way of storing the connection string is to use the Properties folder and create an entry in the Setteings.Settings file. In this case there is a class called MyWebApplication.Properties.Settings. This has a protection level of "internal" meaning that only code from within the same assembly can refer to it. The code attempting to access it is NOT in the same assembly so it fails compilation.... DOH.

Next step was to attempt to rework it so that it got the connection string from the web.config file. After correcting all the places where I found a reference to the connection string, I received the following error.

Exception Details: System.Data.SqlClient.SqlException: An attempt to attach an auto-named database for file C:\Projects\AtlasApp\AtlasApp\App_Data\App_Data\Tasks.mdf failed. A database with the same name exists, or specified file cannot be opened, or it is located on UNC share.

Better, at least this time it's compiling. After spending entirely too much time attempting to figure out why SQL Server User Instances were failing, I eventually re-read the error message, and noticed the path it was attempting to access. C:\Projects\AtlasApp\AtlasApp\App_Data\App_Data\Tasks.mdf

Notice the two App_Data directories. Yep, so simply exclude the App_Data part of the path in the connection string and away it goes, everything now works.

The reason I write this article is because I found very little documentation on the problems. Problems which I thought would be occurring quite frequently if people are using Web Application Projects, but alas the only mentions I have found are a recognition by Microsoft that there are problems, and a vague promises of a white paper on the issue from Microsoft.

In conclusion, I've had quite a frustrating day, and I think Microsoft still have some work to do on Web Application Projects.

using a custom database for persisted SQL Server ASP.Net session state

I am in the process of upgrading a major project from ASP.Net 1.1 to ASP.Net 2.0, and my ideal is to have 2 versions of the web application able to be run so that I can compare what's going on in the .Net 2.0 version to what should go on according to the .Net 1.1 version. This doesn't seem too difficult, just use a different web folder right? Well, this would be the case except that we're using SQL Server persisted session state and although they use a different schema between .Net 1.1 and .Net 2.0, the default database name is the same "ASPState". After a bit of fiddling around I eventually solved the problem.

Step 1 run the following command line from the appropriate .Net 2.0 directory replacing ASPState_2_0 with the database name you wish to use

aspnet_regsql -S localhost -E -ssa
dd -sstype c -d ASPState_2_0

This creates the session state database with the appropriate schema in the specified database name instead of the default.

Step 2 modify the following line in the web.config file accordingly

<sessionState mode="SQLServer" allowCustomSqlDatabase="true" stateConnectionString="tcpip=127.0.0.1:42424" sqlConnectionString="data source=localhost;Initial Catalog=ASPState_2_0;Trusted_Connection=Yes;" cookieless="false" timeout="15" />

Step 3 Manually give the ASPNet user (or whatever user you are running the ASP.Net web application as the appropriate priviledges on the database tables in the newly created session state databse.

MEDC 2006

Just got back from the MEDC 2006 conference where I was lucky enough to recieve an I-Mate SP5 simply for attending.


The sessions were all very interesting, my personal favourite was probably the session on performance in the compact framework.

The other interesting thing I learnt is that people are actually reading my blog. Well, if that's the case I might have to buck up my ideas and actually put some half decent content on.

A new and exciting project

This week I started a new and very exciting project. I can't really give away all the details as the project is still in the planning phase, but what I can say is that it will be a Web 2.0 project. Now I know Web 2.0 can sound like a bit of a buzz word and as such looses meaning, but I think this article by O'Reily explains Web 2.0 in a way that cuts through most of the marketing hype.

The Web 2.0 concepts that we are thinking of using in this project involve the idea of Mash Up's of a few different services, the idea of collaboration and collective intelligence, technologies such as ATLAS (AJAX for ASP.Net) and the concept of the quick release cycle.

It is interesting that I have ended up on this project. I must be the least experienced ASP.Net developer at Readify, and the last time I did anything commercially for the WEB (almost 2 years ago now), was firmly rooted in Web 1.0 mindset. Since then I have been firmly stuck on the Rich Client side of the divide writing Pocket PC and Desktop style applications. The team lead on this project has suggested that this is probably a good thing as we can take my experience designing and working with rich UI's and not be limited in my thinking of what is capable on the web. We are all getting really excited about the potential of this project, and I will give more details as the project progresses.

To Train or Be Trained... That is the question

Over the past 15 months I have been consulting to QSR and it has been a fantastic experience. The only issue has been that since I've been there I have worked exclusively on a winforms .Net 2.0 application. This has meant that my ASP.Net 2.0 skills were fairly rudamentry, and weren't improving beyond the tiny bit of time I've been able to allocate to professional development on the bus on the way to work.
So I decided to do the Readify Professional .Net which has been recently re-vamped for .Net 2.0. So I rang Chris Hewitt who was instructing the course to tell him of my intention. Chris's paraphrased reply was something like ... "You've been using .Net 2.0 for over a year now, you know more about .Net 2.0 than I do, you can teach some of the course".

So this is how I ended up teaching three modules of a course I was originally intending to sit. The 3 modules I taught were Generics (one of my favourite .Net 2.0 features), Click Once (on Chris's suggestion) and refactoring. It is my goal to to be trained up to a point where I can teach the course myself, and it was good sitting under Chris who has extensive teaching experience. I have always enjoyed teaching ever since I supported myself through uni by tutoring high school maths and physics, and I know it is one of the best ways to ensure your own grounding of the material. The thing that scares me the most is questions. I know I shouldn't be scared, we aren't expected to know everything, and in the rare cases when Chris didn't know the answers, it proved to be a real journey of discovery for allpf us as we attempted to find the answers to these questions.

Click Once Deployment Gotcha

In attempting to prepare for a module I am teaching tomorrow, I am putting Click Once through its paces, and I'm finding there are some strange issues. I have been struggling with an unusual error for half the night.

I am Publishing my files to a local site, then openning up internet explorer, navigating to the site and then attempting to Install the application. The application contains a 4 files that require downloading, and it uses SQL Server Express. The Install starts out fine, it then decides it needs to momentarily open a browser window. Now my default browser is Fire Fox, and it decides to open this. At this point firefox decides to prompt me for a username and password. Why I don't know, but seconds after typing in my credentials, it brings up an error saying that it cannot download the application as there are missing files. The install fails.

I eventually figured out that if I change my default browser back to be internet explorer, the problem goes away. Hope this helps someone.

SQL Server Express with Advanced Services

As an avid user of SQL Server Express, it would be remiss of me not to herald the release of the much anticipated SQL Server Express Edition with Advanced Services with a blog post. This adds SQL Server Reporting Services and SQL Server Full text search as well as its very own version of management studio into the mix which now gives it a distinct advantage over its predecessor MSDE.

IMHO the only thing that is now missing from SQL Express is SSIS. I think that adding SSIS to SQL Express would allow people to enter more data quicker and would therefore come up against the size and performance related restrictions of SQL Server Express, and would cause people to upgrade to other SKU’s of SQL Server sooner.

SQL Server 2005 installation problems on various CPU's

As I have become the unlikely SQL Server 2005 "expert" at the company I am currently consulting to, I have been investigating a wide number of installation issue with SQL Server Express. There are a few main categories these problems fall into.

1. Installation of beta versions of SQL Server 2005 or the .Net Framework 2.0 (solution uninstall all beta versions of SQL Server 2005 or .Net 2.0, and if you've installed beta versions of Visual Studio 2005, see http://msdn.microsoft.com/vstudio/support/uninstall/default.aspx)
2. The CPU is NOT fully Pentium III compatible
3. Various issues ranging from disk space to memory to OS service pack level etc...

In this post I want to deal with problem #2, The CPU compatibility problem.
There seem to be a number of CPU's that claim some kind of Pentium III compatibility, and as the hardware and software requirements for SQL Server 2005 state, Pentium III is the minimum hardware requirement.

However, for whatever reason, the SQL Server installer does not check CPU compatibility too thoroughly as we have seen a in a number of cases that the installer will issue a Hardware warning, but happily allow the user to continue to attempt to install SQL Server 2005 but will in the very final stages of the install refuse to start the SQL Server service. I'm not sure of all of the reasons why this error potentially occurs, but I know 1 definite problem is that SQL Server insists on the cache prefethching instructions of the Pentium III CPU. This is missing in a fair amount of CPU's that claim compatibility with Pentium III's. I have asked the guys at Microsoft about the minimum hardware warning, and they confess it is a problem, and are reviewing whether they are going to be more specific in the future, however, I think there needs to be some kind of list of all the different CPU's that one might reasonably expect to be able to install SQL Server 2005 on, but don't seem to have the ability to. So here it is, or at least the results of my research, if anyone else has seen similar problems, please leave a comment with the CPU specs and any links to discussions about the problem on various forums.

Via Eden - http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=304355&SiteID=1

500 MHz AMD Family 5, Model 8, Stepping 12 processor (AMD-K6-2) - http://www.sqljunkies.com/Forums/ShowPost.aspx?PostID=167¬ification_id=421614&message_id=421614

VIA Ezra 800Mhz processor - http://www.sqljunkies.com/Forums/ShowPost.aspx?PostID=167¬ification_id=421614&message_id=421614

Microsoft Virtual PC For Mac OS 7.0.2 (This emulates a Pentium II) http://www.microsoft.com/communities/newsgroups/en-us/default.aspx?dg=microsoft.public.mac.virtualpc&tid=90715b2e-77ba-4f8a-a189-daee29467ff0&cat=en_US_595a5881-be4a-4db4-97b0-5f3a78602e23&lang=en&cr=US&sloc=en-us&m=1&p=1

Transmeta - http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=17395&SiteID=1 and http://lab.msdn.microsoft.com/ProductFeedback/viewfeedback.aspx?feedbackid=cee0915a-6039-4ca6-aa0c-1b76285d4ac3

Teddy Bear code reviews

I have been a big advocate of peer code reviews for some time now (see my previous post on peer code reviews), and I feel very lucky to have been a consultant at a company for the past year and a bit that firmly believe in peer code reviews. As my time comes close to winding up at this company, I am faced with the prospect of going to consult for a company that does not have the same attitude to peer code reviews, and I've been wondering what my approach should be.

My first thought is to attempt to convince the managers at whatever company I end up at that peer reviews are a good idea, and explain the development to the overall development life cycle, and show them the graphs that explain the cost of fixing bugs at various points throughout the software development life cycle, this in my opinion should be enough to convince any decent manager that an extra 2 hours in development is better than an the 20 hours of test/dev/management time if a bug makes it into system test, or worse if it makes it into the wild and is reported by a user. However, being experienced with the way management works, and knowing that not all managers listen to common sense or think that developers shouldn't write bugs in the first place, I've been trying to think how I will attempt to maintain my code quality in such an environment.

I'm not sure how well known the concept of teddy-bearing has become, but the urban myth starts with a development team that had a team lead who was as a developer pretty useless. The developers however, found he was great for one thing. If you were stuck on a programming problem, you could call him over and ask him to have a look. Because he was so useless, the developers would have to step him through the bits of the code that were causing the problem. In the process of explaining the problem at a level that the team lead could understand, the developer would quite often see his/her mistake and be able to fix it right there in front of the team lead which made the team lead feel as though he was immensely important. When this team lead eventually moved on to bigger better and greater things, the team was at a loss to know how they would fix these hard to find bugs, until one day, one of the developers who was stumped on a problem picked up a teddy bear that was lying around the office and sat it on his desk started explaining the problem to the teddy bear. They found that the teddy bear was every bit as effective as the team lead, and had the added advantage of having better people skills. The teddy bear become an office legend and was invited to all the office parties where legend has it he told far better jokes than the old team lead.

From this the concept of teddy-bearing was born, and is used whenever you encounter the phenomenon of explaining your code to someone in order to find a problem, and find the problem yourself with no real constructive input from the person you asked to help.

So what does that have to do with code reviews? Well, I'm a big fan of the whole subversive style of process improvement. If management won't buy source control, install subversion and maintain your own repository, if the team doesn't believe in unit testing download nunit and write your own, if there is no nightly build process, download nant and do it yourself. So if there are no code reviews, buy yourself a teddy bear, give it a name, and sit it on your desk. Fellow colleagues will probably think you are a little unusual, but you're a geek, people generally think you're more than a little unusual anyway.

When you get to a stage where you are completing a chunk of work, sit the teddy-bear beside you and talk it through your code. A lot of the psychology of a code review will be there, and in reality you will have one of the toughest reviewers you can possibly have..... Yourself.

QSR Releases NVivo7

The company I have been consulting to for the past year and a bit (QSR) have today released the product we have been working on, their flagship product NVivo 7.

I'd also like to add that it was released on time, and without stressing out the entire team for months leading up to the release. Just goes to show what good project management and appropriate resource allocation can do.

Testing for developers from the Braider Tester

A blog that I read from time to time has some great resources for testers. The blog is called The Braidy Tester. You may ask why I a developer is reading such a blog, well, I am interested in the entire SDLC, and always like to know what's going on when I throw the next build over the fence to the testers, and also, I don't think it hurts developers to be aware of how to be better testers themselves for testing their own code changes. A fact which The Braidy tester appears to agree with given a recent article on Testing For Developers

The spin detection machine

Interesting concept, a piece of software that detects the amount of spin in a speech. The model is based on analysing the frequency of various kinds of words in a speech and professes to be able to detect levels of deception and insincerity in English text. I must confess, I'm more than a little skeptical. I just think there are so many variables, ie the speakers/speech writers proficiency with the English language, different styles of speech writing, different topics, different audience. All of these things change the way we use language and shape what we say.

I guess the ultimate test will come when they decide to commercialise the software and produce a website dedicated to selling it..... and then run the software over the site. I guess this could be seen as an extension of the halting problem.

Arguments for continuous integration

The company I’m currently consulting to are pretty good when it comes to their internal development systems, although one of the main areas I can see for improvement is in continuous integration. They currently have a full build process is manually initiated, and works pretty well. I really think they need to go to the next step, that of continuous integration.

A couple of days ago I was checking in some very subtle but serious changes to the way we accessed SQL Server (we switched to use User Instances so that we could run as an LUA user). We were all a bit nervous about this change because although it was a relatively minor change, it was a significant unknown, in fact I was the only one who had any idea of how user instances worked. So as part of the code review, the reviewer asked me to run all of the unit tests just to be sure we hadn’t inadvertently broken anything, and sure enough a whole heap of them just curled up and died a horrible death. Being the insecure person that I am I immediately rushed to the conclusion that it was my changes and started questioning my own understanding of user instances. When running the application, I was able to reproduce the error under certain conditions on my machine, but no-one else was able to reproduce the error on their machine. The next day I spent the best part of the morning looking at the problem and eventually discovered that firstly the reason I could reproduce it on my machine was because I was using an obscure (non-default) option in the application, and secondly that the bug occurred with or without my changes from the night before. Armed with this information I then went to the team lead who then spent the next half of the day going back through labels in our source control attempting to find the release that this bug first occurred in. He eventually found it to be due to a check-in about 4 or 5 days prior. This indicated 2 things, firstly people had not been running even the effected unit tests for their check-ins, and secondly if we had continuous integration, and unit tests were run automatically at the point of check-in, we would not have wasted a day and a half of development time tracking down the origin of the bug. I told the development manager that this incident re-affirmed my faith as a unit-test evangelist.

SQL Sever & file permissions

In the course of trying to track down a problem running the application i'm involved in developing at the moment, I stumbled across the following article describing what SQL Server does with file permissions when it attaches to a database file.
The article states that for security reasons it changes the file permissions so that if a file has been placed in an insecure folder, then it attempts to protect it from someone without the privileges from moving or deleting it.
It seems to me to be a little mis-guided. Firstly, I would assume that it's the responsibility of the DBA, to place all database files in an appropriate place with appropriate permissions to the various users that require access. The secure by default paradigm is all well & good, but there is no way to turn this behavior off. Secondly, it doesn't really achieve anything. If a DBA has been dumb enough to place a vitally important database file in a folder that anyone has full control over, then if SQL Server is not attached, the user can easily cut & paste the folder wherever they want which will have the same effect as moving the original file.

Maybe i'm not seeing something here, I can't really see the need for this.

New toy

Well, I'm now the proud new owner of an O2 XDA Atom.

I have been hanging out since my XDA II started playing up, but I wanted to hold out for a Windows Mobile 2005 device.

So far, mostly I have been impressed. I love the ability to switch between lanscape and portrait screen modes, I love the in-built wi-fi, I adore the tiny size (in comparisan to the XDAII) and it seems there have been some improvements to the predictive text which makes it easier to enter data into, but there are a few things I haven't been so impressed with.

Maybe it's just buggy early release software, but I have had three occasions so far where the phone has stopped responding to the on/off button. I have also not had any luck what so ever with the devices alarm. I have had almost a week now where it has not woken me up even though I have set it. My partner says I should read the manual, but I'm sorry, it goes against every rule of usability and good design if I have to read a manual in order to set the freakin alarm clock. I'm happy to read the manual for other more complex tasks, but not to set the alarm clock.

Anyway, I will persevere and hope a firmware update is close. I'm also hoping to get back into some Pocket PC Development, I'm really keen to put sql mobile and the new version of the .Net Compact Framework through their paces.

Answer to question posed at VSLive

The thing I was most nervous about at my VSLive presentation was question time. I had an ingenious plan to avoid question time... I had timed my presentation, and it went for just over 1 hour, but in the words of a famous poet... "The best layed plans of Mice, Men (and Geeks) gang aft aglay". People from the audience stopped me mid-way through to ask questions. Most of the questions I was able to answer, but there was one that I said "I think the answer is yes, but I'll get back to you". The question was... "Does SQL Express support bulk copy?", and after some testing the answer is "Yes". As mentioned in my presentation (slides available here) SQL Express does not support SSIS (The replacement for DTS), however after testing, the bcp utility works fine. The following script can be used to verify that both import and export work fine.


use tempdb

-- Create a test table
create table BulkTest
(TestKey INT NOT NULL,
TestData NVARCHAR(50) NULL,
CONSTRAINT BulkTest_PK_Constraint PRIMARY KEY (TestKey ASC)
)

-- Add some test data
INSERT INTO BulkTest
(TestKey, TestData)
VALUES
(1, 'This is just some test data')

INSERT INTO BulkTest
(TestKey, TestData)
VALUES
(2, 'This is some more test data')

INSERT INTO BulkTest
(TestKey, TestData)
VALUES
(3, 'It is important to have test data')

-- perform bulk Export using bcp command line utility
-- bcp tempdb.dbo.BulkTest out BulkTest.dat -T -c -S ServerName\SQLExpress

delete from BulkTest

-- perform bulk Import using bcp command line utility
-- bcp tempdb.dbo.BulkTest in BulkTest.dat -T -c -S ServerName\SQLExpress

select * from BulkTest


VSLive Seminar post mortem

Well, my VSLive presentation was on Wednesday, and I personally was quite pleased with how it all went. I must confess, I was a little nervous, but I didn't let my nerves get the better of me. I had really positive feedback from people after the seminar, and even one request for the source code for my Photo Album Demo... enjoy! The demo sort of loses a bit by distributing it in this form, as it was more about the process of building it and displaying the Visual Studio integration features of SQL Server Express. There are only 5 lines of user written code in the project, and they are all to do with implementing Drag-Drop on the picture box so that you can add the photos to the database, the rest is all IDE generated.

There was one point that given the discussions after the seminar, I feel I should clarify. I went into a bit of detail describing the SQL Express only feature "User Instances" or RANU (Run As a Normal User). I do think it is an interesting feature and I can see some real potential for it, however, from the discussions after the seminar, it appears that I didn’t stress enough, that User Instances are optional (in the connection string "User Instance = true"). You don't need to use it, in fact, I would suggest that in the majority of usage scenarios, you will use SQL Server Express in your traditional Database server scenarios where you have a dedicated server machine that a DBA logs on to allowing her to perform maintenance tasks like Attaching databases etc... multiple clients would then log onto the database from their own machines with only the privileges they require to perform the tasks they need to. User Instances are only really required when you want to deploy an application in a single user type scenario, and you want to deploy the database file along with the executables and have the user (who by best practice in security, should NOT be an administrator on their own machine) attach dynamically to the database. I hope this clears things up.

For a really good, in depth article on User Instances, see Roger Wolter's msdn article on SQL Express User Instances.

By the way if any of the people in the seminar find my blog, please feel free to leave any feedback on my performance, I would like to know how to improve my presenting.

VS2005 SQL Server 2005 RC/RTM Gotcha with MARS

For those who are familiar with MARS (Multiple Active Result Sets) and have been excitedly using them with versions of visual studio 2005 prior to the the june CTP, you may be interested to know that there has been a change in the default behaviour.

MARS is OFF by default, and any code written to use it as in this code project article will cause an exception if you don't explicitly add the following to your connection string "MultipleActiveResultSets=true".

The exception you will receive is "There is already an open DataReader associated with this Connection which must be closed first".

Another resource of interest is "Getting the MARS Sample working in the june CTP".

Blogging on Blogging

I don't usually blog about blogging, I find it a little too self referential for my liking, but last weekend I was talking to a friend about blogs and he told me he didn't particularly like the idea. His explanation of why he didn't see blogs as a valuable social construct went along the lines that on one side of the spectrum you had articles in journals which had high peer review, but were extremely narrow in scope. Blogs were at the complete opposite end of the spectrum, extremely wide in scope, but absolutely no peer review.

This got me thinking, and caused me to re-read the article that got me into blogging in the first place. I find it strange that whenever I mention Joi Ito, or his paper "Emergent Democracy" even to active bloggers, I get blank looks. I think it's a must read for any serious blogger. In the paper Joi Ito argues for the emergent nature of blogging and related social technologies. By emergent it is taken to mean the "self organising ability of complex systems". In refutation to my friends objection to blogs, I'd like to quote directly from the paper.

"Noise in the system is suppressed, and signal amplified. Peers read the operational chatter at Mayfield's creative network layer. At the social network layer, bloggers scan the weblogs of their 150 acquaintances and pass the information they deem significant up to the political networks. The political networks have a variety of local maxima which represent yet another layer. Because of the six degrees phenomenon, it requires very few links before a globally significant item has made it to the top of the power curve. This allows a great deal of specialization and diversity to exist at the creative layer without causing disruptive noise at the political layer."

I concede that any individual blog article by any individual blogger in and of itself has not been peer reviewed, and this is the way it should be. However the blogsphere by its very nature will self organise in such a way that it will give the effect of peer review / critique. I don't think we have reached utopia yet (my entry in this years "understatement of the year competition"), and I believe that there is still a way to go, still new technologies that are required, different online social structures to be explored, but when I originally read "Emergent Democracy" I was captured by the vision.

Speaking at VS Live

Well, this one hit me out of the blue. I have been asked to speak at the Sydney VS Live conference on SQL Server Express.

My fellow colleague Greg Low dobbed me in for this one, he made the assumption that because I've been working on a commercial project with SQL Server Express since January of this year, I might actually know something about it.

It's all very exciting, but I'm a bit nervous about it all, I haven't done any public speaking for quite some time now. I have a few ideas on what I want to talk about, but I'd love some feedback as to what other people want to hear about SQL Server Express.

DB Performance tip

Normally when I think of DB Performance, I usually concerntrate on indexes, query structure, etc... all of which are quite important, but I have started listening to an msdn web cast series called "A Primer to Proper Sql Server Development" by Kimberly Trip, and in the very first episode was able to find the answer to a performance issue that had been puzzling me and the developers I'm currently working with for some months now.

The problem was that our application that is using SQL Express was having some serious performance issues. It was slow to start up, it took ages to save to the database the first few times, but then things seem to pick up a little, but not quite as much as would be ideal. Any one want to guess what the problem was????? I know all you SQL Server guru's out there have it already, but for the benefit of those who don't.... pre-allocating the size of the log file fixed the problem quite nicely.

We were already pre-allocating the size of our primary datafile to 4MB, so when we used the CREATE DATABASE ... FOR ATATCH statement, the logfile would be created by default at 25% the size of the primary datafile.... 500KB. The application is very database intensive, and filled up the log file very very quickly, and by default, the logfile would auto grow by 10% of it's original size... which would fill up very quickly again, and then auto grow by another 10% etc.... Of course this growing of the log file is quite costly, and we found that by setting the log file to be 4MB by default, with a 4MB file growth, we were able to achieve quite a significant improvement. We cut the initial loading by half, and the first few saves are now instantaneous.

If your working with SQL Server, then I'd highly recommend the web case series, there are lots of other useful tips and hints.

Pointless rant against the full sale of Telstra

As Telstras last few remaining shares are sold off at the hands of a party that really should be there to protect rural Australia from the whims of narrow minded economic rationalism, I'd like to take this opportunity (not so much as a eulogy, however apt the metaphor, more as an exercise in futility) to express a few of my (increasingly pointless) objections.

The free market, according to the economists, is meant to maximize profit. Ignoring the fact that there are no guarantees about how that profit will be distributed, there are a number of axioms that underpins this assumption. One of those axioms is that there are no externalities, or at least if there are, they are compensated. This is quite a big assumption even for some of the simplest economic transactions, but when we come to communications, this axiom doesn't even begin to stand up.

There are numerous externalities most of which are so intangible they can't even begin to be compensated. One of the externalities that most concerns me is that communications is such a valuable resource resource not only to building communities, but also to business in general. The governments 2 billion dollar trust fund is to get current services to the bush, however, being a technology professional, I know how quickly communications technologies change. Telstra are even struggling to get adequate mobile coverage to rural areas, let alone other technologies such as broard band internet. The communications technologies of tomorrow is what will drive business, and without the power to regulate the egalitarian dispersion of these technologies, certain sectors of the community that, those that a privatized company can't see immediate profits to pass on to their share holders. This I believe has the potential to seriously impact on Australia as a whole.