- Output Issue with Transcript in Powershell
-
As I mentioned in my previous blog post, there is a "bug" in the current version of Powershell with the Transcript. For "external" commands, (that is programs that aren't Powershell commands) the transcript will log the input but not output. I first encountered this when I started calling "Robocopy" instead of "Copy-Item" because of the speed and features Robocopy provides over the native "copy-item" command.
My work around is very simple. I need to capture the output anyway to determine if there is an error. So I assign the output stream from Robocopy to a variable and then simply use "Write-Output" to display it in the stream. The drawback is that the interactive user is going to have to wait until the command has completed to see the output - but again, I need to parse the output for errors - so not much I can do about that.
Here's my code snippet:
using Robocopy for speed and more power with exlusion
"copying files with robocopy with the following command: $exec"
$exec="robocopy $sourcePath $destPath /XD _config* Deploy /s"
$out = Invoke-Expression $exec
#write the log to stream
Write-Output $out
#check if there was an error
$fail = $out -match "ERROR :"
if ($fail)
{
throw("Failure in deploying ProcessLauncher Files")
}
Another solution was provided by a reader to the blog Seb from New Zealand (thanks for reading Zeb!) which he got from the Microsoft Powershell Newsgroup: which is to use:
Function Fix-Transcript ()
{
Process{
Write-Host $_
}
}
ipconfig | Fix-Transcript
Stop-Transcript
While it’s a little “noisier” I think I prefer the Write-Output as I think it is a little clearer what’s happening.
- "Logging" Script Output in Powershell
-
I've been using Powershell for deploy scripts from TFS builds at my current gig. I COULD have used MSBuild, but I chose Powershell for a number of reasons (which I'll detail in a future article on the subject). One important feature (particularly for something like deploying "bits" to production) is to save a "log" of all output from the script. In case of failure, you'll want to review the errors. For troubleshooting in the future, it's especially important to know what files were copied etc. And auditors love stuff like that...
A long time ago, in a technology far far away, I wrote database scripts for a Sybase database on UNIX using Kornshell - so I have some (albeit decade old) experience with scripting. Normally you would create the "log" by re-directing the output from the script to a file. The problem is that you have to wait for the script to finish and type a command to see the results. So I would use the "tee" command that would split the output to the console and a file.
When writing my Powershell scripts I found they have the "tee-object" so all was good. I called my script with a line like this:
.{.\_DeployHostApplication.ps1 -Env QA -WebTargetName "\\corpgp01\ARCS-Services-bin"} | tee-object $file
The problem, tee-object only copies the "success" output-stream and not errors. In Kornshell there was stdout and stderr and you could redirect with 2>& (or some such completely obscure and unreadable syntax, it's been a while) that redirects both output streams.
So doing research with the handy tool known as "Google" I looked at various approaches and found out that Powershell has a feature built in (ah those Powershell authors are so smart!) with commands "start-transcript" and "stop-transcript" that writes all output automagically to a file! This was EXACTLY what I needed!
The only issue, without error control, if a severe error occurs that causes the script to fail, the "stop-transcript" won't execute. That "transcripting" continues and everything you type at the Powershell command line continues to be written to the file. (I was getting "recursive like behavior as I was typing the log file and it would append the output, doubling the size of the log as it went on!).
But this is no problem in Powershell as it has Error Trapping built in. It's sort of like the old classic VB "On Error" (as opposed to Try-Catch) where you have a script block that executes if an error occurs. So I wrapped the stop-transcript command in an error trap and all was fine. Here is the new code:
$dt = Get-Date -format "yyyyMMdd_hhmm"
$log = ($MyInvocation.MyCommand.Name).Replace(".ps1","_$dt.log")
$file = "..\..\" + $log
start-transcript $file
trap { stop-transcript; break}
.\_DeployHostApplication.ps1 -Env QA -WebTargetName \\corpgp01\ARCS-Services-bin
stop-transcript
So here is the final script. The first three lines determine the filename I want to log to. The name will match the Powershell script name with a date and time stamp followed by “.log” instead of “.ps1” and two directory tree levels above where the script was located.
Lines 4-5 start the transcript, and uses the error trapping code to stop the transcript and then break out of the script (alternatively I could use “continue” instead of “break” to continue the DeployHostApplication script instead of breaking at the line that caused the error)
Line 6 executes my script, and line 7 will do the stop-transcript when there are no errors.
The only other issue I want to mention, “stop-transcript” only writes POWESHELL output. A subsequent blog post will show how I solved that issue…
- I'm a published author!
-
My article "An Introduction to Continuous Integration with Team Foundation Server 2008" was published in the June 2008 Edition of .NET Developer's Journal! The PDF can be viewed The URL is: http://pdf.sys-con.com/Dotnet/Walk_Spread.pdf (I think that may only be valid while it's the "current issue" and back issues require a paid subscription...)
- UTF-8 is NOT ASCII!
-
At my current client we had an interesting (and probably very common issue). The project we're on is to integrate my client's internal accounting applications (a mixture of Microsoft Dynamics 6.5 and custom applications) with "GetPaid" from Sungard. The vendor asked for a file, we asked what encoding they wanted and they stated "UTF-8". We produced the software to extract the data and run the vendor's utility code to load it - and it fails because of "bad data" in the comments field.
The comments field is a free form field for a customer. My client's intrepid QA Tester showed screen shots of the data in the Accounting System that had no issues, but did include a backwards quote (apostrophe, accent grave or whatever you want to call it). My colleague from Magenic Scott Janssens had written a general-purpose C# utility that took an XML document as input which would specify a stored procedure name and specifications to convert columns from the query to text. As instructed, Scott specified UTF-8 encoding.
Scott did some research (a cautious developer always takes into consideration THEY did something wrong). A further bit of confusion being the database in question was SQL 2000. SQL 2000 doesn't use UTF-8 but the now obsolete UCS-2. But ADO.NET makes that conversion without any issue.
Scott researched the issue and explains it as well (or better) than I can as follows:
Here's the skinny on encoding:
Firstly, UTF-8 is not ASCII. I had this wrong and I suspect SunGard is mistaken in the same way I was.
ASCII (aka ANSI, aka Plain Text) stores a character using a single byte.
UTF-8 is a Unicode encoding which stores a character using 1, 2, 3, or 4 bytes.
In UTF-8 all the ASCII characters are represented with one byte. This makes UTF-8 backwards compatible with ASCII.
The problem we're seeing is when the UTF-8 text in the database contains non-ASCII characters. In the case below, the character is a Unicode apostrophe. This is different than the ASCII apostrophe. The Unicode apostrophe is represented by three bytes. When read with the proper encoding these three bytes display as an apostrophe. When read with the ASCII encoding, each of the three bytes is displayed (because the encoding assumes each byte is a character) with whatever ASCII characters the byte values correspond to. The result is gibberish.
These values in the database are probably the result of the user cutting and pasting text from Word or Outlook.
The file we're sending Sungard is currently encoded in UTF-8. I suspect they want it in ASCII.
Depending on the needs of the project sending the data in ASCII may or may not be an issue. When converting to ASCII, any character that doesn't exist in ASCII is replaced with a question mark. So in our example below, "doesn't" becomes "doesn?t". If we need to keep the strings as they appear in our database, Sungard will need to properly read UTF-8. If we don't care about non-ASCII characters appearing as question marks, then we can send the data to Sungard in ASCII format.
The change to make the converter output ASCII is a one line change, changing the encoding object from UTF8Encoding to ASCIIEncoding.
Interesting stuff - good work Scott!
- Found a Bug in Team Build
-
I reported a problem on the MSDN TFS Build Automation Forum and Jason Pricket from Microsoft verified that it is, in fact, a bug in TFS 2008. My client previously hasn't used CI - so I setup the builds for the Application in TFS as "Daily Builds". Or specifically the build's "Trigger" is "Build Every Week on the following Days" with Monday through Friday checked. There's also a Check box for "Build even if nothing has changed" - which I have UNchecked.
The issue is that I have build automation using a task from MS Build Community Tasks that takes a version number from a Text File to ultimately update the Assembly Version, which I of course then check-back in to Version Control. Obviously, I don't want that check-in to be considered when evaluating whether "nothing has changed"
For CI, this would cause an infinite number of builds, so they added a feature where if you include "***NO_CI***" as a comment on your check-in, that check-in isn't considered when determining whether to trigger a build.
While having builds happen every night regardless of whether anything changed obviously isn't a serious of an issue as infinitely triggering builds, but this problem makes the "Build even if nothing has changed" useless - cluttering up your universe of "Builds" with unneeded ones. This could especially be an issue for Shops with lots of small applications that don't have a lot of development activity against them.
Of course - this is an argument for just turning on CI....
I reported this bug to Microsoft here. Forum post where Microsoft acknowledged it IS a bug is here
- Another Build Glitch
-
So here I am cruising along on my TFS Implementation. Got the code into version control redoing the File References in a way that will end the misery we're living with on the old system and ready to start building the Application. Instead of one big solution, I'm breaking the Application into it's "Deployable Components". For this app, they have two Web Services Projects, a Web application, a DLL for Notification Services, a Console Application and a Click-Once Deployed Windows Form Application. So I created six solutions instead of the one. I create my build types and build.
I start with one of the Web Services and the web application. I get my DLL's built quite nicely, but there's no _PublishedWebSites folder in my binaries tree (let alone on the drop site). I try to dig in and see what's wrong with my build type definition. Can't find anything. I first try Googleing on "How to build Web Applications in TFS" and just about EVERYTHING refers me to a page on MSDN that no longer exists (gotta love that). I keep digging and I'm SURE I'm doing everything correctly.
So I get frustrated and post the problem to the MSDN Forum. That helps me focus - and I decide the next step is to build a "Hello World" application. I build a Web Application with a single page that says "Hello World" and an <ASP:Label> where I pass in the date. Add it to Source Control, setup a build type, build it, and yup - I get the proverbial _PublishedWebsites folder nicely organized to deploy to a server.
Then I did some more Googleing and ran across This blog post from Donn Felker, which I had scanned past and initially didn't realize was my problem. So I open up the .csproj for Hello World - and yup there is a line in there for the Microsoft.WebApplication.targets. I then opened up the csproj for the application I'm migrating from VSS and unlike Don's where the line was commented out, in my projects, it was missing entirely.
So I edited the file, inserted the lines - and presto - I get a _publilshedWebsites folder!
So thanks Donn for blogging about your solution - saved me some serious time!
Now to move on to ClickOnce publishing....
- Keep those Versions Straight!
-
I had been pulled away from the TFS project for a week to help out on another and finally was able to get back to TFS. My client has a Framework they wrote that is a mix of CSLA, Microsoft Enterprise Library and custom code. I have that building successfully so I had moved on to the Application itself. I made some extensive changes to how the referenced DLL's work (including the aforementioned Framework) - so I did that all manually after removing the bindings to source safe.
So I go ahead and add the solution to TFS Source Control and create a build type and BOOM I get a very bizarre error:
Target "CoreCompileSolution" in file "C:\Program Files\MSBuild\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets" from project "c:\TFSBuild\5\BuildType\TFSBuild.proj":
Using "WriteLinesToFile" task from assembly "Microsoft.Build.Tasks.v3.5, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
Task "WriteLinesToFile"
C:\Program Files\MSBuild\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets(978,5): error MSB3491: Could not write lines to file "c:\TFSBuild\5\Sources\Steriworks\Main\Source\SteriWorks.sln.Debug.vsprops". Could not find a part of the path 'c:\TFSBuild\5\Sources\Steriworks\Main\Source\SteriWorks.sln.Debug.vsprops'.
Done executing task "WriteLinesToFile" -- FAILED.
Googleing the problem brought me to this blog post by our friendly neighborhood Magenic TFS guru Steve St Jean: which explained the error - but a symptom caused by different reason. What's going on here is that the Build was trying to create a vsprops file in the solution directory - but this directory is read-only (needless to say - coming straight from the Source Control) so it "blowed up real good". But the question was why?
Digging into the "C:\Program Files\MSBuild\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets(978,5): " was even stranger. Here it is:
<!-- Generate the VCOverride file for C++ projects -->
<WriteLinesToFile
File="$(VsPropsFile)"
Lines="$(VCOverridesOpen) OutputDirectory=%22$(OutDir)%22%3E%0D%0A$(VCOverridesCodeAnalysis)$(AdditionalVCOverrides)$(VCOverridesClose)"
Overwrite="true" />
Well, my project isn't a C++ project and furthermore the variable there refers to Code Analysis which I'm not using (at the moment at least).
So I decide to branch the code (Ah such TFS goodness - that's something always to be avoided in VSS), start whacking parts of the Solution to find out what's causing it. I start to do that and then go to create a new build type when I realize:
I had created the build type in Visual Studio 2005!
My client is still using VSTS2005 for their development. Several of the developers are still on Windows 2000 (they skipped XP and are migrating from Win2K to Vista) and haven't decided when they want to take the plunge to 2008. But since we started the TFS project after Team System 2008 was RTM we decided to use the 2008 Version of TFS.
So when I am working on the source code in Visual Studio - I need to use VS2005 - but when I do some customization for TFS - I need to use 2008. However the Icons are the same - (I changed on my desktop - but on the Task bar - they are the same) - so it can be tricky keeping which one is straight. I knew that you couldn't "queue" a TFS2008 build from VS2005 - and I never intended to create the build type in VS2005 - but I did Obviously, this is another incomparability between 2005 and 2008 as far as creating Build Types are concerned.
So I deleted the Build Type, recreated in Visual Studio 2008 and presto! The error went away!
Now to get the build to work! (It's broken - but at least it's errors I expect to get at this point in the process)
What's going on here is that the Build was trying to create a vsprops file in the solution directory - but this directory is read-only (needless to say - coming straight from the Source Control) so it "blowed up real good". But the question was why?
Digging into the "C:\Program Files\MSBuild\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets(978,5): " was even stranger. Here it is:
<!-- Generate the VCOverride file for C++ projects -->
<WriteLinesToFile
File="$(VsPropsFile)"
Lines="$(VCOverridesOpen) OutputDirectory=%22$(OutDir)%22%3E%0D%0A$(VCOverridesCodeAnalysis)$(AdditionalVCOverrides)$(VCOverridesClose)"
Overwrite="true" />
Well, my project isn't a C++ project and furthermore the variable there refers to Code Analysis which I'm not using (at the moment at least).
So I decide to branch the code (Ah such TFS goodness - that's something always to be avoided in VSS), start whacking parts of the Solution to find out what's causing it. I start to do that and then go to create a new build type when I realize:
I had created the build type in Visual Studio 2005!
My client is still using VSTS2005 for their development. Several of the developers are still on Windows 2000 (they skipped XP and are migrating from Win2K to Vista) and haven't decided when they want to take the plunge to 2008. But since we started the TFS project after Team System 2008 was RTM we decided to use the 2008 Version of TFS.
So when I am working on the source code in Visual Studio - I need to use VS2005 - but when I do some customization for TFS - I need to use 2008. However the Icons are the same - (I changed on my desktop - but on the Task bar - they are the same) - so it can be tricky keeping which one is straight. I knew that you couldn't "queue" a TFS2008 build from VS2005 - and I never intended to create the build type in VS2005 - but I did Obviously, this is another incomparability between 2005 and 2008 as far as creating Build Types are concerned.
So I deleted the Build Type, recreated in Visual Studio 2008 and presto! The error went away!
Now to get the build to work! (It's broken - but at least it's errors I expect to get at this point in the process)
- It’s the Process, Stupid!
-
The most important to keep in mind about Team Foundation Server is that it’s all about improving the software development PROCESS. Yes, it’s great to get rid of Visual Source Safe, to automate builds, and get a lot of nice reports—all easily accessed and integrated in Visual Studio, Office, and so on. Developers love shiny new toys and TFS’s tooling is super cool. But the key thing is to not lose track of the big picture–and in this case, it is the process.
Back in early December, the first thing we did on my current project was to bring in Steve St. Jean from Magenic’s Boston office. Steve is a VSTS MVP and knows the product better than anyone I’ve ever met. We were able to get the client’s key players—the director, applications manager, architect, and change control manager—to dedicate most of their time to the project during Steve’s four-day visit. While the putative purpose of Steve’s visit was to determine how the product could best be used by the client, what REALLY happened was that we put the client’s software development process under a microscope and determined the “why’s” behind what they are doing.
As I mentioned in my previous post, the client has a very tightly integrated environment that extends all the way to deploying the “bits” to production servers. They do a lot of reporting, SOX compliance, and change control in addition to the typical task of building the applications.
The client is very excited about this new tool, but the classic trade-offs exist. Over and over, we had to ask, “Do you want to modify the tool for the process, or the process for the tool?” While TFS is clearly designed to be customized, a) it’s still a relatively new product and b) not EVERYTHING can be customized.
So these questions remained: “What is the process? Why do you do things the way you do?” In many cases there are things that MUST be done and for good reasons. Other things were perhaps done to match their previous tools. But while talking about tools, automation, builds, work items, and reports, it’s important to remember those are just trees; the forest is the development process and it’s critical not to lose sight of that.
At the end of those four days, everyone was VERY satisfied at what we accomplished. We ended up with a better understanding of the process, which gave us a firm grip on where we want to go with the tool.
I STRONGLY recommend that any company considering adopting TFS to go through the exercise of examining their process, really determining needs vs. wants vs. nice-to-haves. It isn’t easy for key managers to dedicate a significant part of a week to such deliberations, but the alternative is a set-up for failure.
- A new Magenic Blogger!
-
I’ve hesitated to blog (particularly on this site) for fear of being “Just another tech guy saying the same old stuff”… There are too many of those types of blogs out there and I've always felt that unless I have something to add that may be of use to others, I don’t want yet another exercise of someone "talking to hear themselves speak".
In early December I began a project where I get to do something completely from scratch, using a relatively new technology that offers a lot of promise. My current client (where I have been consulting since June of 2006) decided to adopt Team Foundation Server. I am “heading up” the project to implement it. (I put that in quotes since for the most part I'm working solo!)
They currently have a very tightly integrated Change Control/Build/Deploy system that is a combination of a purchased application (Visual Intercept from Elsinore Technologies), open source tools (Cruise Control, nAnt) and a LOT of custom code. TFS is a great product with a bright future, but it is still very new – so I think my experiences will be an interesting topic to blog about. This project gives me the opportunity to avoid that feeling that this blog would be a self-indulgent exercise.
That said I’d like to give a little background on myself. I’ve been in this business since 1984, after receiving a BA in Liberal Arts from the University of Illinois at Urbana-Champaign. I Started as an IBM Mainframe/Cobol Developer until the early 1990’s when I made the transition to “Client Server” Technologies using Visual Basic 3.0 on Windows 3.0 client-side, and Sybase SQL Server on AIX Unix server-side. I’ve focused on Microsoft Technologies since then, happily making the jump to .NET in 2002.
In August 2004 I was hired by ESS just before Magenic acquired the firm to become the Magenic Chicago office. I am happy to report it’s the best job I’ve ever had.
In case you’re wondering about the Blog’s Title; I’m also a professional musician - I’ve been playing trombone since fifth grade. I took advantage of the dot-com boom years by doing part-time consulting from home while pursuing a music-career full time. That was also during the “Swing” Craze and the Jump Blues band I played with at the time (initially “The Big Swing” which renamed itself to “The Vanguard Aces” after a management dispute) was very successful.
In 1998, during a tour of Europe, that band’s drummer came up with the idea that I should hook up a guitar fuzz-pedal to my trombone (which I STILL haven’t tried!) and form a band called “Fuzzbone” – which stuck as my nickname. Some musicians in Chicago have forgotten my real name and just know me as “Fuzzbone” or “Fuzzy”. So I figured “Fuzzy’s Blogic” would make a good title for the blog.