This morning I came in to work and discovered my Windows XP desktop in a crashed state, you know the one, the Blue Screen of Death; the same one you see billboard sized at Times Square.
Given that I’m meticulous about patches, clean registry settings, and an army of spyware, malware, and anti-virus detectors, not to mention the machine is used for very limited purposes, it’s very likely this isn’t some bad 3rd party Windows driver. Oh, no, the error message squarely put the blame on the USB driver.
Knowing that, I can think back to what my very last activities were at the end of the day. I saved a file in a simple editor, that file was on my Dell USB stick, and after it saved, I initiated a Windows Reboot, and pulled my USB stick (whose activity light was well extinguished) and walked at the door as Windows was still shutting down.
I’m going to simply conclude that Windows was so “busy” with its shutdown that it didn’t “see” the USB device get removed, and it was left in some horrified state that it had to die (something that does not happen with my Mac, ever). This is further confirmed by the fact that, after a hard power reset, XP came up fine, and all of my diagnostic utilities passed. Windows had just, plain and simply, died.
Sometime after booting, however, I got a message that Windows had detected it had shutdown in a bad manner, and it wanted to know if it was okay to send the report to Microsoft. I’m all for making things better, but I thought it might be interesting to look into the post-Blue Screen of Death activities.
The Blue Screen of Death did a crash dump and some files were written to disk in a directory called C:\Documents and Settings\{username}\Local Settings\Temp\WEReeed.dir00.
The file manifest.txt consisted of name/value pairs separated by an equal sign, in much the same way as the contents of an .ini file might be done, sans section headers.
The more curious contents of this file revealed the server, a url, and some values, what data files were being sent, and a very obscure reference to what might be a “blue” screen report.
Server=watson.microsoft.com
Stage2URL=/dw/bluetwo.asp?BCCode=1000007e&BCP1=C0000005&
BCP2=BA2C4371&BCP3=BA503AF4&BCP4=BA5037F0&
OSVer=5_1_2600&SP=2_0&Product=256_1
DataFiles=C:\DOCUME~1\{username}\LOCALS~1\Temp\
WEReeed.dir00\Mini022207-01.dmp|C:\DOCUME~1\{username}\
LOCALS~1\Temp\WEReeed.dir00\sysdata.xml
ErrorSubPath=blue
The sysdata.xml file consisted of an XML file that listed every device, its description, hardware id, service, and driver, often the version and file size as well. Sure enough, the usehub.sys file was there, buried in the batch. It simply appears this file is trying to collect the configuration of the machine, perhaps to recreate it in the lab for some regression testing and battery of comprehensively abusive test suites. At least that’s what I would hope happens.
The Mini022207-01.dump is clearly the month/day/year-sequence_number of when the dump was made. When the Blue Screen of Death happened, it claimed it was dumping all of physical memory. Given this Mini-Dump is only 92K, some post-processing has clearly taken place.
In my case, the file was clearly a page dump of a section of memory, with what looked like uninitialized memory labeled with the bytes literally reading “PAGE”. Inside, this binary blob it was very easy to make out pgfilter.sys, USBSTOR.SYS, and kmixer.sys. Other device driver names and binary glop followed.
Actually submitting the report showed that watson.microsoft.com (as in the product Dr. Watson) was queried and an IP of 65.54.206.43 came back. An https: exchange was made, and moments later oca.partners.extranet.microsoft.com (131.107.112.111) was ask of the DNS server; more content was sent to that server. wwwbaytest5.microsoft.com (207.46.18.30) was then asked for a certificate, via GET /pki/mscorp/Microsoft%20Secure%20Sever%20Authority(3).crt; a few more of these went back and forth, and wer.microsoft.com (131.107.115.67) got involved, that when my browser reported the human readable response to the report. Compounding matters, no tracking number or email address is provided, so even if I wanted to provide Microsoft with more information to help them fix the problem, I can’t.
After all this happened another thought struck me… Microsoft doesn’t really have a good track record with security, especially when it comes to error checking and services that aren’t used that much. I ponder what would have happened if the information had been tampered with before being sent? Is there invalid input that could send the error reporting systems into a tizzy? Could some bogus changes make their debugger or tool execute malicious code? Would some false data send some poor analysis team chasing fictional ghosts? What would happen if an automated script kiddie generated millions of bogus machine crash reports; how would they get sorted out?
I ask the question because there are quite a number of phone-home-if-you-see-a-problem systems out there in popular open source projects. Seems to me that there should be solid secure conventions to detect if error report data has been tampered with, or is bogus, and to prevent the same kinds of attacks regular systems suffer from. This is something worth spending some design time on, even if it isn’t part of the main product functionality.
Update: Suffered another crash, this time in the ATI driver as the system was doing nothing and changing focus from one window to another. Oddly enough, again, all the diagnostics say the system is fine — I’m going to do a very intensive sweep.
For the curious, the new directory was WERdb4a.dir00 with similar manifest, dump, and sysdata files. WER is the Windows Error Report, and the stuff after it appears to be hex glop. This time it is blaming the video driver, so I’ll be checking if there are any updates with both Dell and ATI.
Hey Walt –
The last 6 PC problems I’ve had which at the surface appeared to be Windows errors (blue screens, unexplained reboots, weird application behavior, registry corruption) were chased down to be hardware problems. 2 cases of faulty memory, 1 bad motherboard, 1 failing hard drive, 1 failing video card, and 1 case of too much dog fur clogging system fans. I built all these systems with ‘supposed’ quality hardware which was thankfully under warrenty. So, you may check that avenue. Memory is an easy check… http://www.memtest86.com
Ever since I’ve put all non-battery powered components having hard drives on UPSs (yes, even my TiVo), I’ve rarely had problems.
Good luck!
That’s the general rule I follow when running Linux and OS X systems, if something is wrong, it’s hardware. In this case, however, everything was working wonderfully until the lastest batch of massive Windows updates, which by the way had the known bad IDE driver problem, which caused a blue screen of death.
I ran disk check, Registry Mechanic, the spyware, malware, and several antivirus checks – all clean. I did update my video driver, though, Dell claimed it knew about a 2005 version, while the ATI site knew about a Dec 2006 version.
Listen, I dont know a whole lot about this stuff but I have a feeling that Window Vista is the problem. I got this stupid bue screen and the same files as you guys. I use Windows XP and this all started after I allowed Windows to update somestuff for me. I have a feeling that it installed some Vista updates. Look in the temp files and see if there are a bunch of fies that say oscheck vista migration. I have a bunch of these and the times they were created were the times that I restarted my computer from that stupid blue screen. Any input?
i have been getting a “Probelm Reports and Solutions” notice from this address: 131.107.115.67. is it safe to allow this communication through my firewall?
I’d say go to read this article about WER, and then decide.
I don’t think there’s any security problems by doing so, if that’s what you mean.