I recently bought a Drobo FS with a lot of storage to keep a lot of my photography and other files backed up, redundant, and available.
Even with a “small” source drive, pumping the data to the Drobo at high speed can take a while. This isn’t the fault of the Drobo, nor the network, it’s that there’s just a lot of stuff to push through the pipe.
But I ran into an odd problem, and I haven’t been able to get a good answer as to what’s happening. This is a call to geeks.
The problem is when a copy operation fails.
This could be because one rebooted the Drobo during a long copy operation, one rebooted the machine during a long copy operation, or, more likely, OS X Finder just aborted for no good reason and rather than allowing one to try the operation again, resume, or skip the problem area, the whole batch stops.
TIP: It can sometimes be better to have multiple (concurrent but blocked) copy requests that were individually initiated, than one mega-copy operation. Finder seems to like smaller sized chunks, plus if something goes wrong, there’s less to deal with.
What you end up with is a situation that’s hard to describe without seeing it. It’s a destination directory that look like this:
- The folders appear as grey.
- The folders can not be selected or opened from Finder.
- The folders do not have the little triangle icon by them.
- The folders can not be renamed from Finder, but can from Terminal.
- The folders can be copied, but they copy as grey.
- The contents of the folders can be seen using Terminal.
- If you use Finder to copy over them, it sees the name in use and makes a similarly named folder with a number after it.
- The source files are not in use.
The question is, then, how to ungrey the folders and finish the copy?
So far, I haven’t found a way.
At the moment, I’m speculating if this is related to kFirstMagicBusyFiletype, kLastMagicBusyFiletype, or kMagicBusyCreationDate as shown in the Finder.h header.
Remember, if I have to delete the directory (which can take a while — if it can be done), and then re-copy everything again (which will take a long while), and still not be certain that copy will complete, it’s a huge investment that may not pay off.
Geeks, I know what you’re thinking — I thought it too:
# ls -ableO@dFGinpqT *
863913 drwx------ 3 501 501 - 264 Feb 13 19:09:02 2011 NormalDirectory/
863912 drwx------ 3 501 501 - 264 Feb 13 19:09:13 2011 WhyIsThisGrey/
# rf -vrf badDir
?Any idea what an OSStatus error -43 is? Or why they’d be an invalid path inside the destination directory?
# stat -x *
File: "NormalDirectory"
Size: 264 FileType: Directory
Mode: (0700/drwx------) Uid: ( 501/ wls) Gid: ( 501/ wls)
Device: 45,12 Inode: 863913 Links: 3
Access: Sun Feb 13 19:09:02 2011
Modify: Sun Feb 13 19:09:02 2011
Change: Sun Feb 13 19:09:02 2011
File: "WhyIsThisGrey"
Size: 264 FileType: Directory
Mode: (0700/drwx------) Uid: ( 501/ wls) Gid: ( 501/ wls)
Device: 45,12 Inode: 863912 Links: 3
Access: Sun Feb 13 19:09:13 2011
Modify: Sun Feb 13 19:09:13 2011
Change: Sun Feb 13 19:09:13 2011
# xattr -l -v -x NormalDirectory WhyIsThisGrey
(nothing returned)
# /usr/bin/GetFileInfo -a -tcdm *
# /usr/bin/GetFileInfo -a NormalDirectory
avbstclinmedz
# /usr/bin/GetFileInfo -a WhyIsThisGrey
avbstclinmedz
# rsync --progress -aPE source destination
Further Thoughts
I found an article that suggests there’s a lot more going on with the file system than most of us give credit for. It talks a lot about the importance of meta data.
More Metadata That I First Thought
Two blog posts, The State of Backup and Cloning Tools under Mac OS X and Extended Attributes led me to playing with the xattr and mdls commands.
xattr didn’t have much interesting.
$ xattr -l SomeGreyDir
com.apple.FinderInfo:
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 |................|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020
The mdls command presented a better clue. Check out the kMDItemFSCreationDate attribute.
$ mdls SomeGreyDir
kMDItemFSContentChangeDate = 2011-02-20 14:02:19 -0500
kMDItemFSCreationDate = 1946-02-14 03:34:56 -0500
kMDItemFSCreatorCode = ""
kMDItemFSFinderFlags = 0
kMDItemFSHasCustomIcon = 0
kMDItemFSInvisible = 0
kMDItemFSIsExtensionHidden = 0
kMDItemFSIsStationery = 0
kMDItemFSLabel = 0
kMDItemFSName = "SomeGreyDir"
kMDItemFSNodeCount = 0
kMDItemFSOwnerGroupID = 501
kMDItemFSOwnerUserID = 501
kMDItemFSSize = 0
kMDItemFSTypeCode = ""
A quick romp through my incomplete folders revealed they all had a magical creation date of 1946-02-14 03:34:56 -0500.
The solution I think I need
I’m looking for a way to locate files with the kMDItemFSCreationDate
attribute set to that magic value, and then change it to whatever is in the
kMDItemFSContentChangeDate.
My suspicion is that this will let Finder, and the Apple command line utilities,
consider the file isn’t busy anymore.
There is probably a easy way to find “busy” files… for example…
In a terminal type each of these commands and hit the return key :
man mdls
man mdfind
man mdutil
But better yet, use Carbon Copy Cloner which probably will not have problems copying and if it does, logs them in detail and the author, who is a god, actually fixes them.
We are having the same problem in my office with a windows server 2003 cluster hosting our file shares. Changing the create date fixes the folders for us but we still don’t know how to prevent it in the first place. We use the setfile command to change the create date(SetFile -d mm/dd/yyyy folderpath ). Set file is part of the xcode tools so you will need to install that to use it. I used the install for 10.6 as I could not find one for 10.7. We also use a python script to find and change bad create dates. It also need xcode tools. I will paste the script below. Good luck.
#!/usr/bin/python
“””
Only runs on OSX. This utility depends on /usr/bin/GetFileInfo and
/usr/bin/SetFile, both included with XCode. This utility scans a directory for
files, checks the create date on the files, and if that date is earlier than
1970, resets the date to 04/21/1977
This is intended to fix issues with OSX and Windows SMB file shares caused
by invalid “create” dates on certain files.
Usage: fix_create_day.py
“””
import sys, getopt
import datetime
import os
import subprocess
# TODO: Check that date is valid before passing to SetFIle
# TODO: Check that GetFileInfo and SetFile exist
# TODO: Error handling for Popen calls
def check_date(path):
p = subprocess.Popen([‘/usr/bin/GetFileInfo’, ‘-d’, path.strip()],
stderr=subprocess.PIPE, stdout=subprocess.PIPE)
output, errors = p.communicate()
if errors:
print “Unable to call GetFileInfo for ” + path
print errors
return output, errors
def fix_date(path, tsf, set_date):
print ‘”{0}”,{1}’.format(path, tsf)
p = subprocess.Popen([‘/usr/bin/SetFile’, ‘-d’,
str(set_date), str(path)], stderr=subprocess.PIPE,
stdout=subprocess.PIPE)
output, errors = p.communicate()
if errors:
print errors
raise “It broke” #TODO
def main():
earliest_allowed = datetime.datetime.strptime(“01/01/1971”, “%m/%d/%Y”)
set_date = “04/21/1977” # Must be formatted as %m/%d/%Y
tocheck = [‘.’]
try:
opts, args = getopt.getopt(sys.argv[1:], “d:” )
except getopt.GetoptError, err:
print str(err) # will print something like “option -a not recognized”
usage()
sys.exit(2)
for o, a in opts:
if o == “-d”:
set_date = a
else:
assert False, “unhandled option”
if args:
tocheck = args
for file in tocheck:
for root, dirs, files in os.walk(file):
files = files + dirs
for file in files:
path = os.path.join(root,file)
output, errors = check_date(path)
if errors:
continue
if not output:
continue
try:
tsf = datetime.datetime.strptime(output.strip(),
“%m/%d/%Y %H:%M:%S”)
except:
print “Unable to parse date time for file {0}: {1}”.format(
path, output.strip())
continue
if tsf < earliest_allowed:
fix_date(path, tsf, set_date)
if __name__ == "__main__":
main()
the following fixed it for me:
SetFile -d “$(GetFileInfo -m greyDir)” greyDir
source:
http://apple.stackexchange.com/questions/49154/how-to-change-the-creation-date-time-of-a-file-in-lion
Hi,
I had a folder “tutorials” which didn’t copy complete and was hidden like yours. What I did:
1. open terminal and browse to the folder containing tutorials
2. create a new directory –
mkdir tutorials2
3. move everything from the ‘broken’ tutorials folder to the other:
mv tutorials/* tutorials2/
4. delete tutorials (doesn’t always work) – but at least you get to see your data in tutorials2 –
rm tutorials
5. (if 4. worked out) rename tutorials2 to to tutorials –
mv tutorials2 tutorials
This is exactly what I did today and it worked fine. It suggests the metadata about the partial copy was stored in the directory itself (and not in the DS_Store or anywhere else).