OS X: Incomplete and partial files

Grey folders inaccessible from Finder? OSStatus error -43? Incomplete or partial folders? Here’s what I think is causing them and quite possibly what you can do about them.

I recently bought a Drobo FS with a lot of storage to keep a lot of my photography and other files backed up, redundant, and available.

Even with a “small” source drive, pumping the data to the Drobo at high speed can take a while. This isn’t the fault of the Drobo, nor the network, it’s that there’s just a lot of stuff to push through the pipe.

About 25 Hours to Go

But I ran into an odd problem, and I haven’t been able to get a good answer as to what’s happening. This is a call to geeks.

The problem is when a copy operation fails.

This could be because one rebooted the Drobo during a long copy operation, one rebooted the machine during a long copy operation, or, more likely, OS X Finder just aborted for no good reason and rather than allowing one to try the operation again, resume, or skip the problem area, the whole batch stops.

TIP: It can sometimes be better to have multiple (concurrent but blocked) copy requests that were individually initiated, than one mega-copy operation. Finder seems to like smaller sized chunks, plus if something goes wrong, there’s less to deal with.

What you end up with is a situation that’s hard to describe without seeing it. It’s a destination directory that look like this:

Incomplete Files and Folders
  1. The folders appear as grey.
  2. The folders can not be selected or opened from Finder.
  3. The folders do not have the little triangle icon by them.
  4. The folders can not be renamed from Finder, but can from Terminal.
  5. The folders can be copied, but they copy as grey.
  6. The contents of the folders can be seen using Terminal.
  7. If you use Finder to copy over them, it sees the name in use and makes a similarly named folder with a number after it.
  8. The source files are not in use.

The question is, then, how to ungrey the folders and finish the copy?

So far, I haven’t found a way.

At the moment, I’m speculating if this is related to kFirstMagicBusyFiletype, kLastMagicBusyFiletype, or kMagicBusyCreationDate as shown in the Finder.h header.

Remember, if I have to delete the directory (which can take a while — if it can be done), and then re-copy everything again (which will take a long while), and still not be certain that copy will complete, it’s a huge investment that may not pay off.

Geeks, I know what you’re thinking — I thought it too:

Was the source drive clean?
Yes. I do a Disk Utility check on my source volumes before copying from them.
Was the destination drive clean?
Yes. I do a Disk Utility check on my destination volumes before copying to them. In the case of the Drobo, I had just formatted it, using the latest firmware, and its dashboard gave the all clear. I even peeked with their sshd application.
Did you check the file permissions?
They’re clean with the regular 700 permissions. Finder’s Info concurs.
# ls -ableO@dFGinpqT *
863913 drwx------ 3 501 501 - 264 Feb 13 19:09:02 2011 NormalDirectory/
863912 drwx------ 3 501 501 - 264 Feb 13 19:09:13 2011 WhyIsThisGrey/
What happens if you delete them with # rf -vrf badDir?
Sometimes that works, actually. Other times the delete command just hangs indefinitely, like the file is busy.That said, using the Path Finder shell, this is what you get if you attempt to delete a directory that’s acting up.

OSStatus Error -43

Any idea what an OSStatus error -43 is? Or why they’d be an invalid path inside the destination directory?

What about extended attributes? Type? Creator?
They’re clean, too. Verfied by Path Finder, stat, xattr, and /usr/bin/GetFileInfo.
# stat -x *
File: "NormalDirectory"
Size: 264 FileType: Directory
Mode: (0700/drwx------) Uid: ( 501/ wls) Gid: ( 501/ wls)
Device: 45,12 Inode: 863913 Links: 3
Access: Sun Feb 13 19:09:02 2011
Modify: Sun Feb 13 19:09:02 2011
Change: Sun Feb 13 19:09:02 2011
File: "WhyIsThisGrey"
Size: 264 FileType: Directory
Mode: (0700/drwx------) Uid: ( 501/ wls) Gid: ( 501/ wls)
Device: 45,12 Inode: 863912 Links: 3
Access: Sun Feb 13 19:09:13 2011
Modify: Sun Feb 13 19:09:13 2011
Change: Sun Feb 13 19:09:13 2011
# xattr -l -v -x NormalDirectory WhyIsThisGrey
(nothing returned)
# /usr/bin/GetFileInfo -a -tcdm *
# /usr/bin/GetFileInfo -a NormalDirectory
avbstclinmedz
# /usr/bin/GetFileInfo -a WhyIsThisGrey
avbstclinmedz

No Extended Attributes, according to Path Finder
What happens if you rsync?
Doing an rsync appears to work, but doing it to the “broken directory” does not fix it after it completes.
# rsync --progress -aPE source destination

Further Thoughts

I found an article that suggests there’s a lot more going on with the file system than most of us give credit for. It talks a lot about the importance of meta data.

More Metadata That I First Thought

Two blog posts, The State of Backup and Cloning Tools under Mac OS X and Extended Attributes led me to playing with the xattr and mdls commands.

xattr didn’t have much interesting.

$ xattr -l SomeGreyDir
com.apple.FinderInfo:
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 |................|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020

The mdls command presented a better clue. Check out the kMDItemFSCreationDate attribute.

$ mdls SomeGreyDir
kMDItemFSContentChangeDate = 2011-02-20 14:02:19 -0500
kMDItemFSCreationDate = 1946-02-14 03:34:56 -0500
kMDItemFSCreatorCode = ""
kMDItemFSFinderFlags = 0
kMDItemFSHasCustomIcon = 0
kMDItemFSInvisible = 0
kMDItemFSIsExtensionHidden = 0
kMDItemFSIsStationery = 0
kMDItemFSLabel = 0
kMDItemFSName = "SomeGreyDir"
kMDItemFSNodeCount = 0
kMDItemFSOwnerGroupID = 501
kMDItemFSOwnerUserID = 501
kMDItemFSSize = 0
kMDItemFSTypeCode = ""

A quick romp through my incomplete folders revealed they all had a magical creation date of 1946-02-14 03:34:56 -0500.

The solution I think I need

I’m looking for a way to locate files with the kMDItemFSCreationDate
attribute set to that magic value, and then change it to whatever is in the
kMDItemFSContentChangeDate.

My suspicion is that this will let Finder, and the Apple command line utilities,
consider the file isn’t busy anymore.

5 thoughts on “OS X: Incomplete and partial files”

  1. There is probably a easy way to find “busy” files… for example…
    In a terminal type each of these commands and hit the return key :
    man mdls
    man mdfind
    man mdutil

    But better yet, use Carbon Copy Cloner which probably will not have problems copying and if it does, logs them in detail and the author, who is a god, actually fixes them.

  2. We are having the same problem in my office with a windows server 2003 cluster hosting our file shares. Changing the create date fixes the folders for us but we still don’t know how to prevent it in the first place. We use the setfile command to change the create date(SetFile -d mm/dd/yyyy folderpath ). Set file is part of the xcode tools so you will need to install that to use it. I used the install for 10.6 as I could not find one for 10.7. We also use a python script to find and change bad create dates. It also need xcode tools. I will paste the script below. Good luck.

    #!/usr/bin/python

    “””
    Only runs on OSX. This utility depends on /usr/bin/GetFileInfo and
    /usr/bin/SetFile, both included with XCode. This utility scans a directory for
    files, checks the create date on the files, and if that date is earlier than
    1970, resets the date to 04/21/1977

    This is intended to fix issues with OSX and Windows SMB file shares caused
    by invalid “create” dates on certain files.

    Usage: fix_create_day.py
    “””

    import sys, getopt
    import datetime
    import os
    import subprocess

    # TODO: Check that date is valid before passing to SetFIle
    # TODO: Check that GetFileInfo and SetFile exist
    # TODO: Error handling for Popen calls

    def check_date(path):
    p = subprocess.Popen([‘/usr/bin/GetFileInfo’, ‘-d’, path.strip()],
    stderr=subprocess.PIPE, stdout=subprocess.PIPE)
    output, errors = p.communicate()
    if errors:
    print “Unable to call GetFileInfo for ” + path
    print errors
    return output, errors

    def fix_date(path, tsf, set_date):
    print ‘”{0}”,{1}’.format(path, tsf)
    p = subprocess.Popen([‘/usr/bin/SetFile’, ‘-d’,
    str(set_date), str(path)], stderr=subprocess.PIPE,
    stdout=subprocess.PIPE)
    output, errors = p.communicate()
    if errors:
    print errors
    raise “It broke” #TODO

    def main():
    earliest_allowed = datetime.datetime.strptime(“01/01/1971”, “%m/%d/%Y”)
    set_date = “04/21/1977” # Must be formatted as %m/%d/%Y
    tocheck = [‘.’]
    try:
    opts, args = getopt.getopt(sys.argv[1:], “d:” )
    except getopt.GetoptError, err:
    print str(err) # will print something like “option -a not recognized”
    usage()
    sys.exit(2)
    for o, a in opts:
    if o == “-d”:
    set_date = a
    else:
    assert False, “unhandled option”
    if args:
    tocheck = args

    for file in tocheck:
    for root, dirs, files in os.walk(file):
    files = files + dirs
    for file in files:
    path = os.path.join(root,file)
    output, errors = check_date(path)
    if errors:
    continue
    if not output:
    continue
    try:
    tsf = datetime.datetime.strptime(output.strip(),
    “%m/%d/%Y %H:%M:%S”)
    except:
    print “Unable to parse date time for file {0}: {1}”.format(
    path, output.strip())
    continue
    if tsf < earliest_allowed:
    fix_date(path, tsf, set_date)

    if __name__ == "__main__":
    main()

  3. Hi,

    I had a folder “tutorials” which didn’t copy complete and was hidden like yours. What I did:
    1. open terminal and browse to the folder containing tutorials
    2. create a new directory – mkdir tutorials2
    3. move everything from the ‘broken’ tutorials folder to the other: mv tutorials/* tutorials2/
    4. delete tutorials (doesn’t always work) – but at least you get to see your data in tutorials2 – rm tutorials
    5. (if 4. worked out) rename tutorials2 to to tutorials – mv tutorials2 tutorials

    1. This is exactly what I did today and it worked fine. It suggests the metadata about the partial copy was stored in the directory itself (and not in the DS_Store or anywhere else).

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.