The Case of the Failed File Copy
The other day a friend of mine called me to tell me that he was having a problem copying pictures to a USB flash drive. He’d been able to copy over two hundred files when he got this error dialog, after which he couldn’t copy any more without getting the same message:
Unfortunately, the message, “The directory or file cannot be created”, provides no clue as to the underlying cause and the dialog explains that the error is unexpected and does not suggest where you can find the “additional help” to which it refers. My friend was sophisticated enough to make sure the drive had plenty of free space and he ran Chkdsk to check for corruption, but the scan didn’t find any problem and the error persisted on subsequent attempts to copy more files to the drive. At a loss, he turned to me.
I immediately asked him to capture a trace with Process Monitor, a real-time file system and registry monitoring tool, which would offer a look underneath the dialogs to reveal actual operating system errors returned by the file system. He sent me the resulting Process Monitor PML file, which I opened on my own system. After setting a filter for the volume in question to narrow the output to just the operations related to the file copy, I went to the end of the trace to look back for errors. I didn’t have to look far, because the last line appeared to be the operation with the error causing the dialog:
To save screen space, Process Monitor strips the “STATUS” prefix from the errors it displays, so the actual operating system error is STATUS_CANNOT_MAKE. I’d never seen or even heard of this error message. In fact, the version of Process Monitor at the time showed a raw error code, 0xc00002ea, instead of the error’s display name, and so I had to look in the Windows Device Driver Kit’s Ntstatus.h header file to find the display name and add it to the Process Monitor function that converts error codes to text.
At that point I could have cheated and searched the Windows source code for the error, but I decided to see how someone without source access would troubleshoot the problem. A Web search took me to this old thread in a newsgroup for Windows file system developers:
Sure enough, the volume was formatted with the FAT file system and the number of files on the drive, including those with long file names, could certainly have accounted for the use of all available 512 root-directory entries.
I had solved the mystery. I told my friend he had two options: he could create a subdirectory off the volume’s root and copy the remaining files into there, or he could reformat the volume with the FAT32 file system, which removes the limitation on entries in the root directory.
One question remained, however. Why was the volume formatted as FAT instead of FAT32? The answer lies with both the USB drive makers and Windows format dialog. I’m not sure what convention the makers follow, but my guess is that many format their drives with FAT simply because it’s the file system guaranteed to work on virtually any operating system, including those that don’t support FAT32, like DOS 6 and Windows 95.
As for Windows, I would have expected it to always default to FAT32, but a quick look at the Format dialog’s pick for one of my USB drives showed I was wrong:
I couldn’t find the guidelines used by the dialog anywhere on the Web, so I looked at the source and found that Windows defaults to FAT for non-CD-ROM removable volumes that are smaller than 4GB in size.
I’d consider this case closed, but I have two loose ends to follow up on: see if I can get the error message fixed so that it’s more descriptive, and lobby to get the default format changed to FAT32.
Wish me luck.
Comments
Anonymous
January 01, 2003
The comment has been removedAnonymous
January 01, 2003
The comment has been removedAnonymous
October 01, 2007
One problem with a FAT32 default for non-CD-ROM removables may be other devices which handle USB storage devices. I for once stumbled upon many devices like car stereos and cameras, which cannot cope with FAT32 and wont read them (usually results in locking up or non-secriptive "cannot read" error messages). Might this be a reason why this guideline for the format dialog has been chosen?Anonymous
October 02, 2007
Interestingly I ran into the same problem two days ago. It reminded me (it turns out now that I was correct) of DOS limitations and so I simply tried to copy the file to another directory and all was well. It good to know what exactly the problem is. And a lot many people may be facing the same problem and not realising what the solution is. Maybe MS can add this to one of their tips section or something.Anonymous
October 02, 2007
The comment has been removedAnonymous
October 02, 2007
It's been years since I've seen a problem with not having enough root directory entries available. I'll admit, I didn't guess the nature of the problem before you explained it, but I feel fairly confident that I'd have diagnosed it properly in the field. I used to see this fairly frequently when Iomega ZIP drives first became popular, because people were treating them like floppy diskettes, and because the new VFAT extensions in Windows 95 contributed to the use of root directory entries. Exhausting all of the root directory entries on a 3 1/2" floppy diskette (112-- see http://www.pcguide.com/ref/hdd/file/fatRoot-c.html) wasn't easy with MS-DOS. You'd need an average file size of about 12.71KB to do that, and although I'm sure a significant number of users saw root directory entry exhaustion on floppy diskettes, I'd imagine most floppies had full data areas before they had full root directories. With the advent of VFAT, and the larger storage capacity of ZIP disks, the conditions for root directory exhaustion became much more favorable. Users weren't familiar with making subdirectories on floppies, and they'd store tons of files in the root directory of their ZIP media. On a 100MB ZIP disk, piling on files of 200KB or less would exhaust the 512 available root directory entries. If you used long filenames, additional root directory entries would be populated with long filename data. MS-DOS and Windows 9X return a 'Disk Full' error when you run out of root directory entries on a FAT volume. I saw multiple cases where users had purchased additional ZIP media because they erroneously thought their other medias were "full", when in reality they just exhausted all the root directory entries. smile I notice that the NTFSD posting referred to plumbing this status code out in Windows XP. I find it quite humorous that Windows Vista, the present-day flagship version of the Windows operating system, returns a more arcane and less helpful message to an error condition than more than decade-old Microsoft operating systems.Anonymous
October 02, 2007
The comment has been removedAnonymous
October 02, 2007
The comment has been removedAnonymous
October 02, 2007
Obviously didn't happen this time but it can also fail if the absolute path gets too long (255? in NTFS).Anonymous
October 02, 2007
u r unbeatable Mark, love you...Anonymous
October 02, 2007
The comment has been removedAnonymous
October 02, 2007
Ha, I've seen that one coming after the first two paragraphs :-) The main problem with FAT32 is that unlike FAT16, Microsoft claims to have patents covering it and thus every manufacturer that wants to pre-format its devices with FAT32 or wants to build a car stereo/DVD player/etc. that supports FAT32 would have to pay royalties to MS. They all try to avoid that.Anonymous
October 02, 2007
The drive in question being under 500 megabytes, FAT16 seems to be a /good/ default choice anyway. Of course the users should store their files under a subdirectory, as you said. Another advantage of FAT vs. FAT32 in this case is the users could dspace it if need be - assuming win 9x of course. Why MS chose not to include a doublespace file system in 2k and over is beyond me. (Yes, I know about EFS, but it's not an answer). Is there a "3rd" party dspace add-on for 2k+ ? Regards -- NinhoAnonymous
October 02, 2007
This is similar yet different to a problem I occasionally encounter. And please contact me if you think you can help, but I can't replicate the situation on demand at the moment. I'll have to wait until I encounter it at home... Anyway, Here is the gist of it: I run as a 'User' user, not an 'Administrator'. I download a zip file containing pictures. I use 7-zip to extract the zip to .[zip name]. I browse the pictures. I then try to Cut the folder and Paste it into a directory that is symlinked from my user directory to a directory on my second hard drive. Sometimes, the Cut / Paste fails saying insufficient privileges, when I certainly DO have privileges for the entire source and destination (it comes from my desktop to something with explicitly defined permissions. I get the UAC dialogue to allow the move to which I supply my Admin password and then I am told that it cannot be done, Try Again? It's weird because it's random. Sometimes deleting the unzipped directory and then re-unzipping the folder will allow it. Sometimes copy and paste (rather than Cut and Paste) will allow it. It just seems like a mix-up to what I feel should be a straightforward task, haha.Anonymous
October 02, 2007
You were the best before you started to work at Microsoft, and you are still the best. I've never seen anyone go so deep into the Windows system since Peter Norton. As a computer technician, I find myself digging deeper in areas I know nothing about and actually learning something. I disarmed a root kit and restored the CD-ROM's functionality by researching, testing, and documenting. Then comes an article like this one where it goes into deep detail about how Windows sees a USB drive, and I learn of another tool to help me troubleshoot issues with processes. I've place this blog as a feed and I plan on reading it every week. Thanks Mark for bringing your technical expertise and determination to make the Windows OS the best it can be.Anonymous
October 02, 2007
So Mark will Windows default to exFAT after Vista SP1 ships? And is exFAT going to be available on XP?Anonymous
October 03, 2007
The comment has been removedAnonymous
October 03, 2007
Please give us the old Mark back! This is soooo uninteresting. :(Anonymous
October 03, 2007
Is there any camera/MP3 player in the market that supports FAT16 but not FAT32?Anonymous
October 03, 2007
Wow, I am a regular reader of this blog and this exact problem happened to me when I tried to put my music files in my new car stereo through the USB interface.Anonymous
October 03, 2007
The comment has been removedAnonymous
October 03, 2007
The comment has been removedAnonymous
October 04, 2007
This is most likley common knowledge to anyone reading this blog, but I often run into similar user issues when they try to copy large (2GB+) files to FAT32 formated drives. Windows reports "insufficient disk space" rather than something more accurate like "file too large."Anonymous
October 04, 2007
The comment has been removedAnonymous
October 04, 2007
Actually, I even did not have to look at the screenshot to told you what the problem may be. Come on, if it was a disk problem it would say "Cannot find the path specified". It was obvious.Anonymous
October 04, 2007
Perhaps it would be a good idea to have chkdsk print a warning message when scanning a disk with only a few directory entries left in its root.Anonymous
October 04, 2007
Those of us who still use DOS-6.22 (or 5.0) are all too familiar with the old 300-entry limit in the Root of C: and 8.3 filenames. Glad to see there are still some out there who understand these old little "quirks" :)Anonymous
October 04, 2007
Adam Ray you wrote:"I often run into similar user issues when they try to copy large (2GB+) files to FAT32 formated drives..." you know you can use fat32formatter to formate your disc is larger then 32gb in one partition. And use a tool called smartcopy (Both tools - fat32formatter, smartcopy you can get from some website) it will copy large files for you. MeiyuAnonymous
October 05, 2007
Many thanks Mark, and stay with usAnonymous
October 05, 2007
@Cheong, eric-ec2link: Yes there are quite a few, I have two of them right here on my desk (one one-year-old MP3-player) and a car stereo (bought half a year ago). Especially when you get to "low budget" devices and/or devices with little memory (less than 1GB), and a lot of customers do, you could get something without FAT32 but working fine with FAT16. Licensing issues as mentioned by Marcel may be a prime suspect ;)Anonymous
October 05, 2007
Boy, this takes me back. I knew the answer as soon as I read the error message. The last time I saw such an error was in the early '90s. It involved a 5.25" floppy. The first time was with a 10MB C: drive running DOS (3.1 I think). Having taught everyone to organize their files into subdirectories, I didn't think I'd ever see the problem again, but, hey, never say never. There's nothing new under the sun.Anonymous
October 07, 2007
An improved Error Message: R: can only hold x files, please put future files in a folder on R: or delete some files.Anonymous
October 08, 2007
The comment has been removedAnonymous
October 08, 2007
[replying to myself] Actually the problem I was referring to seems to be adressed by this hotfix, according to posts on Kasperky forums: http://support.microsoft.com/kb/942435/en-usAnonymous
October 11, 2007
Mark, TrueCrypt at www.truecrypt.org (an open source on-the fly encryption program) requires FAT for using a hidden encryption container. Evidently needs to do this to take advantage of the way FAT does things to hide things. fbigAnonymous
October 13, 2007
I saw this one coming too, having hit the old root-directory limit; more recently, I've had at least two of the users I support hitting the MAX_PATH limitation in Office. (255/260 characters sounds like a lot, but it's only 10 levels with 25-character names - and you're starting off with C:Documents and SettingsUsernameMy Documents as your 'root', too.) I can understand the API still supporting the shorter limits for legacy apps, but surely it's time for new applications to be written without this limitation?Anonymous
October 13, 2007
> "Actually, I'd go so far as saying that if you consider FAT32 instead of FAT16, you might as well go all the way and turn it into a NTFS partition!" Does someone remember Sysinternal's NTFS-Floppy-Tool? Maybe Mark could again code something like that for formatting USB-Sticks in the NTFS-Filesystem. :DAnonymous
October 18, 2007
Funny, I had the same problem with some portable media player. I even blogged about it here: http://www.ragestorm.net/blogs/?p=13Anonymous
October 28, 2007
@Dan: NTFS is actually a case-sensitive filesystem, it's Windows that treats it case-insensitively. It's touched on a bit here: http://geekswithblogs.net/quension/archive/2007/06/14/113220.aspx ...but that post assumes some knowledge already. There should be enough hints to let you search around for missing details though. It's built that way to allow for POSIX support, which is a useful feature when you need it. I would be surprised if NTFS-3g didn't actually have an option to work case insensitively somewhere, but I'm not very familiar with it. Being case sensitive is a actually good default given the environment it works in (POSIX), although I can see why it's confusing.Anonymous
October 28, 2007
>Where is NTFS-Floppy for USB-Sticks? Philipp K., you CAN format your USB stick with NTFS. Just disable write caching for that device.Anonymous
October 29, 2007
Sorry, error in my previous post. ENABLE write caching, you can disable it later, after formatting.Anonymous
November 30, 2007
A similar error can occur using USB hard drives. My client added a Western Digital MyBook 400Gb USB drive and changed to backup program to save there. Months later I get the support call when backups fail. The WD MyBook comes formatted Fat32, not NTFS for obvious compatibility reasons. When the backup file exceeded 4Gb it failed. It would have been nice if the backup program would give a clear error message, but it just said "backup failed".Anonymous
December 07, 2007
I have a camera that's about a year old, supports up to 2 GB SD cards, and only FAT16. Which doesn't waste much space, since average file sizes are 500-1200 KB, much larger than the 32KB cluster size used by FAT16. Cameras don't run into this problem very much, since most cameras automatically create a folder to store all the pictures in. Nothings ever on the root of the drive.Anonymous
December 12, 2007
For those asking about cameras that support FAT16 but not FAT32, here's one example: The Panasonic Lumix TZ1, which was declared the "Best small digital camera" by the Cool Tools blog. http://www.kk.org/cooltools/archives/001340.phpAnonymous
December 13, 2007
The comment has been removedAnonymous
December 19, 2007
The comment has been removedAnonymous
January 01, 2008
more likely companies chose fat over fat32 to minimize patent infringement. Decisions are made by lawyers not coders now days. software patents are bad.Anonymous
January 26, 2008
The comment has been removedAnonymous
January 26, 2008
Slightly more on topic, 15 years ago, the solution was to delete the last file, create a directory, move already copied files there, and continue with the rest. Moving 8Gb of pictures over a slow device link now reminds me of copying some 200 files to a 1.2Mb floppy then, and I'd think twice before restarting from zero ;-)Anonymous
February 15, 2008
Thanks, That was an easy fix after all. I originally formatted my scandisc with my camera. I re formatted it on my computer with the fat32 like you suggested and it work right away. It took about 2 minutes. Thanks, againAnonymous
February 22, 2008
Rune: Your raw files are are 16GB in size? What are you writing them to? Surely not a memory stick! The clusters under FAT32 would be 4K, not 1K, unless the device is SO small that a 16GB file would have no hope of fitting there.Anonymous
February 26, 2008
The comment has been removedAnonymous
March 03, 2008
as i know digital photocameras don't use FAT32. so the standard is FAT16/12 in many mainstream devices ... CompactFlash memories used in professional DSLR ... maybe these machines use legacy OSes inside.Anonymous
April 17, 2008
Similar issue, different error message: http://support.microsoft.com/kb/949073 Hope it helps someoneAnonymous
June 03, 2008
The comment has been removedAnonymous
June 03, 2008
BTW, for those who claim that FAT16 was used instead of FAT32 because of patent issues, you misunderstood the patents. The patents cover FAT LFN, not FAT32Anonymous
July 26, 2008
Unfortunately, MAX_PATH is not only ingrained in the API (WIN32_FIND_x structures) but also in program logic (many many existing programs allocate buffers of MAX_PATH). Today you can create paths longer than MAX_PATH (use the . syntax) but you'll quickly see how many apps fail.Anonymous
August 06, 2008
Anyone run into USB jump drives which work in one computer (Lenovo laptop WinXPHome) but NOT in another computer (Dell Dimension desktop Win XPHome) where it had run before??Anonymous
October 20, 2008
Thanks! I've been trying to get files on to my wife's digital frame all night. You saved me! BobAnonymous
October 29, 2008
Can anyone shed some light on the following problem related to file copying but not related to formatting?
- I develop software, make a build and an installer for it on my desktop (Vista Business). Everything is saved in a folder C:Customers (i.e. not in Documents). The folder is configured as shared. The installer includes the file install.msi.
- Then I copy the installer to my laptop (XP Pro) over the network, from the laptop. Install.msi does not appear in the file listing, nor is it copied. No error messages are displayed.
- Vice versa, if I copy from XP laptop to Vista desktop (driving from the desktop), I can see install.msi in the laptop file listing, but when I try to copy it, I get a message that I don't have sufficient access rights for it. (I think the same effect applies to .BIN files, but I am not sure).
- I have admin rights on both machines. On Vista, UAC is disabled. Any advice? TIA, sliberman at solidusintegration dot com
Anonymous
January 10, 2009
What is file system used for removable drives more then 4GB? I think it must be NTFS. The FAT32 file system can't create files more then 4GB. It is a critical limitation nowadays.Anonymous
January 26, 2009
OT Hallo Mark! i know that it's possible to create hardlinks using FSUTIL HARDLINK CREATE A B but if i have a datablock with 25 hardlinks ... how can i know this? a great sysinternals commandline util could be a one scanning a given path (and /S) enumerating how many hard links each file have ... with a useful /NZ (non zero, or non one) that omits those files that have no-hardlinks ... or ONE hardlink (that in fact is the "normal" state of each datablock, isn't it?) ... a GREAT Mark Russinovich util!!! :) :) thanks mark!Anonymous
February 03, 2009
I have a kind of similar problem, at least in the result: "copy failed" I have a Fat32 formated 8Gb Kingston Data Traveler, plenty of free space. I try to copy a 4.2Gb file from my HD & recieve the message "The disk in the destination drive is full". Can Someone Help Me ?!Anonymous
July 06, 2009
You must be a newbie not to guess the reason right away. Not nutured on DOS 3.3? Many older cameras, or stupid MP3 players (e.g, my first one) only understand FAT16, so it makes some sense for memory cards. But not for USB sticks! It's the reason cameras & MP3 players often create a directory in FAT16 file systems so they don't hit this limit - when a non-root directory fills, it is extended. tOMAnonymous
September 16, 2009
@Didier - I also faced the similar issue. http://www.microsoft.com/communities/newsgroups/en-us/default.aspx?dg=microsoft.public.windowsxp.newusers&tid=e166fd1d-3c63-4e0c-88d1-523a0a773476&cat=&lang=&cr=&sloc=&p=1 @Mark - as I have discussed in above forum, the error message in this case is also misleading. Can it be fixed?Anonymous
March 15, 2010
Hi Mark, I have a 500 Gig external hard drive formatted FAT32 and I have been experiencing the problem of “cannot move the directory or file cannot be created”. I have checked this drive with chkdsk and found some problems that have been fixed. Yet the files still cannot be moved. There is plenty of space, I am trying to move 15 Gig on a drive that has 90 Gig spare capacity. I can copy those files to the computer’s hard drive, but they will not copy or move front the comp’s hard drive to the external drive. Thanks for any help.Anonymous
March 31, 2010
Heres a slightly different issue. Copying approximately 10,000 mp3s (40gigs) to an USB 500gig HDD went flawlessly for 4000, then Home Premium Vista started complaining that it could no longer find the "SOURCE" files. I skipped a few errors then, still in the "Copy" mode, I used VLC to locate and play the files that Vista's Copy routine had lost track of.. Both file systems were NTFS... Any ideas? or simular experiences? I'll check back later after I look for a work-around. Thx for any info.. good luckAnonymous
May 07, 2010
I also noticed the NTFS-3g "feature" with having two files with the same name (save case sensitiveness). I copied some files from an ext3 volume to an NTFS-volume. From within Windows I noticed both files; however - using Total Commander, I didn't try Explorer - I could easily rename one of the files. I didn't want to spend time testing stuff such as "which file is actually renamed", "how important is the order of the two files" etc. Since Linux handled these two files perfectly well, I thought it was peculiar that the underlying FS actually supported this - why spend time doing that? Or was it easier accomplished implementing case insensitiveness in a higher layer?