Best settings for disk with many small files
4 posts · Closed
Hello!
I need a very very fast disk. Each folder on the disk may contain 200-300 thousands of files. What are the best settings for the new RAM disk in my situation?
If I set an image file, will it slow down the RAM disk usage? Or can I save the RAM disk only manually so it won't slow down critical operations suddenly?
Thanks.
I'd recommend to have an image based RAM disk formatted as exFAT, but it's difficult to tell without knowing more about your requirements. Could you please tell me more about your usage scenario for better advice:
- Where those files come from (i.e. created by a certain program or copied from another drive)?
- Is the loss of those files acceptable, or you need to preserve them all between sessions?
- What kind of processing is done on those files and how often do they change?
- How much disk space do they take?
Thank you for the reply!
Is exFAT really faster than NTFS on 200000+ files in a folder?
My answers to the questions:
- The files are created by a program dynamically. Only this program needs to access the files.
- Loss of all at once is acceptable (because it's pretty rare I guess) but data corruption is not acceptable. Backup once a day is ok.
- The files will be read/written very often, simultaneously and at random. It's a kind of database which is accessed on demand via the program (over the Internet). Even more: sometimes large amount of data is requested that is contained in different files. And the responses must be as fast as possible.
- I guess they would take up to 8GB of space.
I have done a few of tests and you are right: NTFS seems to be much faster at creating large quantities of small files. Reading and deleting speeds were similar on both the file systems. Having said that, it's a good idea to place files in different folders with ~1000 files in each folder, see this and this discussion.
All in all, I recommend the following:
- Create a NTFS-formatted image of the required size.
- Create a boot-time RAM disk based on that image and tick "Save Contents to Image".
- Don't enable periodic saving. With the above setting all changes will be saved on shutdown.
- Organise your files in folders if possible, to keep about a 1000 files per folder.
Alternatively, create a volatile RAM disk without an associated image file and simply run an archiving program once a day to compress all the files into a single archive on a hard drive.