Setting up a very fast disk for hundreds of thousands of files

Started by Maris

Setting up a very fast disk for hundreds of thousands of files   20 December 2018, 22:31

Hello!

I need a very very fast disk. Each folder on the disk may contain 200-300 thousands of files. What are the best settings fro the new RAM disk in my situation?
If I set image file, will it slow down the RAM disk usage? Or can I save RAM disk only manually so it won't slow down critical operations suddenly?

Thanks.
SoftPerfect Support forum - Andrew avatar image

Re: Setting up a very fast disk for hundreds of thousands of files   21 December 2018, 01:43

I'd recommend to have an image based RAM disk formatted as exFAT, but it's difficult to tell without knowing more about your requirements. Could you please tell me more about your usage scenario for better advice:
  1. Where those files come from (i.e. created by a certain program or copied from another drive)?
  2. Is the loss of those files acceptable, or you need to preserve them all between sessions?
  3. What kind of processing is done on those files and how often do they change?
  4. How much disk space do they take?

Re: Setting up a very fast disk for hundreds of thousands of files   21 December 2018, 13:11

Thank you for the reply!

Is exFAT really faster than NTFS on 200000+ files in a folder?

My answers to the questions:
  1. The files are created by a program dynamically. Only this program needs to access the files.
  2. Loss of all at once is acceptable (because it's pretty rare I guess) but data corruption is not acceptable. Backup once a day is ok.
  3. The files will be read/written very often, simultaneously and at random. It's a kind of database which is accessed on demand via the program (over the Internet). Even more: sometimes large amount of data is requested that is contained in different files. And the responses must be as fast as possible.
  4. I guess they would take up to 8GB of space.
SoftPerfect Support forum - Andrew avatar image

Re: Setting up a very fast disk for hundreds of thousands of files   21 December 2018, 14:19

I have done a few of tests and you are right: NTFS seems to be much faster at creating large quantities of small files. Reading and deleting speeds were similar on both the file systems. Having said that, it's a good idea to place files in different folders with ~1000 files in each folder, see this and this discussion.

All in all, I recommend the following:
  1. Create a NTFS-formatted image of the required size.
  2. Create a boot-time RAM disk based on that image and tick "Save Contents to Image".
  3. Don't enable periodic saving. With the above setting all changes will be saved on shutdown.
  4. Organise your files in folders if possible, to keep about a 1000 files per folder.

Alternatively, create a volatile RAM disk without an associated image file and simply run an archiving program once a day to compress all the files into a single archive on a hard drive.

Sometimes you can get the answer faster if you try the forum search and/or have a look at the software user manual to see if your question has already been answered.

Our forum rules are simple:

  • Be polite.
  • Do not spam.
  • If possible, check your spelling and grammar.

Author:

Subject

A brief and informative title for your message, approximately 4–8 words:

     

Spam prevention: please enter the following code in the input field below.

 **     **  **    **  **    **  ********  **    ** 
 **     **  **   **    **  **   **        **   **  
 **     **  **  **      ****    **        **  **   
 **     **  *****        **     ******    *****    
 **     **  **  **       **     **        **  **   
 **     **  **   **      **     **        **   **  
  *******   **    **     **     ********  **    ** 

Message: