SSD "size on disk" / formatting issues

[Captain Z]

Just got my first external 2TB SSD for work and am backing up a load of test data from an old 2TB HDD. The old HDD used about 1.3TB of space, but I noticed that less than halfway through transferring the files the new SSD was almost full. When I look at the properties of a typical folder the "size on disk" is many times higher on the SSD than the HDD. For example, the same folder:

HDD:
Size: 3.83 GB
Size on disk: 3.86 GB

SDD:
Size: 3.83 GB
Size on disk: 20.0 GB

Each folder contains many thousands of individual files that are generated throughout an experiment, so I'm guessing this is the issue. Each of those files is typically 135kB in size. The SSD is formatted as exFAT, and from what I can tell online the drive was probably factory formatted so that the sector size is too large for my purposes? It's also taking a lot longer than I expected to transfer/delete files. One folder of 29,000 files has just taken 50 minutes to delete.

Have a possibly got a duff drive? Will a reformat sort it out? Will I lose any efficiency or potentially fuck something up if I reformat it to NTFS or exFAT with a smaller sector size? Help/advice appreciated.

[Zetetic]

What does

Code Select Expand

Get-Disk | Format-List (in PowerShell) tell you about the drives' sector sizes?

[Captain Z]

Possibly made it sound like my computing skills are higher than in reality. I'm not familiar with Powershell, and if I enter that command I get:

The term 'Get-Disk | Format-List' is not recognized... (message continues)

Oh, and maybe I should add, this is a Windows 7 machine.

[Dex Sawash]

Could try

Code Select Expand

getontheamphetamines.aspx

[Uncle TechTip]

You won't fuck out up with a smaller sector size, so worth a try if you can reformat without data loss. I've never had to set the size, it should be fine in default. NTFS won't hurt but it's reputed to cause more writes, which shortens the life (only marginal)

[Captain Z]

You won't fuck out up with a smaller sector size, so worth a try if you can reformat without data loss. I've never had to set the size, it should be fine in default. NTFS won't hurt but it's reputed to cause more writes, which shortens the life (only marginal)

Ta. In the end, I just went to reformat it and noticed that the allocation size was set at 1024 kB, which I'm guessing was the factory setting. I formatted as exFAT with the lowest allocation size (128 kB?) and now the data seems to be taking up a much more sensible size on disk.

[buzby]

Ta. In the end, I just went to reformat it and noticed that the allocation size was set at 1024 kB, which I'm guessing was the factory setting. I formatted as exFAT with the lowest allocation size (128 kB?) and now the data seems to be taking up a much more sensible size on disk.

I know this is an old thread, but since it's been resurrected, I'll add this for future reference. On flash memory-based media (i.e SD cards and SSDs) the manufacturers base the Allocation Unit Size on the size of a flash memory block (each flash memory chip is divided into blocks for erasing). In your 2TB SSD, due to the internal arrangement and addressing mode of the flash chips, 1024kB was chosen by the manufacturer because it probably aligned with the minimum flash block size for write and erase operations. It will likely be addressing 4 or 8 flash memory chips in parallel (depending on how they have arranged the drive's internal address bus, a 64-bit wide location would be made up of 1 word from 4 chips in parallel, or 1 byte from 8 chips in parallel) with 256kB or 128kB blocks in parallel to form that 1024kB block.

This is wasteful of storage space, but is intended to reduce the number of erase/write actions and hence prolong the drive's lifespan. Reducing the Allocation Unit Size to the exFAT default of 128kB is more efficient at using the space for storage, but will increase the number of erase & write cycles if modifying, writing or deleting files stored in each flash block (basically the block has to be read into RAM, erased and then written to each time the data in a block changes).

In your example with 135kb files, with the default 1024kB Allocation Unit Size, one file will take up one 1024kb flash block and writing or deleting it will only be one action on that flash block. Reducing the Allocation Unit Size to 128kb means you could fit 4 files into that 1024kB flash block (each 135kB file will take up two 128kB Allocation Units), but writing or deleting any one of those 4 files will result in a read-erase-write cycle on that flash block.

Large-capacity SD and SSD storage is optimised for large files that ideally do not change very often (i.e. digital video, photography or audio files). Large numbers of small (< 512kB) files is very much a worst case scenario for an SSD.

[Sebastian Cobb]

That's interesting, I thought SSD's were quite often used for storing databases on due to the fast access speed meaning you didn't have to cache so much in ram. I suppose in many cases it would be storing indexes that'd really thrash them.

Is it just that server-grade ones are built with this sort of thing in mind?

[Captain Z]

Thanks for the extra info, @buzby. With the project finished, at least 2 backups and knowing that ultimately the small files aren't too important, I will probably just end up using the SSD as my normal day-to-day portable storage and keeping the HDD as the main data backup in the cupboard.