Fast 1TB swap file for generation about tb HOT 5 CLOSED

@mstembera
Sorry for not responding earlier. Somehow I missed the notification of the opening of this issue.

It seems Optane drives don't wear out nearly as quickly as regular flash drives, and they are faster, so it might be acceptable to use them as swap to get the 1TB virtual RAM necessary for 7-men generation. Generation of a pawnless 7-men TB with the current generator (with a few adaptations to allow 7 men) would require 2 x 462 x 64^5 bytes, which is 924 GiB or 992 GB.

It would make sense though to rewrite the generator before seriously embarking on 7-men generation. Most 7-men pawnless piece combinations have at least two pieces of the same type and color, which can be used to approximately halve the required memory. For 6-piece in-RAM generation it made sense to ignore this (keeping index calculations simpler and faster and total generation time still very acceptable), but for 7-piece generation the situation is different. Also the compression algorithm might have to be revised for greater speed.

from tb.

https://www.anandtech.com/show/12163/spotted-15-tb-intel-optane-ssd-900p
So if I got the 960gb optane with my 32 threads and 128gb ram it might work.
I read that ddr3-2133 gets 3 to 4 million iops and optane gets 400k. So generation would take 10x the time roughly. I am curious what queue depth we could get, since 32 threads should get a queue depth of 32 which is enough that RAID should boost performance.

from tb.

@syzygy1 Thank you for a very informative reply. It looks like Intel might even make DIMMS using 3D Xpoint which I would assume to be very large capacity per $.
https://www.anandtech.com/show/12041/intel-to-launch-3d-xpoint-dimms-in-2h-2018
I will close this since it was simply a question and not an issue.

from tb.

from tb.

My attempt to generate has started, if it went south may I use this thread to communicate?

I also plan to host direct downloads for all generated files.

from tb.