Can we make HDTCat work under windows? about hdt-java HOT 2 CLOSED

Hi,

sorry I just saw now the issue .... yes this is also the problem we run into when writing hdtCat. As far as I remember: We are using mapped files and this is a feature that is not done by the JVM but by the operating system. For some reasons in Windows this is not working (or at least it was not at that time). I cannot find anymore the related ticked on the JDK issue tracker, so I do not know if it was resolved. But when I saw it it was open already since many years .... Maybe there is an easy bug fix but I do not know one.

Salut Dennis

I think it is something related to this https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6359560

from hdt-java.

After fixing HDTcat to work under Windows, I was able to evaluate the initial idea (question 1 in my initial post) - to a limited extent. Result: It might be the thing to do in situations where the available memory just isn't enough to generate a HDT file in one go. However, if it is possible to generate the whole HDT file in memory, it is much faster than the map-reduce approach (map to HDT chunks, reduce chunks using HDTcat). The difference in execution time is probably due to the many more filesystem accesses required in the map-reduce approach as each triple is processed multiple times. If the system could be changed to use in-memory data structures instead of memory mapped files for smaller amounts of data (i.e., data that fit in memory), this result might change as well.

from hdt-java.