marcelmay / hfsa Goto Github PK

commons-io 2.8.0 release notes

• info.picocli:picocli 2.2.1 -> 3.7.0
• org.apache.hadoop:hadoop-common 2.7.5 -> 2.7.7
• org.apache.hadoop:hadoop-hdfs 2.7.5 -> 2.7.7
• com.google.guava:guava 24.0-jre -> 27.0-jre

Testing only:

• org.openjdk.jmh:jmh-core 1.20 -> 1.21
• org.openjdk.jmh:jmh-generator-annprocess 1.20 -> 1.21
• org.assertj:assertj-core 3.10.0 -> 3.11.1

Picoli release notes

Tool incorrectly reports symlinks.

Occurs when very large files lead to auto extending bucket size (see #5 ).

Support Hadoop 2.x and 3.x fsimage versions.

Hadoop 3 changed the fsimage layout when introducing new HDFS feature erasure encoding.

Eg you might only see NPE but not that there was an OoME when loading image data.

This regression was introduced when fixing #40 .

Current guava version 27.0-jre results in shade warning, which is fixed in 27.0.1-jre :

[WARNING] failureaccess-1.0.jar, guava-27.0-jre.jar define 2 overlapping classes: 
[WARNING]   - com.google.common.util.concurrent.internal.InternalFutureFailureAccess
[WARNING]   - com.google.common.util.concurrent.internal.InternalFutures

See guava issue 3302 for details.

Check if a directory contains any children.

See marcelmay/hadoop-hdfs-fsimage-exporter#22

• Provide initial map size to avoid too many resizes

Using a fsimage file, show all files and directories owned by a specified list of users

Auto-resizing is resizing exactly one bucket less than required.

https://github.com/junit-team/junit4/blob/HEAD/doc/ReleaseNotes4.13.2.md

Refactor inodes handling (storing, parsing, extracting) into a new class in order to enable future alternative implementations (inodes as single huge byte array or (direct memory) ByteBuffer etc ...)

https://assertj.github.io/doc/#assertj-core-3-19-0-release-notes

Files directly under root receive the full path including file name in FsVisitor.onFile(...).

https://fastutil.di.unimi.it/CHANGES

Micro tuning file creation from about ~9s to ~7s per 100 dirs / 26k files.

Update info.picocli:picocli from 3.7.0 to 3.9.6

• Write a micro benchmark for loading/visiting/visitingParallel for invocation time and GC overhead
• Evaluate partial parsing of INode protobuf data for fetching INode.id

Report the top small files

• by path
• by user

Simplifies test assertions.

Increase from default 8KiB to 64KiB, for FSImage files on non-SSD HDs.

Support CLI option -v for INFO or -vv for DEBUG output.

i do not find the change time and access time of the file, is there that stat?
if not , can upgrade the project to ?
thank you.

Check if value is null Instead of first testing if key exists:

if(map.contains(key)) {
    ... map.get(key);
}```

Update overall dependencies:

• com.google.guava:guava .......................... 27.1-jre -> 29.0-jre
• it.unimi.dsi:fastutil ..................................... 8.2.2 -> 8.4.1
• org.apache.hadoop:hadoop-common ...... 3.2.0 -> 3.3.0
• org.apache.hadoop:hadoop-hdfs ............. 3.2.0 -> 3.3.0
• org.apache.hadoop:hadoop-hdfs-client ... 3.2.0 -> 3.3.0
• org.slf4j:jcl-over-slf4j ................................ 1.7.26 -> 1.7.30
• org.slf4j:slf4j-api ....................................... 1.7.26 -> 1.7.30
• org.slf4j:slf4j-log4j12 ................................ 1.7.26 -> 1.7.30
• commons-io:commons-io ......................... 2.6 -> 2.7
• info.picocli:picocli ..................................... 3.9.6 -> 4.5.1

Test scope:

• org.assertj:assertj-core ................................... 3.13.2 -> 3.17.2
• org.openjdk.jmh:jmh-core ............................... 1.21 -> 1.25.2
• org.openjdk.jmh:jmh-generator-annprocess ... 1.21 -> 1.25.2
• junit:junit .......................................................... 4.12 -> 4.13

https://github.com/google/guava/releases/tag/v30.1.1

Print inode content details, for eg verification or debugging.

Support printing out the hfsa version via option -V

For consistency with java.io.File and unix, support POSIX-like double slashes in paths when querying FSImages.

Path examples:

• '//foo///bar'
• '///'

• Use fastutils specialized primitive collections, eg for the Map<Long,long[]>

For reuse, eg in https://github.com/marcelmay/hadoop-hdfs-fsimage-exporter

Print out a warning message when max JVM heap is less than 2x fsimage size and recommendation for JVM heap size.

The root INode has always the same INode ID and can be easily cached to prevent repeated reconstruction from INode bytes cache.

Memory cost for 1 million inodes are about 8 MiB (( ~ number inodes * sizof(long) ).

Add helper for handling block storage policy.

When calling computeBucketUpperBorders on a newly created instance:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
	at de.m3y.hadoop.hdfs.hfsa.util.SizeBucket$Bucket2nModel.computeBucketUpperBorders(SizeBucket.java:78)
	at de.m3y.hadoop.hdfs.hfsa.util.SizeBucket.computeBucketUpperBorders(SizeBucket.java:124)

Update com.google.guava:guava:21.0 to latest 24.0-jre

https://mail.openjdk.java.net/pipermail/jmh-dev/2021-March/003218.html
https://mail.openjdk.java.net/pipermail/jmh-dev/2021-March/003171.html
https://mail.openjdk.java.net/pipermail/jmh-dev/2021-May/003237.html
https://mail.openjdk.java.net/pipermail/jmh-dev/2021-May/003286.html

Currently, hfsa lib fails processing compressed fsimages (see marcelmay/hadoop-hdfs-fsimage-exporter/issues/34 ).

• Add test processing compressed fsimage
• Fix: Pass a default configuration instance to FSImageUtil.wrapInputStreamForCompression

Provide a CLI toggle for the fsimage generator for enabling/disabling compression.

Completely skip sections of no interest

Fine tune for smaller footprint, as Hadoop Maven dependencies (transitively) pull in a lot.