This library makes reading WARC files in Java extremely easy. It is open source and compatible with the latest WARC standard (WARC-1.1).
The library is a fork of the "WARC Reader" provided by Mixnode.
These are basic usage examples for the library. When the library was created the main goal was to make something easy to use and lightweight. Because of this the examples are rather short and self-describing.
Stream a WARC file from an URL and print the payload (response body) to the console.
final URL warcUrl = new URL(
"https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2018-43/segments/1539583508988.18/warc/CC-MAIN-20181015080248-20181015101748-00000.warc.gz");
WarcRecordStreamFactory.streamOf(warcUrl)
.filter(WarcRecord::isResponse)
.map(entry -> ((ResponseContentBlock) entry.getWarcContentBlock()).getPayloadAsString())
.forEach(System.out::println);Read WARC records from a file one by one using the WarcReader class.
final WarcReader warcReader = new WarcReader(new FileInputStream(
new File("C:\\warc-test\\CC-MAIN-20180716232549-20180717012549-00001.warc.gz")));
boolean hasNext = true;
while (hasNext) {
try {
final Optional<WarcRecord> optionalWarcRecord = warcReader.readRecord();
optionalWarcRecord
.filter(WarcRecord::isResponse)
.map(warcRecord -> ((ResponseContentBlock) warcRecord.getWarcContentBlock())
.getPayloadAsString())
.ifPresent(System.out::println);
hasNext = optionalWarcRecord.isPresent();
} catch (WarcParsionException e) {
e.printStackTrace();
}
}If you want a Flux of WarcRecords you should use the reactive module like:
final URL warcUrl = new URL(
"https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2018-43/segments/1539583508988.18/warc/CC-MAIN-20181015080248-20181015101748-00000.warc.gz");
WarcRecordFluxFactory.buildWarcRecordFlux(warcUrl)
.filter(WarcRecord::isResponse)
.map(entry -> ((ResponseContentBlock) entry.getWarcContentBlock()).getPayloadAsString())
...The library is available in maven central.
You can use it with maven:
<dependency>
<groupId>com.github.bottomless-archive-project</groupId>
<artifactId>java-warc</artifactId>
<version>1.2.0</version>
</dependency>Or gradle:
implementation 'com.github.bottomless-archive-project:java-warc:1.2.0'If you want to use the reactive module use you can use it with maven:
<dependency>
<groupId>com.github.bottomless-archive-project</groupId>
<artifactId>java-warc-reactive</artifactId>
<version>1.2.0</version>
</dependency>Or gradle:
implementation 'com.github.bottomless-archive-project:java-warc-reactive:1.2.0'
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent