datacoon / metawarc Goto Github PK

24.0 4.0 0.0 79 KB

metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)

License: MIT License

Python 100.00%

warc warc-files webarchiving metadata osint osint-python

metawarc's Introduction

metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)

metawarc (pronounced me-ta-warc) is a command line WARC files processing tools. Its goal is to make CLI interaction with files inside WARC archives so easy as possible. It provides a simple metawarc command that allows to extract metadata from images, documents and other files inside WARC archives.

Main features

Built-in WARC support
Metadata extraction for a lot of file formats
Low memory footprint
Documentation
Test coverage

File formats supported

MS Office OLE: .doc, .xls, .ppt
MS Office XML: .docx, .xlsx, .pptx
Adobe PDF: .pdf
Images: .png, .jpg, .tiff, .jpeg, .jp2

Installation

Any OS

A universal installation method (that works on Windows, Mac OS X, Linux, …, and always provides the latest version) is to use pip:

# Make sure we have an up-to-date version of pip and setuptools:
$ pip install --upgrade pip setuptools

$ pip install --upgrade metawarc

(If pip installation fails for some reason, you can try easy_install metawarc as a fallback.)

Python version

Python version 3.6 or greater is required.

Usage

Synopsis:

$ metawarc [command] [flags]  inputfile

See also metawarc --help and metawarc [command] --help for help for each command.

Examples

Extract metadata of all supported file types from 'digital.gov.ru.warc.gz' and output results to default filename 'metadata.jsonl':

$ metawarc metadata digital.gov.ru.warc.gz

Extract metadata for .doc and .docx file types from 'digital.gov.ru.warc.gz' and output results to default filename 'metadata.jsonl':

$ metawarc metadata --filetypes doc,docx digital.gov.ru.warc.gz

Extract metadata for .doc and .docx file types from 'digital.gov.ru.warc.gz' and output results to filename 'digital_meta.jsonl':

$ metawarc metadata --filetypes doc,docx --output digital_meta.jsonl digital.gov.ru.warc.gz

Commands

Metadata command

Extracts metadata from files inside .warc files. Returns JSON lines output for each file found.

Extract metadata for .doc and .docx file types from 'digital.gov.ru.warc.gz' and output results to filename 'digital_meta.jsonl':

$ metawarc metadata --filetypes doc,docx --output digital_meta.jsonl digital.gov.ru.warc.gz

Analyze command

Returns list of mime mimetypes with stats as number of files and total files size for each mime type. Will be merged or replaced by 'stats' command that uses sqlite db to speed up data processing

Analyzes 'digital.gov.ru.warc.gz' and output results of list of mime types as table to console

$ metawarc analyze digital.gov.ru.warc.gz

Index command

Generates 'metawarc.db' SQLite database with records HTTP metadata. Requred for 'stats' command to calculate stats quickly

Analyzes 'digital.gov.ru.warc.gz' and writes 'metawarc.db' with HTTP metadata.

$ metawarc index digital.gov.ru.warc.gz

Stats command

Same as 'analyze' command but uses 'metawarc.db' to speed up data processing. Returns total length and count of records by each mime or file extension.

Processes data in 'metawarc.db' and prints total length and count for each mime

$ metawarc stats -m mimes

Processes data in 'metawarc.db' and prints total length and count for each file extension

$ metawarc stats -m exts

Export command

Extracts HTTP headers, WARC headers or text content from WARC file and saves as NDJSON (JSON lines) data file.

Exports http headers from 'digital.gov.ru.warc.gz' and writes as 'headers.jsonl'

$ metawarc export -t headers -o headers.jsonl digital.gov.ru.warc.gz

Exports WarcIO index from 'digital.gov.ru.warc.gz' and writes as 'data.jsonl' with fields listed in '-f' option.

$ metawarc export -t warcio -f offset,length,filename,http:status,http:content-type,warc-type,warc-target-uri -o data.jsonl digital.gov.ru.warc.gz

Exports text (HTML) content from 'digital.gov.ru.warc.gz' and writes as 'content.jsonl'

$ metawarc export -t content -o content.jsonl digital.gov.ru.warc.gz

List command

Prints list of records with id, offset, length and url using 'metawarc.db'. Accepts list of mime types or list of file extensions or query as WHERE clause

Prints all records with mime type (content type) 'application/zip'

$ metawarc list -m 'application/zip'

Prints all records with file extensions 'xls' and 'xlsx'

$ metawarc list -e xls,xlsx

Prints all records with size greater than 10M and file extension 'pdf'

$ metawarc list -q 'content_length > 10000000 and ext = "pdf"'

Dump command

Dumps records payloads as files using 'metawarc.db' as WARC index. Accepts list of mime types or list of file extensions or query as WHERE clause. Adds CSV file 'records.csv' to the output directory with basic data about each dumped record.

Dumps all records with mime type (content type) 'application/zip' to 'allzip' directory

$ metawarc dump -m 'application/zip' -o allzip

Dumps all records with file extensions 'xls' and 'xlsx' to 'sheets' directory

$ metawarc dump -e xls,xlsx -o sheets

Dumps all records with size greater than 10M and file extension 'pdf' to 'bigpdf' directory

$ metawarc dump -q 'content_length > 10000000 and ext = "pdf"' -o 'bigpdf'

metawarc's People

Contributors

Stargazers

Watchers

metawarc's Issues

Error with pip installed version

metawarc metadata flashfrozen-jwat-recompressed.warc.gz

2022-04-11 12:25:39,961 - root - DEBUG - Preparing flashfrozen-jwat-recompressed.warc.gz
[warn] Skip parser 'FAT12': stream is smaller than 512.0 bytes
[warn] Skip parser 'FAT16': stream is smaller than 512.0 bytes
[warn] Skip parser 'FAT32': stream is smaller than 512.0 bytes
[warn] Skip parser 'LinuxSwapFile': stream is smaller than 4096.0 bytes
[warn] Skip parser 'MSDos_HardDrive': stream is smaller than 512.0 bytes
[warn] Skip parser 'PIFVFile': Invalid magic number
[warn] Skip parser 'ElfFile': Invalid magic
[warn] Skip parser 'MachoFatFile': stream is smaller than 4180.0 bytes
[warn] Skip parser 'MachoFile': Invalid magic
[warn] Skip parser 'PRCFile': False
[warn] Skip parser 'FAT12': stream is smaller than 512.0 bytes
[warn] Skip parser 'FAT16': stream is smaller than 512.0 bytes
[warn] Skip parser 'FAT32': stream is smaller than 512.0 bytes
[warn] Skip parser 'LinuxSwapFile': stream is smaller than 4096.0 bytes
[warn] Skip parser 'MSDos_HardDrive': stream is smaller than 512.0 bytes
[warn] Skip parser 'PIFVFile': Invalid magic number
[warn] Skip parser 'ElfFile': Invalid magic
[warn] Skip parser 'MachoFatFile': stream is smaller than 4180.0 bytes
[warn] Skip parser 'MachoFile': Invalid magic
[warn] Skip parser 'PRCFile': False
[warn] Skip parser 'FAT12': Invalid FAT12 signature
[warn] Skip parser 'FAT16': Invalid FAT16 signature
[warn] Skip parser 'FAT32': Invalid FAT32 signature
[warn] Skip parser 'LinuxSwapFile': Unknown magic string
[warn] Skip parser 'MSDos_HardDrive': Invalid signature
[warn] Skip parser 'PIFVFile': Invalid magic number
[warn] Skip parser 'ElfFile': Invalid magic
[warn] Skip parser 'MachoFatFile': Invalid magic
[warn] Skip parser 'MachoFile': Invalid magic
[warn] Skip parser 'PRCFile': False
[warn] [] Error when getting size of 'header': delete it
[warn] Skip parser 'FAT12': Invalid FAT12 signature
[warn] Skip parser 'FAT16': Invalid FAT16 signature
[warn] Skip parser 'FAT32': Invalid FAT32 signature
[warn] Skip parser 'LinuxSwapFile': Unknown magic string
[warn] Skip parser 'MSDos_HardDrive': Invalid signature
[warn] Skip parser 'PIFVFile': Invalid magic number
[warn] Skip parser 'ElfFile': Invalid magic
[warn] Skip parser 'MachoFatFile': Invalid magic
[warn] Skip parser 'MachoFile': Invalid magic
[warn] Skip parser 'PRCFile': False
[warn] [] Error when getting size of 'header': delete it
[warn] Skip parser 'FAT12': Invalid FAT12 signature
[warn] Skip parser 'FAT16': Invalid FAT16 signature
[warn] Skip parser 'FAT32': Invalid FAT32 signature
[warn] Skip parser 'LinuxSwapFile': stream is smaller than 4096.0 bytes
[warn] Skip parser 'MSDos_HardDrive': Invalid signature
[warn] Skip parser 'PIFVFile': Invalid magic number
[warn] Skip parser 'ElfFile': Invalid magic
[warn] Skip parser 'MachoFatFile': stream is smaller than 4180.0 bytes
[warn] Skip parser 'MachoFile': Invalid magic
[warn] Skip parser 'PRCFile': False
[warn] [] Error when getting size of 'header': delete it
[warn] Skip parser 'FAT12': stream is smaller than 512.0 bytes
[warn] Skip parser 'FAT16': stream is smaller than 512.0 bytes
[warn] Skip parser 'FAT32': stream is smaller than 512.0 bytes
[warn] Skip parser 'LinuxSwapFile': stream is smaller than 4096.0 bytes
[warn] Skip parser 'MSDos_HardDrive': stream is smaller than 512.0 bytes
[warn] Skip parser 'PIFVFile': Invalid magic number
[warn] Skip parser 'ElfFile': Invalid magic
[warn] Skip parser 'MachoFatFile': stream is smaller than 4180.0 bytes
[warn] Skip parser 'MachoFile': Invalid magic
[warn] Skip parser 'PRCFile': False
[warn] [] Error when getting size of 'header': delete it
[warn] Skip parser 'FAT12': Invalid FAT12 signature
[warn] Skip parser 'FAT16': Invalid FAT16 signature
[warn] Skip parser 'FAT32': Invalid FAT32 signature
[warn] Skip parser 'LinuxSwapFile': Unknown magic string
[warn] Skip parser 'MSDos_HardDrive': Invalid signature
[warn] Skip parser 'PIFVFile': Invalid magic number
[warn] Skip parser 'ElfFile': Invalid magic
[warn] Skip parser 'MachoFatFile': Invalid magic
[warn] Skip parser 'MachoFile': Invalid magic
[warn] Skip parser 'PRCFile': False
[warn] [] Error when getting size of 'header': delete it
[warn] Skip parser 'FAT12': Invalid FAT12 signature
[warn] Skip parser 'FAT16': Invalid FAT16 signature
[warn] Skip parser 'FAT32': Invalid FAT32 signature
[warn] Skip parser 'LinuxSwapFile': Unknown magic string
[warn] Skip parser 'MSDos_HardDrive': Invalid signature
[warn] Skip parser 'PIFVFile': Invalid magic number
[warn] Skip parser 'ElfFile': Invalid magic
[warn] Skip parser 'MachoFatFile': Invalid magic
[warn] Skip parser 'MachoFile': Invalid magic
[warn] Skip parser 'PRCFile': False
[warn] [] Error when getting size of 'header': delete it
[warn] Skip parser 'FAT12': Invalid FAT12 signature
[warn] Skip parser 'FAT16': Invalid FAT16 signature
[warn] Skip parser 'FAT32': Invalid FAT32 signature
[warn] Skip parser 'LinuxSwapFile': Unknown magic string
[warn] Skip parser 'MSDos_HardDrive': Invalid signature
[warn] Skip parser 'PIFVFile': Invalid magic number
[warn] Skip parser 'ElfFile': Invalid magic
[warn] Skip parser 'MachoFatFile': Invalid magic
[warn] Skip parser 'MachoFile': Invalid magic
[warn] Skip parser 'PRCFile': False
[warn] [] Error when getting size of 'header': delete it
[warn] Skip parser 'FAT12': Invalid FAT12 signature
[warn] Skip parser 'FAT16': Invalid FAT16 signature
[warn] Skip parser 'FAT32': Invalid FAT32 signature
[warn] Skip parser 'LinuxSwapFile': Unknown magic string
[warn] Skip parser 'MSDos_HardDrive': Invalid signature
[warn] Skip parser 'PIFVFile': Invalid magic number
[warn] Skip parser 'ElfFile': Invalid magic
[warn] Skip parser 'MachoFatFile': Invalid magic
[warn] Skip parser 'MachoFile': Invalid magic
[warn] Skip parser 'PRCFile': False
[warn] [] Error when getting size of 'header': delete it
[warn] Skip parser 'FAT12': Invalid FAT12 signature
[warn] Skip parser 'FAT16': Invalid FAT16 signature
[warn] Skip parser 'FAT32': Invalid FAT32 signature
[warn] Skip parser 'LinuxSwapFile': stream is smaller than 4096.0 bytes
[warn] Skip parser 'MSDos_HardDrive': Invalid signature
[warn] Skip parser 'PIFVFile': Invalid magic number
[warn] Skip parser 'ElfFile': Invalid magic
[warn] Skip parser 'MachoFatFile': stream is smaller than 4180.0 bytes
[warn] Skip parser 'MachoFile': Invalid magic
[warn] Skip parser 'PRCFile': False
[warn] [] Error when getting size of 'header': delete it
[warn] Skip parser 'FAT12': Invalid FAT12 signature
[warn] Skip parser 'FAT16': Invalid FAT16 signature
[warn] Skip parser 'FAT32': Invalid FAT32 signature
[warn] Skip parser 'LinuxSwapFile': Unknown magic string
[warn] Skip parser 'MSDos_HardDrive': Invalid signature
[warn] Skip parser 'PIFVFile': Invalid magic number
[warn] Skip parser 'ElfFile': Invalid magic
[warn] Skip parser 'MachoFatFile': Invalid magic
[warn] Skip parser 'MachoFile': Invalid magic
[warn] Skip parser 'PRCFile': False
[warn] Skip parser 'FAT12': Invalid FAT12 signature
[warn] Skip parser 'FAT16': Invalid FAT16 signature
[warn] Skip parser 'FAT32': Invalid FAT32 signature
[warn] Skip parser 'LinuxSwapFile': Unknown magic string
[warn] Skip parser 'MSDos_HardDrive': Invalid signature
[warn] Skip parser 'PIFVFile': Invalid magic number
[warn] Skip parser 'ElfFile': Invalid magic
[warn] Skip parser 'MachoFatFile': Invalid magic
[warn] Skip parser 'MachoFile': Invalid magic
[warn] Skip parser 'PRCFile': False
[warn] [] Error when getting size of 'header': delete it
[warn] Skip value width_dpi=0 (filter)
[warn] Skip value height_dpi=0 (filter)
Traceback (most recent call last):
File "/home/lepore/anaconda3/bin/metawarc", line 8, in
sys.exit(main())
File "/home/lepore/anaconda3/lib/python3.9/site-packages/metawarc/main.py", line 11, in main
exit_status = cli()
File "/home/lepore/anaconda3/lib/python3.9/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/home/lepore/anaconda3/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/lepore/anaconda3/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/lepore/anaconda3/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/lepore/anaconda3/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/lepore/anaconda3/lib/python3.9/site-packages/metawarc/core.py", line 35, in metadata
acmd.metadata(input, filetypes.split(',') if filetypes else None, fields, output=output)
File "/home/lepore/anaconda3/lib/python3.9/site-packages/metawarc/cmds/extractor.py", line 137, in metadata
result = processWarcRecord(record, url, filename, mime=h)
File "/home/lepore/anaconda3/lib/python3.9/site-packages/metawarc/cmds/extractor.py", line 87, in processWarcRecord
parser = createParser(temp.name)
File "/home/lepore/anaconda3/lib/python3.9/site-packages/hachoir/parser/guess.py", line 136, in createParser
stream = FileInputStream(filename, real_filename, tags=tags)
File "/home/lepore/anaconda3/lib/python3.9/site-packages/hachoir/stream/input_helper.py", line 38, in FileInputStream
return InputIOStream(inputio, source=source, **args)
File "/home/lepore/anaconda3/lib/python3.9/site-packages/hachoir/stream/input.py", line 412, in init
InputStream.init(self, size=size, **args)
File "/home/lepore/anaconda3/lib/python3.9/site-packages/hachoir/stream/input.py", line 136, in init
raise NullStreamError(source)
hachoir.stream.input.NullStreamError: Input size is nul (source='file:/tmp/tmpqv9quokupng')!
lepore-desktop:~/Downloads/working/warc/webarchive-discovery/warc-indexer/src/test/resources/wikipedia-mona-lisa$

NameError: name 'unknown' is not defined

$ metawarc dump -e mp3,ogg -o audio
Wrote 790a1d27-01c8-460a-968b-d2c6cf22aa31.mp3, url https://cdn.echo.msk.ru/snd/2022-03-02-osoboe-1908-3631063.mp3
...
...
Wrote f9204bd8-c662-4cff-9261-df686e6383d6.mp3, url https://cdn.echo.msk.ru/snd/2021-10-25-personalnovash-1505-sd-3551552.mp3
Traceback (most recent call last):
File "/home/me/.local/bin/metawarc", line 8, in
sys.exit(main())
File "/home/me/.local/lib/python3.10/site-packages/metawarc/main.py", line 12, in main
exit_status = cli()
File "/usr/lib/python3.10/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/usr/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/me/.local/lib/python3.10/site-packages/metawarc/core.py", line 215, in dump
acmd.dump(mimes=mimes, exts=exts, query=query, output=output)
File "/home/me/.local/lib/python3.10/site-packages/metawarc/cmds/dump.py", line 140, in dump
filename = record.warc_id + '.' + get_ext_from_content_type(record.content_type)
File "/home/me/.local/lib/python3.10/site-packages/metawarc/cmds/dump.py", line 62, in get_ext_from_content_type
return unknown
NameError: name 'unknown' is not defined

In my WARC file (Плющев выкладывал ссылку в телеграм) exist 'audio/mp3'
I add 'audio/mp3' : 'mp3', in MIME_EXT_MAP and error is disappear
Thanks

example of "bad" record in db
INSERT INTO record VALUES(3483,'920138fa-d58c-4d15-b3bb-c8521c93199b','audio/mp3',200,'https://cdn.echo.msk.ru/snd/2009-03-06-
vlast-2000.mp3','2009-03-06-vlast-2000.mp3','mp3','echo.msk.ru.warc.gz',10075372887,13434884,13991489,'"{"Server": "nginx"
, "Date": "Thu, 03 Mar 2022 08:36:36 GMT", "Content-Type": "audio/mp3", "Content-Length": "13990981", "Connection
": "keep-alive", "Last-Modified": "Fri, 29 Oct 2021 19:44:09 GMT", "x-rgw-object-type": "Normal", "ETag": "\"5
dcebd6aca8b9f6b9579dacf4ccd03ce\"", "x-amz-request-id": "tx0000000000000106f1bbf-0061e352aa-3e13cf9-dt2", "Cache": "
MISS", "X-Cached-Since": "2022-01-15T23:03:06+00:00", "Access-Control-Allow-Origin": "*", "X-ID": "rtc-up-gc5",
"Accept-Ranges": "bytes", "X-SHARD": "rtc-shard0-default_443"}"');

Feature Request

I like both of the analyze and metadata commands, however in my agency we need a detailed file listing that includes things like file name, byte size, file extension, preferably in CSV output. I understand the need for JSON output for the detailed metadata, but is there a chance for a new switch that outputs detailed information as above in CSV format? Not including the repeating fields with extracted data for the specific file types.

Thanks!

Some errors

python metawarc metadata digital.gov.ru.warc.gz
Traceback (most recent call last):
File "/home/gitpod/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/gitpod/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/workspace/metawarc/metawarc/main.py", line 18, in
main()
File "/workspace/metawarc/metawarc/main.py", line 10, in main
from .core import cli
ImportError: attempted relative import with no known parent package

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.