mediaarea / mediaconch Goto Github PK

MediaConch (funded by PREFORMA)

JavaScript 1.57% CSS 0.95% HTML 3.44% Shell 0.62% C++ 83.40% C 1.10% NSIS 0.29% Pascal 0.12% Makefile 0.28% CMake 0.28% C# 0.57% Java 0.91% AutoIt 0.05% Python 0.41% PureBasic 0.02% PHP 1.57% Batchfile 0.74% XSLT 2.62% M4 0.89% SCSS 0.16%

mediaconch's Introduction

MediaConch README

MediaConch is an open source software project consisting of a toolset that aims to further develop the standardization and validation of preservation-level audiovisual files used within various memory institutions and communities. The software consists of an implementation checker, policy checker, reporter and fixer that will work together to offer its users an advanced level of ability to validate, assess and find solutions to repair the digital files within their collections. Accessible via either the command line, a graphical user interface (GUI), or web-based shell, the MediaConch project will serve to provide detailed individual and batch-level conformance checking analysis using its adaptable, flexible and interoperable software application interface. With a project focus dedicated to furthering the longevity of Matroska, Linear Pulse Code Modulation (LPCM) and FF Video Codec 1 (FFV1) as recommended digital preservation audiovisual formats, MediaConch will anticipate and contribute to the further development of the standardization of these formats. The MediaConch open source project was created and is currently under development by MediaArea, notable for the creation of MediaInfo, an open source media checker software application.

Website: 🐚 MediaConch project.

Table of Repositories

MediaConch

The original repository for the MediaConch project, this repository holds all public documentation related to Phase I of the project (the design phase) and some metadata-related work.

MediaConch_SourceCode

This repository hosts the source code for MediaConch, the GUI.

MediaConchOnline

This is the source code for MediaConchOnline, the online version of the MediaConch shell.

MediaConch-Website

This is the repository for content hosted on https://mediaarea.net/MediaConch/.

MediaInfoLib

This is the repository for the analyzing engine. We expand the generic engine in order to add Matroska and FFV1 tests.

MediaAreaXml

This repository holds XSD (XML Schema Definitions) for MediaConch, MediaInfo, and MediaTrace.

MediaConch_SampleFiles

This repository contains sample files used to test MediaConch.

MediaConch_MKVSurvey

This repository holds a research corpus used in the development of the MediaConch.

How to Contribute

Please see the Contribution Guide for contributing code. Contributions via feedback on our Issues page are also very welcome.

Release Schedule

MediaArea intends to release various versions of all relevant source codes and executables for each of the deployment platforms that the project will be configured to perform upon successfully. For stable versions of the software, new downloads and rolling releases will be provided and made available on a monthly basis. Stable versions will take into account software fixes, updates, and bug reports throughout the development phase and additionally will have gone through a QA process during that time.

Certain deployed (LTS) versions, upholding the build of the stable versions, will be provided and released during the required delivery stages of the PREFORMA project and will be developed as sustainable for a long period of time within the open source project.

New nightly builds and updates of the source code will also be made available to download during all stages. This ensures that all users and organizations will have access to downloading the most up-to-date version of code that exists throughout the project.

Downloads will be made available through a public repository with a functioning issue tracker (GitHub). In conjunction with the releases, a roadmap will be created in order to track these updates publicly and encourage open collaborative usage and issue feedback. Both the older and more recent development, stable, and deployed (LTS) versions will be made available to users of any level, throughout these multiple platforms, for the entirety of the project. If a user wishes to download an older version of the source code or executable, MediaArea will have this option available.

All source codes and updates are and will be made accessible on the following platforms:

MS Windows
macOS
Linux (Ubuntu, Fedora, Debian, and Suse)

Monthly releases and nightly builds are available on the 🐚 MediaConch Downloads page or via MediaConchOnline.

Funding and Licensing

This project has received funding from PREFORMA, co-funded by the European Commission under its FP7-ICT Programme. All software and source code developed by MediaArea during the PREFORMA project will be provided under the following two open source licenses: GNU General Public License 3.0 (GPLv3 or later), Mozilla Public License (MPLv2 or later).

All open source digital assets for the software developed by MediaArea during the PREFORMA project will be made available under the open access license: Creative Commons license attribution – Sharealike 4.0 International (CC BY-SA v4.0). All assets will exist in open file formats within an open platform (an open standard as defined in the European Interoperability Framework for Pan-European eGovernment Service (version 1.0 2004)).

Licensing of third party libraries

The software relies on third party libraries. Such libraries have their own license:

Media analysis library: MediaInfoLib, (c) MediaArea.net SARL, initially BSD-2-Clause license, relicensed to GPLv3 or later / MPLv2 or later
Base classes library: ZenLib, (c) MediaArea.net SARL, initially zlib license, relicensed to GPLv3 or later / MPLv2 or later
C++ standard library: libstdc++ (GPLv3+ with linking exception), libc++ (MIT license, could be relicensed to GPLv3 or later / MPLv2 or later), Visual C++ library (Microsoft proprietary license) or any other C++ standard library
GUI (optional): Qt, LGPLv3+ license, compatible with GPLv3 or later / MPLv2 or later
WebUI engine (optional) : Symphony, MIT license, could be relicensed to GPLv3 or later / MPLv2 or later
WebUI design (optional) : Bootstrap, MIT license, could be relicensed to GPLv3 or later / MPLv2 or later
XML parser (optional): TinyXML-2, zlib license, could be relicensed to GPLv3 or later / MPLv2 or later
gzip decompression routine (optional): zlib, zlib license, could be relicensed to GPLv3 or later / MPLv2 or later
FTP, FTPS, SFTP, HTTP, HTTPS (optional): libcurl, MIT license, could be relicensed to GPLv3 or later / MPLv2 or later
Base64 (optional): base64 by Bob Withers, public domain, could be relicensed to GPLv3 or later / MPLv2 or later
AES (optional): aes by Brian Gladman, BSD-2-Clause license, compatible with GPLv3 or later / MPLv2 or later
MD5 (optional): md5 by Colin Plumb, public domain, could be relicensed to GPLv3 or later / MPLv2 or later
SHA-1 (optional): sha1 by Brian Gladman, BSD-2-Clause license, compatible with GPLv3 or later / MPLv2 or later
SHA-2 (optional): sha2 by Brian Gladman, BSD-2-Clause license, compatible with GPLv3 or later / MPLv2 or later
HMAC (optional): hmac by Brian Gladman, BSD-2-Clause license, compatible with GPLv3 or later / MPLv2 or later

Compilation has been tested with:

GCC for Windows, Mac, Linux, GPLv3+ license
LLVM for Mac, Linux, NCSA Open Source License, could be relicensed to GPLv3 or later / MPLv2 or later
Microsoft Visual Studio, Microsoft proprietary license
Apache web server for Windows, Mac, Linux, Apache license, compatible with GPLv3 or later / MPLv2 or later

Test Files

MediaArea’s test files exist in the SampleTestFiles repository.

It is anticipated that a large library of reference media and test files will be created to highlight the different outcomes associated with issues and errors that may arise in regards to certain files and specifications push through the software. The test files will either be self-created, solicited, or pulled from a variety of online reference libraries.

This curated selection of tests will include the following:

files that conform to the relevant file format’s technical specifications
files that do not conform and therefore deliberately deviate from the file format’s technical specifications (in association with the appropriately coded error messages)
specific files that originate from and/or adhere to the technical specifications of the file formats from participating memory institutions (including examples that both conform to and deviate from the requirements)

Because it is crucial to the stimulation of a sustained and well documented open sourced community, the resulting issues and feedback from the testing of these created and solicited files will also be documented and will contain information on the relevant version of software used for the test.

mediaconch's People

Contributors

Stargazers

Watchers

Forkers

tribouille verwinv guillaumeroques jeromemartinez foundscapes ablwr chlara pavel-pimenov acidburn0zzz g-maxime mrandreastoth

mediaconch's Issues

Functional Component Planning: Implementation Checker

Draft wiki of related howtos

How to generate various simple and complex Matroska files.
How to encode sample ffv1 and sample LPCM files
How to crc-check ffv1.
How to validate matroska via mkvalidate

Associate regex tests with Matroska tags

Identifying which Matroska tags could have their content validated using regex and design such expression: http://www.matroska.org/technical/specs/tagging/index.html. For instance ISBN, URL, EMAIL, could have associated regex.

generate data visualizations for related project media files

simple MKV structure
complex MKV structure
FFV1 version 0
FFV1 version 1
FFV1 version 2
FFV1 version 3
LPCM in WAV
LPCM in MKV

Test files -- Need some for MKV

report on role of sample report

The "Reference and Test Files" section of https://github.com/MediaArea/PreFormaMediaInfo/blob/master/DesignNarrative.md should be extended. In our original proposal notes we said we would:

Identification of and analysis of sample libraries of relevant project file formats. archive.org, multimedia.cx, communitycontributed media Identification of diverse set of file format expressions created by unique software.

review mkv_metadatafixer_cli doc

Can you review 24dcf61 and perhaps into a greater amount of outline or structure to it to fill in. I anticipate following each GUI section with the CLI section since the GUI section may better communicate a user-oriented vision of the use.

The architecture is defined elsewhere but this is intended to cover functional requirements.

Stream / container conflicts

consider project rename

Currently our documentation refers to the Proposal Project name as "PreForma MedaInfo". This is a bit of a placeholder name. Should we consider this official or design a new name?

Draft XML schema for conformance reporting

Design conformance checker API

set up freenode irc

From early proposal by Jerome and I, "Establish online community project resources, github, irc channel, issue tracker, etc". Can you set up an IRC channel and then I will change language in report to say it is done. Perhaps #mediainfo?

Bullet lists no well formated in PDF / HTML outputs

bullet lists are OK in Markdown but are wrong in HTML output and PDF output.
Example: search for "Ubuntu" in the final docs.

Functional Component Planning: Reporter

Describe Intended Release schedule

Design human-readable version of conformance report

Expand on intro in Architecture section

Reference from eval strategy:

tem I2,1 – Architecture
– description: This item concerns infrastructural aspects, technical specifications and sys- tem features of a system. In particular it has to be evaluated by considering:
(i) Interoperability: this item concerns the degree at which the solution can interoperate with other components and solutions;
(ii) Scalability: this item concerns the degree at which the solution is scalable and expan- sible;
(iii) Portability: source code must be built for portability between technical deployment platforms (platform independent);
(iv) Modularity: source code must be built in a modular fashion for improved maintain- ability;
(v) Deployment: the Conformance Checker must allow for deployment in the five infras- tructures/ environments defined in the Challenge Brief, i.c. PREFORMA website, stand alone, networked, in legacy system and in test environment;
(vi) Interface: the Conformance Checker must interface with other software systems via APIs;
– item weight: β2,1 = 30%

Outline a plan for standardization of ffv1 via open standards org (ITU-T, IETF)

http://www.ffv1.info/
https://en.wikipedia.org/wiki/FFV1
http://ffmpeg.org/~michael/ffv1.html (official ffmpeg spec)

create mockup images for metadata fixer ui

I think the functional requirements documentation of the metadata fixer is sufficient to set up some images for support the reports or presentation.

https://github.com/MediaArea/PreFormaMediaInfo/blob/6a81950431b7ca4fdcb5b878e37faf1f8b99590c/Design/GUI/MKVMetadataFixer.md

Add html header / body elements (valid HTML) in Design report html

review StyleGuide.md to ensure consistency with requirements of D4.3

See section 2.1 of PREFORMA_D4.3_Functions of the Open Source Portal_v1.0.pdf

Itemize Conformity Checks for ffv1

Design (or adopt) means to express conformity checks (intent, outcome assessment, severity, etc)

Potential conformance test registry

Functional Component Planning: Policy Checker

Functional Component Planning: Metadata fixer

Eval strategy: Commercial feasibility

Noting the 15% weight (in the category) of this section of the evaluation strategy documents and seeing that we have 0% on it right now:

description: The reviewer has to evaluate the extent to which the approach demonstrates commercial feasibility, and whether it is a realistic commercialization plan or route to market; in particular it has to be considered the following aspects: (i) integration with text, image, moving image editors; (ii) integration with digital repositories; (iii) integration with transcoding software; (iv) integration of additional conformance checkers; (v) integration of additional reporters; (vi) providing consulting services; (vii) providing customization services; (viii) providing support services.

@dericed is gonna pass me the original proposal to pull strategy out and apply to document

remove meaningful tests in matroska checking registry

A check that an elements contents "MUST be in binary format" would always be true (right?), at least with non-zero sized element payloads. I propose these binary checks be refined or removed.

Update SoftwareArchitectureLayers.png

To represent the architecture

Split report assembly process to provide several outputs.

Add introduction to design section

Summary with links to each section

https://github.com/MediaArea/PreFormaMediaInfo/blob/master/Design/Introduction.md

Develop and deploy questionairre for user communities, regarding general conformance checks as well as ffv1, mkv, lpcm

Design playback interface that presents frame w/ associated with frame based metadata

Sample_data sample_value
presentation_timestamp 00:00:00.880000
height 720
width 486
frame_size 220425
framemd5 f5d45832b63103108fe02d81fc0986e0
version 3
micro_version 4
crc value b344ba17
keyframe 1
coder_type 0
error_concealment 0
slices 4
slicecrc [ 764efa88 3dda1e11 db47671c 4a3bbd9e ]
colorspace_type 1
bits_per_raw_sample 8
chroma_planes 1
h_chroma_subsample 0
v_chroma_subsample 0
alpha_plane 1
num_h_slices_minus1 1
num_v_slices_minus1 1
other_data (38 bytes)

Design application UI for conformance check that accomodates checks at container, codec, or frame level

Develop style guide and rules for development environment and practices (ie commit logging practices, coding style)

http://ffmpeg.org/developer.html#Coding-Rules-1 (example from ffmpeg)

Summarizing shell services and features

Make sure we are hitting all of these points:

(v) automated checks: The Shell component of the Conformance Checker must allow for automating the procedures for checking, reporting and fixing preservation file;
(vi) periodical checks: The Shell component of the Conformance Checker must allow for configuring fully automated, periodical checks;
(vii) batch processing: The Shell component of the Conformance Checker must allow for batch processing of extensive file sets;
(viii) addotopma: The Shell component of the Conformance Checker must allow for con- figuration of additional components in particular implementation checkers, policy check- ers and reporters for other preservation file formats that are developed in the PRE- FORMA ecosystem;
(ix) use by non-expert users: The Shell component of the Conformance Checker must allow for use by non-expert users;
(x) operate without Internet: The Shell component of the Conformance Checker must be operational in a closed zone with no Internet access.

Outline plan for documentation and training of conformance tools

Outline community tools (forums, mailing lists, issue tracker, etc)

Diagram of conformance reports within workflow

Roles defined

Consolidate existing documentation and add to it into one doc

Design UI for MKV metadata editing

OAIS section in mkv metadata document?

The section ends with a fragement about metadata and fixity in OAIS context.
https://github.com/MediaArea/PreFormaMediaInfo/blob/6a81950431b7ca4fdcb5b878e37faf1f8b99590c/Design/GUI/MKVMetadataFixer.md

Partly this section repeats what is in the header. Can you either consolidate this closing OAIS closing into the introduction, move it to a separate section that focuses on oais, or simply cut it so the document ends properly.

extend vision for use of open source portal

Although issue trackers, releases, and roadmap are run through github, there are many requirements to coordinate these components through the Open Source Portal. I am not clear on how much control or write access we have to the Open Source Portal (obviously none at the moment), but we should elaborate on our plans. I started a documentation fragment on it here 4771bda.