MinerU is a one-stop, open-source, high-quality data extraction tool, includes the following primary features:

• Magic-PDF PDF Document Extraction
• Magic-Doc Webpage & E-book Extraction

Magic-PDF is a tool designed to convert PDF documents into Markdown format, capable of processing files stored locally or on object storage supporting S3 protocol.

Key features include:

• Support for multiple front-end model inputs
• Removal of headers, footers, footnotes, and page numbers
• Human-readable layout formatting
• Retains the original document's structure and formatting, including headings, paragraphs, lists, and more
• Extraction and display of images and tables within markdown
• Conversion of equations into LaTeX format
• Automatic detection and conversion of garbled PDFs
• Compatibility with CPU and GPU environments
• Available for Windows, Linux, and macOS platforms

• PDF-Extract-Kit
  • A Comprehensive Toolkit for High-Quality PDF Content Extraction

• Python >= 3.9

pip install magic-pdf

cp magic-pdf.template.json ~/magic-pdf.json
magic-pdf pdf-command --pdf "pdf_path" --model "model_json_path"

After the program has finished, you can find the generated markdown files under the directory "/tmp/magic-pdf".

magic-pdf --help

image_writer = DiskReaderWriter(local_image_dir)
image_dir = str(os.path.basename(local_image_dir))
jso_useful_key = {"_pdf_type": "", "model_list": model_json}
pipe = UNIPipe(pdf_bytes, jso_useful_key, image_writer)
pipe.pipe_classify()
pipe.pipe_parse()
md_content = pipe.pipe_mk_markdown(image_dir, drop_mode="none")

s3pdf_cli = S3ReaderWriter(pdf_ak, pdf_sk, pdf_endpoint)
image_dir = "s3://img_bucket/"
s3image_cli = S3ReaderWriter(img_ak, img_sk, img_endpoint, parent_path=image_dir)
pdf_bytes = s3pdf_cli.read(s3_pdf_path, mode=s3pdf_cli.MODE_BIN)
jso_useful_key = {"_pdf_type": "", "model_list": model_json}
pipe = UNIPipe(pdf_bytes, jso_useful_key, s3image_cli)
pipe.pipe_classify()
pipe.pipe_parse()
md_content = pipe.pipe_mk_markdown(image_dir, drop_mode="none")

Demo can be referred to demo.py

Magic-Doc is a tool designed to convert web pages or multi-format e-books into markdown format.

Key Features Include:

• Web Page Extraction

  • Cross-modal precise parsing of text, images, tables, and formula information.

• E-Book Document Extraction

  • Supports various document formats including epub, mobi, with full adaptation for text and images.

• Language Type Identification

  • Accurate recognition of 176 languages.

• Magic-Doc Outstanding Webpage and E-book Extraction Tool

LICENSE.md

The project currently leverages PyMuPDF to deliver advanced functionalities; however, its adherence to the AGPL license may impose limitations on certain use cases. In upcoming iterations, we intend to explore and transition to a more permissively licensed PDF processing library to enhance user-friendliness and flexibility.

• PaddleOCR
• PyMuPDF
• fast-langdetect
• pdfminer.six

@misc{2024mineru,
    title={MinerU: A One-stop, Open-source, High-quality Data Extraction Tool},
    author={MinerU Contributors},
    howpublished = {\url{https://github.com/opendatalab/MinerU}},
    year={2024}
}