cutright / imrt-qa-data-miner Goto Github PK

View Code? Open in Web Editor NEW

13.0 4.0 5.0 156 KB

Scans a directory for IMRT QA results

License: MIT License

Python 100.00%

data-mining radiation-oncology qa pdfminer

imrt-qa-data-miner's Introduction

IMRT-QA-Data-Miner

Scans a directory for IMRT QA results.

THIS PROJECT HAS MOVED

This project has since been sponsored by the AAPM's IMRT Working Group. There is a new project here: https://github.com/IQDM/IQDM-PDF

The new version does not have plotting built in, but we plan to use DVHA Stats to take care of that once enough testing has been done, and more vendors are added.

Install

pip install iqdm

How to run

To scan a directory for IMRT QA report files and genereate a results .csv file:

iqdm <initial-scan-dir>

To launch a trending dashboard (and open the resulting link):

iqdm <results-csv-file-path>

Screenshot of dashboard:

Command line usage

usage: iqdm [-h] [-ie] [-od OUTPUT_DIR] [-rd RESULTS_DIR] [-all]
            [-of OUTPUT_FILE] [-ver] [-nr] [-df] [-p PORT]
            [-wo WEBSOCKET_ORIGIN]
            [file_path]

Command line interface for IQDM

positional arguments:
  file_path             Initiate scan if directory, launch dashboard if
                        results file

optional arguments:
  -h, --help            show this help message and exit
  -ie, --ignore-extension
                        Script will check all files, not just ones with .pdf
                        extensions
  -od OUTPUT_DIR, --output-dir OUTPUT_DIR
                        Output stored in local directory by default, specify
                        otherwise here
  -rd RESULTS_DIR, --results-dir RESULTS_DIR
                        Results assumed to be stored in local directory by
                        default, specify otherwise here
  -all, --process-all   Process all identified report files, otherwise only
                        new reports will be analyzed
  -of OUTPUT_FILE, --output-file OUTPUT_FILE
                        Output will be saved as <report_type>_results_<time-
                        stamp>.csv by default. Define this tag to customize
                        file name after <report_type>_
  -ver, --version       Print the IQDM version
  -nr, --no-recursive-search
                        Include this flag to skip sub-directories
  -df, --day-first      Assume day first for ambiguous dates in trending
                        dashboard
  -p PORT, --port PORT  Specify port of trending dashboard webserver
  -wo WEBSOCKET_ORIGIN, --allow-websocket-origin WEBSOCKET_ORIGIN
                        Allow a websocket origin other than localhost, see
                        bokeh documentation

Notes

This script was written specifically for SNC Patient and Delta4, but I'd be happy to include support for other vendors if someone could provide some anonymized example reports.

Vendor Compatibility

Sun Nuclear: SNC Patient
- ArcCheck compatibility contributed by Marc Chamberland
ScandiDos: Delta4
This is still in beta, but the reported csv data is largely correct (reported energy might be off). The class parses much more data (including individual beam results), but isn't currently in csv nor validated.

Contributing

If you'd like to contribute code to support a new vendor, please create a new python file in the parsers directory containing a new class. This class should include the following to be compatible:

PROPERTIES
- identifiers
  this is a list of strings that collectively and uniquely are found in a report type
- columns
  a list of strings indicating the columns of the csv to be output
- csv
  a string of values for each column, delimited with DELIMITER in utilities.py
- report_type
  a string succinctly describing the report, this will be used in the results filename created in main.py
METHODS
- process_data(text_data)
  processing the data does not occur until this is called

imrt-qa-data-miner's People

Contributors

Stargazers

Watchers

Forkers

mchamberland cat2tom lynch829 jon-george

imrt-qa-data-miner's Issues

Merge ArcCheck and MapCheck parsers?

As far as I can tell, MapCheck and ArcCheck reports are identical, except for the name at the top of the report (MapCheck or ArcCheck).

Should we merge the two parsers into one, then?

I have a few cases that even though we acquired two ArcCheck measurements, the report of the combined results show "MapCHECK" at the top of the report... We'd need to re-work the way it determines the report type to allow both ArcCheck and MapCheck, in any upper/lower case combination.

Request to process only new PDFs

Feature request to ignore previously processed PDFs

arccheck.py produces ValueError: {'Minimum': 'n/a', 'Maximum': 'n/a', 'Average': 'n/a', 'Stdv': 'n/a'} is not in list

I finally tried out the ArcCheck module and got:
ValueError: {'Minimum': 'n/a', 'Maximum': 'n/a', 'Average': 'n/a', 'Stdv': 'n/a'} is not in list

Will post a suggested change momentarily.

Handle PDFSyntaxError in convert_pdf_to_text

We should try to skip bad PDF files to prevent a single bad file stopping the entire analysis:

Traceback (most recent call last):
File "c:\Users\m310767.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\ptvsd_launcher.py", line 43, in
main(ptvsdArgs)
File "c:\Users\m310767.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\old_ptvsd\ptvsd_main_.py", line 432, in main
run()
File "c:\Users\m310767.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\lib\python\old_ptvsd\ptvsd_main_.py", line 342, in run_module
run_module_as_main(target, alter_argv=True)
File "c:\Users\m310767\AppData\Local\Continuum\anaconda3\envs\py3\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "c:\Users\m310767\AppData\Local\Continuum\anaconda3\envs\py3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\m310767\dev\IMRT-QA-Data-Miner\IQDM\main.py", line 150, in
main()
File "C:\Users\m310767\dev\IMRT-QA-Data-Miner\IQDM\main.py", line 140, in main
no_recursive_search=args.no_recursive_search)
File "C:\Users\m310767\dev\IMRT-QA-Data-Miner\IQDM\main.py", line 63, in process_files
process_file(file_path, output_file, output_dir)
File "C:\Users\m310767\dev\IMRT-QA-Data-Miner\IQDM\main.py", line 68, in process_file
row, report_type, columns = pdf_to_qa_result(file_path) # process file
File "C:\Users\m310767\dev\IMRT-QA-Data-Miner\IQDM\main.py", line 30, in pdf_to_qa_result
text = convert_pdf_to_txt(abs_file_path)
File "C:\Users\m310767\dev\IMRT-QA-Data-Miner\IQDM\pdf_to_text.py", line 31, in convert_pdf_to_txt
check_extractable=True):
File "c:\Users\m310767\AppData\Local\Continuum\anaconda3\envs\py3\lib\site-packages\pdfminer\pdfpage.py", line 129, in get_pages
doc = PDFDocument(parser, password=password, caching=caching)
File "c:\Users\m310767\AppData\Local\Continuum\anaconda3\envs\py3\lib\site-packages\pdfminer\pdfdocument.py", line 585, in init
raise PDFSyntaxError('No /Root object! - Is this really a PDF?')
pdfminer.pdfparser.PDFSyntaxError: No /Root object! - Is this really a PDF?

In this case, the offending file is a broken PDF of 0 size.

More robust date parsing for trending

https://github.com/cutright/IMRT-QA-Data-Miner/blob/master/trending_arccheck.py#L111

SNC Patient software actually allows two date formats for the PDF reports:
<month>/<day>/<year>
<day>/<month>/<year>

The easiest way to deal with this would be to enforce one format, but the nice thing would be to offer an option to the user.

Also, there are some sanity checks that could be done.

Any idea? I've never had to deal with Pinnacle scripts before. Someone suggested that WindowList should be XtWindowList in Pinnacle 16, but that didn't work.

if 'Any' in active_gamma or gamma_crit in active_gamma:
   try:
      y_float = float(self.data[select_y.value][i])
   except ValueError:
      continue
   new_data['x'].append(self.x[i])
   new_data['y'].append(y_float)
   new_data['id'].append(self.data['Patient ID'][i])