ome / omero-scripts Goto Github PK

Core OMERO Scripts

Home Page: https://pypi.org/project/omero-scripts/

Python 100.00%

omero scripts python

omero-scripts's Introduction

OMERO Core Scripts

This directory contains OMERO scripts which use the OMERO.scripts API. All scripts (e.g. *.py) present in the directory will be automatically distributed with all binary builds. Which file-endings will be detected and how they will be launched are both configured centrally in the server. .py, .jy, and .m (MATLAB) files should all be detected by default starting with OMERO 5.

Directory	Description
analysis_scripts	crunch images to produce numerical results and similar tasks
export_scripts	take one or more images as an input, and produce a representation for exchange
figure_scripts	take one or more images as an input, and produce a summary representation
import_scripts	are run on images after import for extra processing
util_scripts	perform other miscellaneous tasks like cleaning up or optimizing OMERO itself

OMERO User Scripts

If you would like to provide your own scripts for others to install into their OMERO installations, please see https://openmicroscopy.org/info/scripts

Testing

Integration tests under test/ require an OMERO server with scripts installed. The tests are run by Travis for open PRs using omero-test-infra to deploy OMERO via Docker containers.

To run tests locally:

# All tests
$ python setup.py test

# Single test in a single file
$ python setup.py test -t test/integration/test_util_scripts.py -k test_dataset_to_plate

Usage

See https://omero-scripts.readthedocs.io/en/stable/

Release process

This repository uses bump2version to manage version numbers. To tag a release run:

$ bumpversion release

This will remove the .dev0 suffix from the current version, commit, and tag the release.

To switch back to a development version run:

$ bumpversion --no-tag [major|minor|patch]

specifying major, minor or patch depending on whether the development branch will be a major, minor or patch release. This will also add the .dev0 suffix.

Remember to git push all commits and tags.s essential.

The CI pipeline will automatically deploy the tag onto PyPI.

Copyright

2010-2021, The Open Microscopy Environment

omero-scripts's People

Contributors

Stargazers

Watchers

omero-scripts's Issues

Investigate use of Shapely for exporting Points coordinates

From #173 (comment)
In the Batch_ROI_Export script, we could look at the option to export a simplified Polygon, to preserve the shape but without a long list of coordinates. See https://shapely.readthedocs.io/en/latest/manual.html#object.simplify
As used at by Vitessce, see https://docs.google.com/presentation/d/1QzKYP6sXPefBMNfY4PW4H0AMoWVbw9HeNoweLmJg17Y/edit#slide=id.g823da5adf3_0_105

RFE: change default branch to master

It sits oddly to still be working from develop here when we don't much use dev_<n>_<n> and so many of the related repositories use master so maybe we could use master on this repository too and adjust CI accordingly.

Movie ROI figure errors on py3

See line 21 of https://docs.google.com/spreadsheets/d/1ry6vS0dmsylFQDAiIZDrPC3kXNpnHSMtK-2yj5X8jaU/edit#gid=420881033

"Move ROI Figure script ends with


          Traceback (most recent call last):

  File ""/home/omero/workspace/OMERO-web/omero-virtualenv/lib/python3.6/site-packages/django/core/handlers/exception.py"", line 41, in inner
    response = get_response(request)

  File ""/home/omero/workspace/OMERO-web/omero-virtualenv/lib/python3.6/site-packages/django/core/handlers/base.py"", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)

  File ""/home/omero/workspace/OMERO-web/omero-virtualenv/lib/python3.6/site-packages/django/core/handlers/base.py"", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File ""/home/omero/workspace/OMERO-web/OMERO.web/lib/python/omeroweb/decorators.py"", line 485, in wrapped
    retval = f(request, *args, **kwargs)

  File ""/home/omero/workspace/OMERO-web/OMERO.web/lib/python/omeroweb/decorators.py"", line 535, in wrapper
    context = f(request, *args, **kwargs)

  File ""/home/omero/workspace/OMERO-web/OMERO.web/lib/python/omeroweb/webclient/views.py"", line 3698, in script_ui
    raise ex

  File ""/home/omero/workspace/OMERO-web/OMERO.web/lib/python/omeroweb/webclient/views.py"", line 3693, in script_ui
    params = scriptService.getParams(long(scriptId))

  File ""/home/omero/workspace/OMERO-web/OMERO.web/lib/python/omero/gateway/__init__.py"", line 4748, in __call__
    return self.handle_exception(e, *args, **kwargs)

  File ""/home/omero/workspace/OMERO-web/OMERO.web/lib/python/omeroweb/webclient/webclient_gateway.py"", line 2131, in handle_exception
    e, *args, **kwargs)

  File ""/home/omero/workspace/OMERO-web/OMERO.web/lib/python/omero/gateway/__init__.py"", line 4745, in __call__
    return self.f(*args, **kwargs)

  File ""/home/omero/workspace/OMERO-web/OMERO.web/lib/python/omero_api_IScript_ice.py"", line 842, in getParams
    return _M_omero.api.IScript._op_getParams.invoke(self, ((scriptID, ), _ctx))

omero.ValidationException: exception ::omero::ValidationException
{
    serverStackTrace = 
    serverExceptionClass = 
    message = Can't find params for 3855!
No stdout.
stderr is in file 5078:
---------------------------------
Traceback (most recent call last):
  File ""./script"", line 836, in <module>
    run_script()
  File ""./script"", line 728, in run_script
    ckeys.sort()
AttributeError: 'dict_keys' object has no attribute 'sort'

---------------------------------

}

<WSGIRequest: GET '/web/webclient/script_ui/3855/?Image=985'>
        "

Populate metadata on SPW

In my hands, the Populate Metadata UI script does not work on plates.

workflow:
Find a plate with no attachments, e.g. https://workshop.openmicroscopy.org/webclient/?show=plate-102
or https://merge-ci.openmicroscopy.org/web/webclient/?show=plate-13855 (user-3)

Run on it the script, using the CSV as below

# header well,plate,s,d,l,d
Well,Plate,Drug,Concentration,Cell_Count,Percent_Mitotic
A1,plate1_1_013,DMSO,10.1,10,25.4
A2,plate1_1_013,DMSO,0.1,1000,2.54
A3,plate1_1_013,DMSO,5.5,550,4
B1,plate1_1_013,DrugX,12.3,50,44.43

Observe a crash

Traceback (most recent call last):
  File "./script", line 124, in populate_metadata
    ctx.parse_from_handle(data)
AttributeError: 'ParsingContext' object has no attribute 'parse_from_handle'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./script", line 181, in <module>
    run_script()
  File "./script", line 173, in run_script
    message = populate_metadata(client, conn, script_params)
  File "./script", line 128, in populate_metadata
    ctx.preprocess_from_handle(data_for_preprocessing)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_metadata/populate.py", line 945, in preprocess_from_handle
    self.preprocess_data(reader)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_metadata/populate.py", line 1019, in preprocess_data
    column, original_value, row)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_metadata/populate.py", line 414, in resolve
    return self.wrapper.resolve_plate(column, row, value)
AttributeError: 'PlateWrapper' object has no attribute 'resolve_plate

cc @sbesson @will-moore

Populate metadata and BOM

The Populate Metadata script fails if the CSV file which is fed into it has a BOM marker.
The BOM marker is easily created by following workflow

Open Microsoft Excel. Import (i.e. do not Open) a CSV file. Edit some bits of this file and save it again as CSV (that is how a user would inadvertently obtain a BOM merked file).
You can set or unset the using VIM. (command set nobomb to unset, set bomb to set).

Suggestion: Make the Populate Metadata script robust to handle CSV files with BOM. similarly to ome/omero-metadata@44f69e9

cc @sbesson @manics

omero_metadata.populate.MetadataError: 
Column type 'ï»¿well' unknown.
Choose from following: plate,well,image,dataset,roi,d,l,s,b

populate metadata error handling

E.g. see errors reported at https://forum.image.sc/t/how-to-use-cellprofiler-per-image-results-from-plate-analyses-as-source-for-omero-parade/59894/19

The actual error ValueError: Empty Double or Long value. Use --allow_nan to convert to NaN is buried among lots of others.
including:

Traceback (most recent call last):
  File "./script", line 124, in populate_metadata
    ctx.parse_from_handle(data)
AttributeError: 'ParsingContext' object has no attribute 'parse_from_handle'

Probably the metadata plugin needs to do better too.

Replace mencoder with imageio-ffmpeg

https://pypi.org/project/imageio-ffmpeg/ should offer more control in the Make_Movie.py script than using mencoder:

omero-scripts/omero/export_scripts/Make_Movie.py

Lines 141 to 159 in 4a6eec0

    
           def build_avi(size_x, size_y, filelist, fps, movie_name, format): 
        
               """ Encodes. """ 
        
               program = 'mencoder' 
        
               args = "" 
        
               if (format == WMV): 
        
                   args = ' mf://'+filelist + ' -mf w=' + str(size_x) + ':h=' + \ 
        
                       str(size_y) + ':fps=' + str(fps) + \ 
        
                       ':type=jpg -ovc lavc -lavcopts vcodec=wmv2 -o %s' % movie_name 
        
               elif (format == QT): 
        
                   args = ' mf://'+filelist + ' -mf w=' + str(size_x) + ':h=' + \ 
        
                       str(size_y) + ':fps='+str(fps) + \ 
        
                       ':type=png -ovc lavc -lavcopts vcodec=mjpeg:vbitrate=800  -o %s' \ 
        
                       % movie_name 
        
               else: 
        
                   args = ' mf://'+filelist + ' -mf w=' + str(size_x) + ':h=' + \ 
        
                       str(size_y) + ':fps=' + str(fps) + \ 
        
                       ':type=jpg -ovc lavc -lavcopts vcodec=mpeg4 -o %s' % movie_name 
        
               log(args) 
        
               os.system(program + args)

Improved delimiter sniffing for .csv files

Hello,

Adding to a previous discussion we had #195, I would like to propose an improvement to the delimiter sniffing technique we use.
Why do I come up with this now? Because @SchmChris found a .csv which could not be properly resolved with the current code.
the .csv in question: test.csv

the current code reads:

with open(temp_name, 'rt', encoding='utf-8-sig') as file_handle:
            try:
                delimiter = csv.Sniffer().sniff(
                    file_handle.read(500), ",;\t").delimiter
                print("Using delimiter: ", delimiter,
                      " after reading 500 characters")
            except Exception:
                file_handle.seek(0)
                try:
                    delimiter = csv.Sniffer().sniff(
                        file_handle.read(1000), ",;\t").delimiter
                    print("Using delimiter: ", delimiter,
                          " after reading 1000 characters")
                except Exception:
                    file_handle.seek(0)
                    try:
                        delimiter = csv.Sniffer().sniff(
                            file_handle.read(2000), ";,\t").delimiter
                        print("Using delimiter: ", delimiter,
                              " after reading 2000 characters")
                    except Exception:
                        print("Failed to sniff delimiter, using ','")
                        delimiter = ","
            # reset to start and read whole file...
            file_handle.seek(0)
            data = list(csv.reader(file_handle, delimiter=delimiter))

I propose ditching the fixed values of characters we feed into the sniffer and go for a dynamic approach where we read the first quarter, half , three quarters or the whole file.

from math import floor
with open("temp_name", 'rt', encoding='utf-8-sig') as file_handle:
    file_length = len(file_handle.read(-1))
    try:
        delimiter = csv.Sniffer().sniff(
            file_handle.read(floor(file_length/4)), ",;\t").delimiter
        print("Using delimiter: ", delimiter,
              f" after reading {floor(file_length/4)} characters")
    except Exception:
        file_handle.seek(0)
        try:
            delimiter = csv.Sniffer().sniff(
                file_handle.read(floor(file_length/2)), ",;\t").delimiter
            print("Using delimiter: ", delimiter,
                  f" after reading {floor(file_length/2)} characters")
        except Exception:
            file_handle.seek(0)
            try:
                delimiter = csv.Sniffer().sniff(
                    file_handle.read(floor(file_length*0.75)), ",;\t").delimiter
                print("Using delimiter: ", delimiter,
                      f" after reading {floor(file_length*0.75)} characters")
            except Exception:
                file_handle.seek(0)
                try:
                    delimiter = csv.Sniffer().sniff(
                        file_handle.read(file_length), ",;\t").delimiter
                    print("Using delimiter: ", delimiter,
                          " after reading all characters")
                except Exception:
                    print("Failed to sniff delimiter, using ','")
                    delimiter = ","
# reset to start and read whole file...
	file_handle.seek(0)
	data = list(csv.reader(file_handle, delimiter=delimiter))

This will cost maybe a bit more time, but will more reliably lead to a successful "sniffing" of the delimiter.

Does this sound like a good idea, am I missing some issue or is there maybe a more elegant approach to implement this?

When this more robust method is implemented (in whatever fashion) I would also like to adapt the code for populate_metadata.py and OMERO.parade to include this, as the delimiter-issue will come up for any german localized Excel version and possibly prevents new OMERO users from utilizing this part of the OMERO functionality.

Populate_Metadata: add support for bulkmap context

See https://forum.image.sc/t/are-attributes-in-an-images-table-searchable/27999/5

In addition to the table generation functionality, a future extension of the Populate_Metadata script (or a separate script if needed) would be to be convert the table into key/value pairs via a script.

Immediately, this could achieved either by attaching a file containing the configuration file to use for the conversion. Depending on ome/omero-metadata#33, the script could also use a default configuration to turn every column of every row into a key/value pair.

split view figure z-projection with one z returns an error

Discovered whilst creating a split-view-figure to test #159 on py3-ci

Select a multi-channel image with one Z. Create a split view figure, selection Z-projection. Run the script and you'll receive an error:

    serverExceptionClass = ome.conditions.ValidationException
    message = Z interval value cannot be >= 1

Expected behaviour: Should work, a maximum projection of an image with one Z should be the same as the original image. Alternatively disable the option.

export roi: values rounding

iviewer and the script both allow to export values
one round (iviewer) the value e.g. 66.95859863900198 => 66.959
the script does not.
This will probably confuse user

RFE: Batch ROI export

For plates, have a well (row, column, e.g. A1) as a parameter in the csv
For plates, the activities do not suggest "navigate to the plate where I just attached the csv" icon (unlike for P/Ds)
For multi-selections (select several Datasets or several Plates or several Projects or Screens, then run the script) or single Projects and Screens with multiple Datasets and Plates respectively, the parameter missing is the originating P/D/S/P

Keyval from CSV support choice of Value separator

Currently, if you have a value that contains ; then the value is split on this token and multiple Key:Value pairs are created from the same Key:

omero-scripts/omero/annotation_scripts/KeyVal_from_csv.py

Line 275 in d3fb626

vals = row[i].strip().split(';')

However, I wanted to use this script with a csv I downloaded from
https://zenodo.org/record/7308444/files/Davis%20lab%20200%20YFP%20trap%20expression.csv?download=1
which works with images from https://zenodo.org/record/7308444/files/Figures.zip?download=1
(paper at https://rupress.org/jcb/article/222/6/e202205129/214092/Systematic-analysis-of-YFP-traps-reveals-common)

That csv uses commas to separate multiplr values, e.g. "synapse, cytoplasm, plasma membrane".
I manually changed the script to split on ", " and this worked great, but it would be nice to have the option to choose this separator:

cc @Tom-TBT since I think your changes at https://github.com/German-BioImaging/omero-scripts/tree/xtnd_support_kvpairs remove this support for multiple Values?

Use script on other shapes

Hi all !

I wanted to the use the Images_From_ROIs script to crop my images on our OMERO, but was surprised to see that this is limited to rectangles as shown here:

omero-scripts/omero/util_scripts/Images_From_ROIs.py

Line 146 in 88219a3

if type(shape) == omero.model.RectangleI:

I wanted to modify it to accept all types of shapes but couldn't find anything about getting the bounding box for any other shape, did I miss something ?

Thanks for your help !

RemoveKeyVal - should delete key-value annotations and support HCS plates

#191 (comment)

RemoveKeyVal: Strips off all key-val pairs in the special namespace from selected items. Issue is that it does not really remove the annotations (as in "delete from the database") but somehow hides from the view. They resurface under certain conditions like applying KeyVal_from_csv again. This one should stay and be debugged IMHO.

Also, mpievolbio-scicomp#1 (review) should support HCS plates.

AttributeError: 'FreeTypeFont' object has no attribute 'getsize'

Reported at ome/omero-figure#522

This is the same issue as found on OMERO.figure, where Pillow 10 has removed font.getsize() ome/omero-figure#523

Running Split View Figure script fails with:

Traceback (most recent call last):
  File "./script", line 802, in <module>
    run_script()
  File "./script", line 790, in run_script
    [file_annotation, message] = split_view_figure(conn, script_params)
  File "./script", line 630, in split_view_figure
    fig = make_split_view_figure(
  File "./script", line 389, in make_split_view_figure
    text_height = font.getsize("Textq")[1]
AttributeError: 'FreeTypeFont' object has no attribute 'getsize'

cc @juliomateoslangerak

Make_Movie choose labels positions

See https://forum.image.sc/t/features-of-omero-movie/45013
Would like to be able to choose different positions for the labels.

Error while exporting ROI

Dear Team,

Hope you are doing well.

while exporting ROI we are getting below error in the version "OMERO.web 5.16.0." , could you please help us to resolve this error?

This method cannot handle tiled images yet.
roi_export_error.txt

Regards,
Mohammedhusen Khatib

Make_Movie overlay ROIs

Add ability to include ROIs in exported movies, as requested: https://forum.image.sc/t/features-of-omero-movie/45013/3

Kymograph.py numpy.math depreciation warning

Seen in a stderr log on my fresh dev server with numpy v1.26.4:

It comes from omero/analysis_scripts/Kymograph.py

DeprecationWarning: np.math is a deprecated alias for the standard library math module (Deprecated Numpy 1.25). Replace usages of np.math with math

export roi: unit display

area (µm) This should be area (µm²)

py3 issues

Batch_ROI_Export failing because of unicode um characters at

 File "./script", line 335, in run_script
    result = batch_roi_export(conn, script_params)
  File "./script", line 280, in batch_roi_export
    file_ann = write_csv(conn, export_data, script_params, symbol)
  File "./script", line 238, in write_csv
    csv_file.write("\n".join(csv_rows))
UnicodeEncodeError: 'ascii' codec can't encode character '\xb5' in position 64: ordinal not in range(128)

Populate_Metadata.py - opening script dialog gives:

omero.ValidationException: exception ::omero::ValidationException
{
    serverStackTrace = 
    serverExceptionClass = 
    message = Can't find params for 266058!
No stdout.
stderr is in file 266442:
---------------------------------
Traceback (most recent call last):
  File "./script", line 43, in <module>
    from omero_metadata.populate import ParsingContext
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_metadata/populate.py", line 87
    Report bugs to [email protected]""" % (error, cmd, cmd)
                                                          ^
SyntaxError: invalid syntax

	def build_avi(size_x, size_y, filelist, fps, movie_name, format):
	""" Encodes. """
	program = 'mencoder'
	args = ""
	if (format == WMV):
	args = ' mf://'+filelist + ' -mf w=' + str(size_x) + ':h=' + \
	str(size_y) + ':fps=' + str(fps) + \
	':type=jpg -ovc lavc -lavcopts vcodec=wmv2 -o %s' % movie_name
	elif (format == QT):
	args = ' mf://'+filelist + ' -mf w=' + str(size_x) + ':h=' + \
	str(size_y) + ':fps='+str(fps) + \
	':type=png -ovc lavc -lavcopts vcodec=mjpeg:vbitrate=800 -o %s' \
	% movie_name
	else:
	args = ' mf://'+filelist + ' -mf w=' + str(size_x) + ':h=' + \
	str(size_y) + ':fps=' + str(fps) + \
	':type=jpg -ovc lavc -lavcopts vcodec=mpeg4 -o %s' % movie_name
	log(args)
	os.system(program + args)