I'm new to pydicom so more likely than not I'm doing something wrong. I have a script that will iterate over a list of series instance UIDs and issue a SERIES-level C-FIND against a DICOM SCP. Every so often (but quite often) I get an error on the screen stating:

WARNING [2023-10-26 18:24:00,572]: Received unexpected C-FIND service message

I can't pinpoint from which series it's coming, and I have tried a few that should be triggering this but when I issue just one C-FIND I don't get that warning.

This is the whole code, it just runs over and over a list of ser uids:

    def cfind(self, series_uid):
        # debug_logger()
        ae = AE()
        ae.ae_title = self.calling_ae
        ae.add_requested_context(StudyRootQueryRetrieveInformationModelFind)

        # Associate with the peer AE at IP 127.0.0.1 and port 11112
        logging.getLogger().setLevel(logging.FATAL)
        assoc = ae.associate(self.ip, self.port, ae_title=self.called_ae)
        if assoc.is_established:
            # Send the C-FIND request

            # Create our Identifier (query) dataset
            ds = Dataset()
            ds.add_new(0x0020000E, 'UI', series_uid)
            ds.add_new(0x0020000D, 'UI', '')
            ds.QueryRetrieveLevel = 'SERIES'

            ds.add_new(0x00100010, 'PN', '')  # Patient Name
            ds.add_new(0x00100020, 'LO', '')  # MRN
            ds.add_new(0x00080060, 'CS', '')  # Modality
            ds.add_new(0x00080050, 'SH', '')  # Acc#
            ds.add_new(0x00081030, 'LO', '')  # Study description
            ds.add_new(0x0008103E, 'LO', '')  # Series description

            responses = assoc.send_c_find(ds, StudyRootQueryRetrieveInformationModelFind)
            dicom_result = None
            for (status, identifier) in responses:
                if status and status.Status in [0xff00, 0xff01]:
                    try:
                        logging.info("MRN:\t\t\t" + identifier.PatientID)
                        logging.info("PatientName:\t" + str(identifier.PatientName))
                        logging.info("Study desc:\t\t" + identifier.StudyDescription)
                        logging.info("Series desc:\t\t" + identifier.SeriesDescription)
                        logging.info("Acc#:\t\t\t" + identifier.AccessionNumber)
                        logging.info("Study Ins UID:\t\t\t" + identifier.StudyInstanceUID)
                        logging.info("Series Ins UID:\t\t\t" + identifier.SeriesInstanceUID)
                        dicom_result = DicomResult(MRN=identifier.PatientID,
                                                   PatientName=identifier.PatientName,
                                                   SeriesDescription=identifier.SeriesDescription,
                                                   StudyDescription=identifier.StudyDescription,
                                                   AccessionNumber=identifier.AccessionNumber,
                                                   SeriesInstanceUID=identifier.SeriesInstanceUID,
                                                   StudyInstanceUID=identifier.StudyInstanceUID)
                        break
                    except AttributeError as ae:
                        logging.error('C-FIND query status: 0x{0:04X}'.format(status.Status))
                        logging.error(f"Got error when trying to extract elements from C-FIND response: {ae} - SUID: {study_uid}")
                elif status and status.Status in [0x0000]:
                    # Final response with no particular data. PHS PACS responds with this at the end of a C-FIND.
                    break
                else:
                    raise ValueError('Connection timed out, was aborted or received invalid response.\nStatus: 0x{0:04X}'.format(status.Status))

            # Release the association
            assoc.release()
            logging.getLogger().setLevel(logging.INFO)
            return dicom_result
        else:
            logging.getLogger().setLevel(logging.INFO)
            raise ValueError(f'Association rejected, aborted or never connected - parameters {self.__str__()}')

I don't see any errors or data missing, but that just warnings sent out to stdout/stderr (not sure which - didn't check).

Any ideas?

i having difficulties in deployment if someone have a proper guide on how deploy , it would be really helpful.

It would be good to know what tasks are pending / running in the worker database (redis) to go along with the new metrics view.

It should be easy to look into the errors for a batch, etc. A web interface view and possibly command line viewer would be good.

while the images have attributes for Study and Session, it doesn't make sense to model them from the application's stand point. The model should be redone to have a single Batch, indicative of a folder of images, each of which is represented as an Image (still) in the database. The workers would then pass around batch ids instead of the list of images ids.

There should be a development / getting started base, so a user can go from 0 --> running easily, and reference the optional "FYI / if interested" pages linked appropriately.

in the future, we would want the user to have more power to control tasks, and processing, and (given that the action to get the files lives outside the application) we should read this from some kind of config file, otherwise go to defaults. A better idea might be to have the user register the C-MOVE in the application, and specify these things when setting that up.

https://nrdr.acr.org/Portal/HELP/DIR/DataElementsInDIR-Header-SR.pdf

Specific note to:

also send the series instance, series number and instance number in to the API for de-identification.

Sendit Base

• Create sendit base image to save building time ready for testing.

We would want to be able to quickly deploy the main application Dockerfile, or entirely different tools with the same google APIs, off of this base. Then the Dockerfile for sendit can simply update the (regularly changing) libraries:

FROM pydicom/sendit-base

# update deid
WORKDIR /opt
RUN git clone -b development https://github.com/pydicom/deid
WORKDIR /opt/deid
RUN python setup.py install

# som
WORKDIR /opt
RUN git clone https://github.com/vsoch/som
WORKDIR /opt/som
RUN python setup.py install

WORKDIR /code
ADD . /code/
CMD /code/run_uwsgi.sh

EXPOSE 3031

No targuzz

• Images should be represented on the level of dicoms
  This makes a lot of sense to me in terms of metadata - we want to represent metadata about images, not about zipped up things that need to be unzipped first. We can also very easily view a dicom in a browser from a url, and this isn't the case with .tar.gz (unless it's another format like nifti).

User Friendly Config File

If we can see some day being able to deploy a sendit instance for a researcher, the configuration needs to be easy and stupid. The harder part is generation of the deid recipe, but for the rest of it, it should come down to reading a file that gets integrated into their custom build and then drives the application. It might also make sense to represent the config in the database as a model, that way one instance can have several (and the input folders for each are defined when created) and changes can be made without stopping/restarting the application.

Som BigQuery Client

• Implement bigquery client into som-tools, use for metadata
  We would want to use BigQuery instead of Datastore. This is ready to go and needs testing.

Testing

I want to do the following tests to generally get a "move images" and "move metadata" strategy. It comes down to testing batched uploads (in sync), batched uploads (separate images from metadata) vs. rsynch (more risky but a lot faster according to others).

• Test speed with bigquery + metadata + storage
• Test speed with caching metadata + storage
• If time still slow, investigate rsync

Changes for Dasher

• changes to dasher endpoint (session?)
  I'll leave this to Susan to ping me when we absolutely need changes.

Note - this is still a Stanford hosted server, without PHI on cloud

and those functions should be the ones also used by the command.

Currently, the storage collection (eg, the study name based on IRB) is in the config settings:

GOOGLE_STORAGE_COLLECTION=None # define here or in your secrets

This is fine for testing, but should be a variable associated with a batch when we go out of testing.

this will decouple the de-identification functions (for header and pixel data) from using the endpoint.

https://pydicom.github.io/deid