umccr / cttso-ica-to-pieriandx Goto Github PK
View Code? Open in Web Editor NEWTransfers data from GDS / creates PierianDx case / run and informatics job for each sample.
License: MIT License
Transfers data from GDS / creates PierianDx case / run and informatics job for each sample.
License: MIT License
If a lambda is accidentally launched twice, both batch jobs are invoked at the exact same time.
Two options to resolve this:
This would be triggered on creation of a PR,
Would need to use aws credentials in bastion to see one particular cloud formation event, and to get the status of that cloudformation.
[ERROR] AttributeError: 'Series' object has no attribute 'redcap_is_complete'
Traceback (most recent call last):
File "/var/task/lambda_code.py", line 1450, in lambda_handler
processing_df = submit_libraries_to_pieriandx(processing_df)
File "/var/task/lambda_code.py", line 327, in submit_libraries_to_pieriandx
processing_df["submission_arn"] = processing_df.apply(
File "/var/lang/lib/python3.9/site-packages/pandas-1.4.3-py3.9-linux-x86_64.egg/pandas/core/frame.py", line 8845, in apply
return op.apply().__finalize__(self, method="apply")
File "/var/lang/lib/python3.9/site-packages/pandas-1.4.3-py3.9-linux-x86_64.egg/pandas/core/apply.py", line 733, in apply
return self.apply_standard()
File "/var/lang/lib/python3.9/site-packages/pandas-1.4.3-py3.9-linux-x86_64.egg/pandas/core/apply.py", line 857, in apply_standard
results, res_index = self.apply_series_generator()
File "/var/lang/lib/python3.9/site-packages/pandas-1.4.3-py3.9-linux-x86_64.egg/pandas/core/apply.py", line 873, in apply_series_generator
results[i] = self.f(v)
File "/var/task/lambda_code.py", line 329, in <lambda>
if x.is_validation_sample or (x.is_research_sample and not x.redcap_is_complete)
File "/var/lang/lib/python3.9/site-packages/pandas-1.4.3-py3.9-linux-x86_64.egg/pandas/core/generic.py", line 5575, in __getattr__
return object.__getattribute__(self, name)
Portal Workflow Run is set to UTC time,
Pieriandx Case Creation date is a few hours behind (US / Eastern time).
When matching the correct workflow id to the pieriandx submission id, we use the portal UTC date but the pieriandx date may be the day before.
This means that the job will be attempted to be resubmitted as the lims lambda script dismisses the join between the portal data and the pieriandx submission as the portal workflow run id cannot be the day after the pieriandx submission.
Since we only get the date from PierianDx, we cannot convert from US Eastern to UTC timezone. Instead, grab the portal date, and convert to us eastern and then perform check
Slack: https://umccr.slack.com/archives/C8CG6K76W/p1688192222630239
CloudWatch link
This CDK CLI is not compatible with the CDK library used by your application. Please upgrade the CLI to the latest version.
(Cloud assembly schema version mismatch: Maximum schema version supported is 20.0.0, but found 31.0.0)
Error simply states "Cannot be found in merged df" when there is no merged row matching subject id, library id, portal run id.
Previously taking the last element in the array has been successful, however we have one case with the following content in the cttso lims
| pieriandx_case_id | pieriandx_case_accession_number | pieriandx_case_creation_date | pieriandx_case_identified | pieriandx_panel_type | pieriandx_workflow_id | pieriandx_workflow_status | pieriandx_report_status |
|-------------------|---------------------------------|------------------------------|---------------------------|----------------------|-----------------------|---------------------------|-------------------------|
| 227147 | SBJ03034_L2300015_001 | 2023-01-15 0:00:00 | TRUE | MAIN | 190454 | canceled | complete |
However this case has a new job that is successful (190459).
Rather than taking the last element, it may be worth sorting based on ID, knowing that the IDs are created chronologically.
2023-07-13 03:14:53,348 - INFO - lambda_code - lambda_handler : LineNo. 1566 - Got 2 rows to replace
--
[ERROR] KeyError: "['pieriandx_case_id'] not in index"
Traceback (most recent call last):
File "/var/task/lambda_code.py", line 1570, in lambda_handler
pieriandx_job_status_missing_df = update_pieriandx_job_status_missing_df(
pieriandx_job_status_missing_df, merged_df
)
File "/var/task/lambda_code.py", line 759, in update_pieriandx_job_status_missing_df
merged_df = merged_df[[ File
"/var/lang/lib/python3.10/site-packages/pandas-1.5.3-py3.10-linux-x86_64.egg/pandas/core/frame.py", line 3813, in __getitem__ indexer = self.columns._get_indexer_strict(key, "columns")[1]
File "/var/lang/lib/python3.10/site-packages/pandas-1.5.3-py3.10-linux-x86_64.egg/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict self._raise_if_missing(keyarr, indexer, axis_name)
File "/var/lang/lib/python3.10/site-packages/pandas-1.5.3-py3.10-linux-x86_64.egg/pandas/core/indexes/base.py", line 6133, in _raise_if_missing raise KeyError(f"{not_found} not in index")
Disable event bridge trigger from lambda if processing df contains items that exist in the update_df (this shouldn't happen)
pieriandx_submission_time is empty, and is causing issues with safe-guard against resubmissions.
Traceback (most recent call last): File "/var/task/lambda_code.py", line 1788, in lambda_handler processing_df = submit_libraries_to_pieriandx(processing_df)
File "/var/task/lambda_code.py", line 348, in submit_libraries_to_pieriandx processing_df["submission_arn"] = processing_df.apply(
File "/var/lang/lib/python3.11/site-packages/pandas-1.5.3-py3.11-linux-x86_64.egg/pandas/core/frame.py", line 9568, in apply
return op.apply().__finalize__(self, method="apply")
File "/var/lang/lib/python3.11/site-packages/pandas-1.5.3-py3.11-linux-x86_64.egg/pandas/core/apply.py", line 764, in apply return self.apply_standard()
File "/var/lang/lib/python3.11/site-packages/pandas-1.5.3-py3.11-linux-x86_64.egg/pandas/core/apply.py", line 891, in apply_standard results, res_index = self.apply_series_generator()
File "/var/lang/lib/python3.11/site-packages/pandas-1.5.3-py3.11-linux-x86_64.egg/pandas/core/apply.py", line 907, in apply_series_generator results[i] = self.f(v)
File "/var/task/lambda_code.py", line 350, in <lambda> if x.glims_is_validation is True
File "/var/lang/lib/python3.11/site-packages/pandas-1.5.3-py3.11-linux-x86_64.egg/pandas/core/generic.py", line 5902, in __getattr__ return object.__getattribute__(self, name) | [ERROR] AttributeError: 'Series' object has no attribute 'glims_is_validation' Traceback (most recent call last):
File "/var/task/lambda_code.py", line 1788, in lambda_handler processing_df = submit_libraries_to_pieriandx(processing_df)
File "/var/task/lambda_code.py", line 348, in submit_libraries_to_pieriandx processing_df["submission_arn"] = processing_df.apply(
File "/var/lang/lib/python3.11/site-packages/pandas-1.5.3-py3.11-linux-x86_64.egg/pandas/core/frame.py", line 9568, in apply return op.apply().__finalize__(self, method="apply")
File "/var/lang/lib/python3.11/site-packages/pandas-1.5.3-py3.11-linux-x86_64.egg/pandas/core/apply.py", line 764, in apply return self.apply_standard()
File "/var/lang/lib/python3.11/site-packages/pandas-1.5.3-py3.11-linux-x86_64.egg/pandas/core/apply.py", line 891, in apply_standard results, res_index = self.apply_series_generator()
File "/var/lang/lib/python3.11/site-packages/pandas-1.5.3-py3.11-linux-x86_64.egg/pandas/core/apply.py", line 907, in apply_series_generator results[i] = self.f(v)
File "/var/task/lambda_code.py", line 350, in <lambda> if x.glims_is_validation is True
File "/var/lang/lib/python3.11/site-packages/pandas-1.5.3-py3.11-linux-x86_64.egg/pandas/core/generic.py", line 5902, in __getattr__ return object.__getattribute__(self, name)
Error message
Circular dependency between resources: [cttsoicatopieriandxprodlimsmakerlambdastacklfBF1A5A5E, cttsoicatopieriandxprodlimsmakerlambdastacklftrigAllowEventRuleCttsoIcaToPieriandxPipelineStackProdcttsoicatopieriandxprodLimsMakerLambdaStagecttsoicatopieriandxprodlimsmakerlambdastackcttsoicatopieriandxprodlimsmakerlambdastacklfA1060767738057BF, cttsoicatopieriandxprodlimsmakerlambdastacklftrigCA91B158, cttsoicatopieriandxprodlimsmakerlambdastacklfEventInvokeConfig2D261CF4, cttsoicatopieriandxprodlimsmakerlambdastackLambdaExecutionRoleDefaultPolicy9FF70FF7, cttsoicatopieriandxprodlimsmakerlambdastackssmcdklambdaeventruleparameterC1114276, cttsoicatopieriandxprodlimsmakerlambdastackssmcdklambdaparameter157B45B6
[ERROR] KeyError: 'pieriandx_submission_time'
Traceback (most recent call last):
File "/var/task/lambda_code.py", line 1703, in lambda_handler cleanup_duplicate_rows(merged_df, cttso_lims_df, excel_row_number_mapping_df)
File "/var/task/lambda_code.py", line 1366, in cleanup_duplicate_rows
merged_df_dedup = bind_pieriandx_case_submission_time_to_merged_df(merged_df_dedup, cttso_lims_df)
File "/var/task/lambda_code.py", line 1464, in bind_pieriandx_case_submission_time_to_merged_df pieriandx_case_submission_time = row['pieriandx_submission_time']
File "/var/lang/lib/python3.11/site-packages/pandas-1.5.3-py3.11-linux-x86_64.egg/pandas/core/series.py", line 981, in __getitem__ return self._get_value(key)
File "/var/lang/lib/python3.11/site-packages/pandas-1.5.3-py3.11-linux-x86_64.egg/pandas/core/series.py", line 1089, in _get_value loc = self.index.get_loc(label)
File "/var/lang/lib/python3.11/site-packages/pandas-1.5.3-py3.11-linux-x86_64.egg/pandas/core/indexes/base.py", line 3804, in get_loc raise KeyError(key) from err
Related to aws/aws-cdk#26128 (comment)
Batch Submission was successful
2023-07-08 01:08:47,906 - INFO - cttso-ica-to-pieriandx - main : LineNo. 103 - Creating case object on PierianDx for case SBJ00595_L2300857_001
--
But then next iteration of LIMS got case none for this subject / library combination (about 10 mins later)
[INFO] 2023-07-08T01:56:26.123Z c32b6ae0-2b70-4d5a-822e-625e6695be6e Got case 'None' for pending analysis SBJ00595 L2300857
Over the weekend we had the cttso lims launch 16 samples via 33 analyses.
Here is the table of launches from the lims script against subject id / library id combinations with each cell representing the pieriandx accession id generated
18 March 1pm | 18 March 2pm | 18 March 3pm | |
---|---|---|---|
SBJ03127_L2300341 | SBJ03127_L2300341_001 | SBJ03127_L2300341_002 | SBJ03127_L2300341_003 |
SBJ03130_L2300344 | SBJ03130_L2300344_001 | SBJ03130_L2300344_002 | SBJ03130_L2300344_003 |
SBJ03132_L2300346 | SBJ03132_L2300346_001 | SBJ03132_L2300346_002 | SBJ03132_L2300346_003 |
SBJ03133_L2300347 | SBJ03133_L2300347_001 | SBJ03133_L2300347_002 | SBJ03133_L2300347_003 |
SBJ03128_L2300342 | SBJ03128_L2300342_001 | SBJ03128_L2300342_002 | SBJ03128_L2300342_003 |
SBJ03129_L2300343 | SBJ03129_L2300343_001 | SBJ03129_L2300343_002 | SBJ03129_L2300343_003 |
SBJ03137_L2300351 | SBJ03137_L2300351_001 | SBJ03137_L2300351_002 | SBJ03137_L2300351_003 |
SBJ00596_L2300355 | SBJ00596_L2300355_001 | SBJ00596_L2300355_002 | |
SBJ03140_L2300354 | SBJ03140_L2300354_001 | SBJ03140_L2300354_002 | |
SBJ03141_L2300356 | SBJ03141_L2300356_001 | SBJ03141_L2300356_002 | |
SBJ03136_L2300350 | SBJ03136_L2300350_001 | SBJ03136_L2300350_002 | |
SBJ03138_L2300352 | SBJ03138_L2300352_001 | SBJ03138_L2300352_002 | |
SBJ03139_L2300353 | SBJ03139_L2300353_001 | SBJ03139_L2300353_002 | |
SBJ03135_L2300349 | SBJ03135_L2300349_001 | ||
SBJ03134_L2300348 | SBJ03134_L2300348_001 | ||
SBJ03131_L2300345 | SBJ03131_L2300345_001 |
It may not be possible as to why this occurred. Could this have been something like GSheets not returning the full dataframe to pandas? Or likewise with pieriandx case endpoint? Is there a better way to ensure that pending samples are updated? Should these be updated before other sources are merged?
Research sample that is not validation but has no redcap entry should be placed through the validation pipeline (not the clinical pipeline), fix code that is causing this issue.
Would be a useful metric to be able to look back on to make sure all library submission types are as expected.
This means the lambda images are unable to deploy
Panel Type not being considered when submission lambda.
APIError: {'code': 503, 'message': 'The service is currently unavailable.', 'status': 'UNAVAILABLE'}
Traceback (most recent call last):
File "/var/task/lambda_code.py", line 1464, in lambda_handler
glims_df: pd.DataFrame = get_cttso_samples_from_glims()
File "/var/lang/lib/python3.10/site-packages/lambda_utils-0.0.1-py3.10.egg/lambda_utils/gspread_helpers.py", line 93, in get_cttso_samples_from_glims
glims_df: pd.DataFrame = Spread(spread=get_glims_sheet_id(), sheet="Sheet1").sheet_to_df(index=0)
File "/var/lang/lib/python3.10/site-packages/gspread_pandas-3.2.2-py3.10.egg/gspread_pandas/spread.py", line 387, in sheet_to_df
vals = self.sheet.get_all_values()
File "/var/lang/lib/python3.10/site-packages/gspread-5.8.0-py3.10.egg/gspread/utils.py", line 701, in wrapper
return f(*args, **kwargs)
File "/var/lang/lib/python3.10/site-packages/gspread-5.8.0-py3.10.egg/gspread/worksheet.py", line 452, in get_all_values
return self.get_values(**kwargs)
File "/var/lang/lib/python3.10/site-packages/gspread-5.8.0-py3.10.egg/gspread/utils.py", line 701, in wrapper
return f(*args, **kwargs)
File "/var/lang/lib/python3.10/site-packages/gspread-5.8.0-py3.10.egg/gspread/worksheet.py", line 425, in get_values
return fill_gaps(self.get(range_name, **kwargs))
File "/var/lang/lib/python3.10/site-packages/gspread-5.8.0-py3.10.egg/gspread/utils.py", line 701, in wrapper
return f(*args, **kwargs)
File "/var/lang/lib/python3.10/site-packages/gspread-5.8.0-py3.10.egg/gspread/worksheet.py", line 818, in get
response = self.spreadsheet.values_get(range_name, params=params)
File "/var/lang/lib/python3.10/site-packages/gspread-5.8.0-py3.10.egg/gspread/spreadsheet.py", line 182, in values_get
r = self.client.request("get", url, params=params)
File "/var/lang/lib/python3.10/site-packages/gspread_pandas-3.2.2-py3.10.egg/gspread_pandas/util.py", line 305, in request
raise error
File "/var/lang/lib/python3.10/site-packages/gspread_pandas-3.2.2-py3.10.egg/gspread_pandas/util.py", line 292, in request
return ClientV4.request(client, *args, **kwargs)
File "/var/lang/lib/python3.10/site-packages/gspread-5.8.0-py3.10.egg/gspread/client.py", line 92, in request
raise APIError(response)
From
https://github.com/umccr/cttso-ica-to-pieriandx/actions/runs/5505905943/workflow#L35
Should be
${{
steps.get_all_changed_files.outputs.any_changed == 'true' &&
( steps.get_all_changed_files.outputs.all_changed_files != steps.get_deployment_changed_files.outputs.all_changed_files )
}}
line_terminator is now lineterminator in pandas version 2.0.0. This is a breaking change for PierianDx submission.
Related to #46
Few steps to fix this:
This can result in duplicate batch submissions
Old base image is causing some errors in deployment
<html>
<body>
<!--StartFragment-->
Installing into a conda env
--
306 | Traceback (most recent call last):
307 | File "/opt/conda/bin/mamba", line 7, in <module>
308 | from mamba.mamba import main
309 | File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 51, in <module>
310 | from mamba import repoquery as repoquery_api
311 | File "/opt/conda/lib/python3.9/site-packages/mamba/repoquery.py", line 9, in <module>
312 | from mamba.utils import init_api_context, load_channels
313 | File "/opt/conda/lib/python3.9/site-packages/mamba/utils.py", line 17, in <module>
314 | from conda.core.index import _supplement_index_with_system, check_allowlist
315 | ImportError: cannot import name 'check_allowlist' from 'conda.core.index' (/opt/conda/lib/python3.9/site-packages/conda/core/index.py)
<!--EndFragment-->
</body>
</html>
Updates from #25 means mypy is out of sync with boto3, instead use pip-review with pip-reqs to update all dependencies
Currently sitting on version 2.39, may require a bit of work for beta modules like aws-batch-alpha
CannotStartContainerError: Error response from daemon: failed to initialize logging driver: failed to create Cloudwatch log stream: AccessDeniedException: User: arn:aws:sts::843407916570:assumed-role/cttso-ica-to-pieriandx-de-cttsoicatopieriandxdevba-E60G
Remove the following lines
## DEBUG
if subject_id == "SBJ00595" and library_id.startswith("L2100178"):
print(mini_df)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.