tanlin2013 / mbl Goto Github PK

Describe the problem

When calling mlflow client to list the artifacts in runs, an error arises and says Access Denied.

Code to reproduce the issue

from mlflow.tracking.client import MlflowClient

client = MlflowClient(tracking_uri="my_uri")

run = client.search_runs(
    experiment_ids="the_id",
    filter_string="some_condition_string",
    run_view_type=ViewType.ACTIVE_ONLY,
)

artifacts = client.list_artifacts(run.info.run_id)

Error Log

An error occurred (AccessDenied) when calling the ListObjectsV2 operation.

What Happened?

Turns out MLFLOW_S3_ENDPOINT_URL pointing to the minio server needs to be set both on server and client side. MLflow doesn't do that part for you.

Reference

mlflow/mlflow#1813

Inverse Participation Ratio (IPR) can be an indicator to many-body localization. An alternative approach is the associated participation entropies (PE) in the 2nd reference.

References

1. https://en.wikipedia.org/wiki/Purity_(quantum_mechanics)#Inverse_Participation_Ratio_(IPR)
2. https://arxiv.org/abs/1411.0660

Reverse the spectrum with an overall constant -1 doesn't seem to help for extracting the highest energy, results didn't converge to a stable region, although the reason requires further thoughts.

In turn, it's better to use a tsdrg variant, which reserve highest energies as each fusing step, and so there is no need to reverse the Hamiltonian with any overall const. We can just feed in the MPO into this tsdrg variant.

Reference

tanlin2013/tnpy@8c36865

Currently we only support distribution on single node (workstation), it will be helpful to work with Slurm cluster.

Possible choices are:

• Use Slurm cluster directly
• Install Kubernetes on Slurm
• Use Shifter container
• Use Singularity container

Reference

1. https://docs.ray.io/en/latest/cluster/slurm.html#python-interface-slurm-scripts
2. https://slurm.schedmd.com/containers.html
3. https://tin6150.github.io/psg/blogger_container_hpc.html#:~:text=Singularity%20work%20completely%20independent%20of%20Docker.&text=Shifter%20primary%20workflow%20is%20to,not%20need%20to%20be%20installed.&text=A%20validated%20image%20containing%20high%20energy%20particle%20physics%20workflow.

To find the proper energy window looking for level statistic, it's essential to rescale the energy spectrum with respect to its upper and lower energy bounds in prior.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• chore(deps): update dependency bandit to v1.7.5
• chore(deps): update dependency flake8-pyproject to v1.2.3
• chore(deps): update dependency safety to v2.3.5
• fix(deps): update dependency awswrangler to v2.20.1
• fix(deps): update dependency pandas to v1.5.3
• chore(deps): update dependency black to v23.10.0
• chore(deps): update dependency cruft to v2.15.0
• chore(deps): update dependency flake8-bugbear to v23.9.16
• chore(deps): update dependency ipython to v8.16.1
• chore(deps): update dependency mypy to v1.6.1
• chore(deps): update dependency nbsphinx to ^0.9.0
• chore(deps): update dependency pymysql to v1.1.0
• chore(deps): update dependency pytest to v7.4.2
• chore(deps): update dependency pytest-cov to v4.1.0
• chore(deps): update dependency pytest-mock to v3.12.0
• chore(deps): update dependency vulture to v2.10
• fix(deps): update dependency constructs to v10.3.0
• fix(deps): update dependency mlflow to v2.7.1
• fix(deps): update dependency modin to ^0.24.0
• fix(deps): update dependency numpy to v1.26.1
• fix(deps): update dependency orjson to v3.9.9
• fix(deps): update dependency pandera to ^0.17.0
• fix(deps): update dependency plotly to v5.17.0
• fix(deps): update dependency ray to v2.7.1
• fix(deps): update dependency streamlit to v1.27.2
• fix(deps): update dependency tqdm to v4.66.1
• fix(deps): update python
• chore(deps): update actions/checkout action to v4
• chore(deps): update actions/setup-node action to v3
• chore(deps): update dependency @types/node to v18
• chore(deps): update dependency commitizen to v3
• chore(deps): update dependency moto to v4
• chore(deps): update dependency pre-commit to v3
• chore(deps): update dependency sphinx to v7
• chore(deps): update dependency sphinx-book-theme to v1
• chore(deps): update dependency ts-node to v10
• chore(deps): update dependency typescript to v5
• chore(deps): update docker/build-push-action action to v5
• chore(deps): update docker/login-action action to v3
• chore(deps): update docker/metadata-action action to v5
• chore(deps): update docker/setup-buildx-action action to v3
• chore(deps): update docker/setup-qemu-action action to v3
• chore(deps): update jest monorepo to v29 (major) (@types/jest, jest, ts-jest)
• fix(deps): update dependency awswrangler to v3
• fix(deps): update dependency pandas to v2
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency protobuf to v3.20.2 [security]
• fix(deps): update dependency aws-cdk-lib to v2.80.0 [security]
• chore(deps): update dependency cryptography to v41 [security]
• chore(deps): update dependency mlflow to v2 [security]
• chore(deps): update minio/mc docker digest to 01c0df5
• chore(deps): update minio/minio docker digest to 8129b85
• chore(deps): update mysql/mysql-server docker digest to 5b40d96
• chore(deps): update dependency @types/node to v10.17.60
• chore(deps): update dependency aws-cdk to v2.102.0
• chore(deps): update dependency boto3 to v1.28.68
• chore(deps): update nginx docker tag to v1.25.2
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Compute the modularity for the following 2 quantities in tSDRG

• Entanglement entropy
• Connected correlation function

See Also

Feature: modularity in tSDRG #8

Reference

https://journals.aps.org/prb/abstract/10.1103/PhysRevB.105.064202?fbclid=IwAR3wpy-h4oJsyUcjQA7y0UinfHo0JviS1uunMyl70F11FvNpa36xzE5UWw0

Add authentication to the load balancer of Mlflow UI.

AWS

• Cognito + MLflow Tracking Dashboard

On-premise workstation

• Basic auth + SSL + MLflow Tracking Dashboard
• SSL + Minio Console
• Hide http path and other sensitive data in nginx.conf through environment vars

Reference

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-authenticate-users.html#cognito-requirements

This comment may help to reduce the time.

python-poetry/poetry#2094 (comment)
https://github.com/john-sandall/poetry-speed-test#run-poetry-in-ci

Background

mlflow/mlflow#5945

Resolved in

https://github.com/mlflow/mlflow/releases/tag/v1.26.1

Blockers

• tensorboard (2.9.1) depends on protobuf (>=3.9.2,<3.20)
• streamlit (1.10.0) depends on protobuf (>=3.12,<4) (https://discuss.streamlit.io/t/streamlit-run-with-protocbuf-error/25632)

• tensorflow/tensorboard#5708
• streamlit/streamlit#4779

TODO

• update mlflow to 1.26.1
• update protobuf >= 4.21

Background

MLflow offers automatic tracking to the code version, but that requires one to execute the program through MLflow Project.

Should we

• Implement the code versioning tags manually, or
• Use MLflow Project instead

Pros & Cons

1. MLflow Project can trigger runs in docker environment, but it's not clear in the documentation how one can elaborate features in docker-compose to here, e.g. network mode, docker volume, shm size, etc.
2. MLflow Project support multistep workflows, apart from current candidate Apache Airflow.
3. This means we have to install MLflow at system level.
4. An alternative way is to set MLflow Project environment as system environment, but actually it runs in docker already. But, though in this way we can track git version automatically, docker version is still required to be retrieved manually.

Reference

1. https://mlflow.org/docs/latest/tracking.html#concepts
2. https://mlflow.org/docs/latest/projects.html#building-multistep-workflows

A lots of panics have appeared in mlflow, ray and datawrangler associated block.

For aws-data-wrangler, moto may help for test.

EDITED

• Since this project is data-oriented, we still run the test against the real one in Athena table, without using moto
• Gap ratio parameter should be tested with the mock data generated from poisson and wigner-dyson distribution
• Absolute offset wasn't correctly constructed
• level_id should be re-generated if in total_sz-constrained mode
• Filter out unexpected negative energy gap after taking diff