Code Monkey home page Code Monkey logo

tscat_gui's Introduction

sciqlop_logo


What Is SciQLop?

SciQLop (SCIentific Qt application for Learning from Observations of Plasmas) is a powerful and user-friendly tool designed for the visualization and analysis of in-situ space plasma data.

Using SciQLop will let you:

  • have a super easy access to tens of thousands of products from the top main data archives in the world,
  • explore multivariate time series effortlessly, with lightning-fast and transparent downloads as you scroll, zoom in, and zoom out,
  • visualize custom products with simple python code executed on-the-fly,
  • easily label time intervals and make or edit catalog of events graphically and rapidely,
  • analyze your data in jupyter notebooks,
sciqlop_logo

Heliophysicists now benefit from decades of space exploration through many spacecraft missions. Exploring this massive amount of data to find events of interest, build catalogs, and conduct statistical multi-mission data analysis can be a daunting task if not having the right tool.

SciQLop aims at being this tool! A simple lightweight yet powerful graphical interface coupled to the limitless options brought by the Jupyter Notebook integration, that focuses on providing users with the easiest possible way to explore, label and analyze huge amounts of data. SciQLop is also the right tool for teaching space physics and in situ spacecraft data handling to students effortlessly.

Main Features

  • Interactive and responsive: SciQLop can handle millions of data points without compromising on interactivity. Users can scroll, zoom, move, and export plots with ease.

    SciQLop smooth navigation
  • User-friendly: Accessing data in SciQLop is as simple as a drag and drop from the tens of thousands of products readily available. Custom user products defined in Python behave exactly the same way and bring infinite possibilities.

    SciQLop drag and drop
  • Jupyter notebook integration: SciQLop can be used as a backend for Jupyter notebooks, allowing users to create and manipulate plots from within their notebooks, define new products and much more.

    SciQLop Jupyter integration
  • Catalogs: SciQLop provides a catalog system that allows users to easily label events in their data or visualize existing catalogs of events.

    SciQLop catalogs
  • Evolving and growing list of examples: SciQLop comes with a growing list of examples that demonstrate how to perform common tasks such as loading data, creating plots, and using the catalog system.

    SciQLop examples

Upcoming features

  • community-driven plugins repository: SciQLop will soon have a plugin system that will allow users to extend the software's capabilities by installing community-driven plugins.
  • catalogs coediting: SciQLop will allow users to coedit catalogs, making it easier to collaborate on event labeling and visualization, thereby also improving reproducibility of space physics studies.

How to install SciQLop

Mac Users

Since SciQLop 0.7.1 we produce a Mac App Bundle that you can download from the latest release page just pick the right architecture for your Mac (ARM64 for Apple M1/2/3 chips and x86_64 for intel ones).

Linux Users

If you are using a Linux distribution, you may not need to install anything, you can just download the AppImage from the latest release and run it (after making it executable).

From sources

Warning: Due to this issue you should not use any * Python* version higher than 3.10.x.

Since SciQLop depends on specific versions of PySide6 you should use a dedicated virtualenv for SciQLop to avoid any conflict with any other Python package already installed in your system.

  • Using releases from PyPi
python -m pip install sciqlop
  • Using the latest code from GitHub
python -m pip install git+https://github.com/SciQLop/SciQLop

Once installed the sciqlop launcher should be in your PATH and you should be able to start SciQLop from your terminal.

sciqlop

or

python -m SciQLop.app

Experimental Python API Examples:

The following API examples are for early adopters and will likely change a bit in the future!

  • Creating plot panels:
from SciQLop.backend import TimeRange
from datetime import datetime

# all plots are stacked
p = main_window.new_plot_panel()
p.time_range = TimeRange(datetime(2015, 10, 22, 6, 4, 30).timestamp(), datetime(2015, 10, 22, 6, 6, 0).timestamp())
p.plot("speasy/cda/MMS/MMS1/FGM/MMS1_FGM_BRST_L2/mms1_fgm_b_gsm_brst_l2")
p.plot("speasy/cda/MMS/MMS1/DIS/MMS1_FPI_BRST_L2_DIS_MOMS/mms1_dis_bulkv_gse_brst")
p.plot("speasy/cda/MMS/MMS1/DIS/MMS1_FPI_BRST_L2_DIS_MOMS/mms1_dis_energyspectr_omni_brst")

# tha_peif_sc_pot and tha_peif_en_eflux will share the same plot 
p2 = main_window.new_plot_panel()
p2.plot("speasy/cda/THEMIS/THA/L2/THA_L2_ESA/tha_peif_en_eflux")
p2.plots[0].plot("speasy/cda/THEMIS/THA/L2/THA_L2_ESA/tha_peif_sc_pot")
p2.plot("speasy/cda/THEMIS/THA/L2/THA_L2_ESA/tha_peif_velocity_dsl")

NOTE: An easy way to get product paths, is to drag a product from Products Tree to any text zone or even your Python terminal.

  • Custom products:
from datetime import datetime

import numpy as np

from SciQLop.backend.pipelines_model.easy_provider import EasyVector, EasyScalar


# The following functions can do anything from loading data from a file to any complex computation, they should not 
# block the GUI since they will be run in background.

def my_vect(start: datetime, stop: datetime) -> (np.ndarray, np.ndarray):
    x = np.arange(start.timestamp(), stop.timestamp(), 0.1) * 1.
    y = np.empty((len(x), 3))
    y[:, 0] = np.cos(x / 100.) * 10.
    y[:, 1] = np.cos((x + 100) / 100.) * 10.
    y[:, 2] = np.cos((x + 200) / 100.) * 10.
    return x, y


def my_scalar(start: datetime, stop: datetime) -> (np.ndarray, np.ndarray):
    x = np.arange(start.timestamp(), stop.timestamp(), 0.1) * 1.
    y = np.empty((len(x), 1))
    y[:, 0] = np.cos(x / 100.) * 10.
    return x, y


my_vector_provider = EasyVector(path='some_root_folder/my_hello_world_vector', get_data_callback=my_vect,
                                components_names=["x", "y", "z"], metadata={})
my_scalar_provider = EasyScalar(path='some_other_root_folder/my_hello_world_scalar', get_data_callback=my_scalar,
                                component_name="x", metadata={})

More examples can be found in the examples folder, they are also available from the welcome screen.

How to contribute

Just fork the repository, make your changes and submit a pull request. We will be happy to review and merge your changes. Reports of bugs and feature requests are also welcome.

Credits

The development of SciQLop is supported by the CDPP.
We acknowledge support from the federation Plas@Par

tscat_gui's People

Contributors

dependabot[bot] avatar jeandet avatar pboettch avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

Forkers

pboettch jeandet

tscat_gui's Issues

filtered events and manually added events should appear differently

events can be associated with a dynamic catalog for two reasons:

  • because it satisfy the selection predicate applied on the whole database
  • because it's been manually assigned to this catalog.

when displaying such a catalog it is not obvious anymore which event belong to which category and so what to expect when changing the filter.

A solution could be to represent events originating from a filter and those from manual assignment differently. Could be a color (maybe a little too much) or a little icon in a column to indicate its origin.

Catalogue TreeModel

  • accessing via tscat
  • returning catalogues (also dynamic ones)
  • Catalogue become visible in TreeView

export/import crash

Description

Export a catalogue to json (about 100k events)
import the catalog from that json file
the import takes forever (much longer than the creation) and eventually crashes

log :

sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (sqlite3.IntegrityError) UNIQUE constraint failed: catalogues.uuid
[SQL: INSERT INTO catalogues (uuid, name, author, predicate, tags, removed, attributes) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('a09e4b86-21a6-4cfb-b59c-579f3e54fb4b', 'THEMIS_MPcrossings_2007-2016_V1', 'toto', None, '', 0, '{}')]
(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)
Traceback (most recent call last):
  File "/home/jeandet/Documents/prog/GH-stuff/tscat_gui/tscat_gui/model.py", line 172, in data
    return item.text()
  File "/home/jeandet/Documents/prog/GH-stuff/tscat_gui/tscat_gui/model.py", line 65, in text
    return get_entity_from_uuid_safe(self._uuid).name
  File "/home/jeandet/Documents/prog/GH-stuff/tscat_gui/tscat_gui/utils/helper.py", line 14, in get_entity_from_uuid_safe
    catalogues = tscat.get_catalogues(tscat.filtering.UUID(uuid))
  File "/home/jeandet/.local/lib/python3.10/site-packages/tscat/__init__.py", line 302, in get_catalogues
    for cat in backend().get_catalogues(base_dict):
  File "/home/jeandet/.local/lib/python3.10/site-packages/tscat/orm_sqlalchemy/__init__.py", line 229, in get_catalogues
    for c in self._create_query(base, orm.Catalogue, 'events', removed=base['removed']):
  File "/home/jeandet/.local/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2900, in __iter__
    return self._iter().__iter__()
  File "/home/jeandet/.local/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2907, in _iter
    result = self.session.execute(
  File "/home/jeandet/.local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1711, in execute
    conn = self._connection_for_bind(bind)
  File "/home/jeandet/.local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind
    return self._transaction._connection_for_bind(
  File "/home/jeandet/.local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind
    self._assert_active()
  File "/home/jeandet/.local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active
    raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (sqlite3.IntegrityError) UNIQUE constraint failed: catalogues.uuid
[SQL: INSERT INTO catalogues (uuid, name, author, predicate, tags, removed, attributes) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('a09e4b86-21a6-4cfb-b59c-579f3e54fb4b', 'THEMIS_MPcrossings_2007-2016_V1', 'toto', None, '', 0, '{}')]
(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)

Export/import formats

today tscat only allows export to JSON.
exporting in other formats such as VOTable could be useful

catalog provider tree

It would be nice to have some kind of tree for catalog providers as we have for data providers in sciqlop. Users would benefit from browsing graphically catalogs from AMDA, either shared or private.

Shared catalogs (from AMDA) could be set to read-only. Editing them would require to duplicate them locally.

Create EditWidget

Create EditWidget: dynamic creation of attribute-widget (one per type) (Might be longer than one day)

drag and drop

  • events should be movable with drag and drop, for instance to move or copy them into a catalog
  • catalog should be movable with D&D too

filter appearance rework

today the GUI shows: "filter" and a text field next to it.
It would probably be better to just have the text field with "catalog to filter" or something like that, written in italic, inside the text box, and when the text box is clicked the text disappears and only reappears if the text box is empty and unselected

External control with Signals and Slots

Signals:

  • event_selected(uuid)
  • catalogue_selected(uuid)
  • event_changed(uuid)
  • catalogue_changed(uuid)

Slot:

  • update_event_range(uuid, start, stop)
  • create_event(start, stop, author, catalogue_uuid)
  • move_to_trash(uuid)
  • save()

Handle events

Display, edits, move-to-trash, restore, delete events

Add refresh button

Need a button to refresh the whole view while keeping everything selected as it is.

cannot undo "delete permanently"

delete permanently a catalogue (it was empty does it matter?)
the undo option is available but it does not put the catalogue back

bulk selection

it should be possible to bulk edit events or catalog and perform operations on them.
Two operations at least:

  • move/copy them somewhere (export, trash, catalog, etc.)
  • edit parameters in the property panel

bulk selection of different objects in nature such as catalog AND events could in principle be possible if there is an intersection in their property to edit, but not part of this issue.

re-work the "property panel"

  • put section titles (custom and global) in bold or something that let them easily distinguished from field names
  • maybe remove the UUID, the user does not care
  • rename the name of the different fields so to be more straightforward what they should be used for. For instance the "global" section title is not clear what is "global" (vs "local"?). "Mandatory" is perhaps more accurate but also a bit verbose, maybe we could just put no section name for those and just name the custom section?
  • could the type of the custom attribute(?) be inferred rather than asking the user to choose it? what happens if I wrongly choose string and put "2" for instance ?

GUI design

GUI design (main-menu, toolbar, TreeViews (empty), edit-area) - based on GUI maquette. - without any content.

selecting a catalog after selecting an event from that catalog

selecting a catalog shows its properties
selecting an event from that catalog shows the event properties
re-selecting the (same) catalog leaves the properties of the event whereas it should show again those of the catalog
selecting another catalog, however, shows the properties of that catalog OK.

Event list sorting

Sorting by tags, attributes, products is possible but it is not intuitive what it does or even what it should do.
Probably should not be possible.

export catalog via drag&drop

use case: select a catalog, drag&drop it into my email or my desktop, etc. exports it as the default format in config (e.g. votable)

show catalog "meta data"

users want to know some information about catalogs such as the number of events, the min start and max start dates, missions concerned if any given in attributes, etc. (TBD)

these info should be accessible in several ways:

  • as a contextual info when the on focus
  • in the property panel in some section named like "metadata"
  • the number of events could appear on the right of the catalog name or at the end of the line

Edit Dynamic Catalogue

Catalogue predicate is shown in attribute-list, clicking edit will open a model dialog allow to create/edit as list of simple-predicates combined with either AnyOf or AllOf.

delete icons

Icons for "Delete" and "delete permanently" are misleading.
There could be only one maybe with a little arrow on the side to select the mode either "move to trash" or "delete permanently", with the "move to trash" options checked by default?

Add a real entry point

Actually we test tscat_gui by launching examples/tscat-gui.py, this should be turned into a real entry point inside tscat-gui package.

manipulating graphically large catalogs (100k) is tedious

  • clicking on the catalog first takes several seconds to load all the events and display them before the catalog appears to be selected graphically, which gives an awkward feeling something is wrong
  • selecting several catalogs one after the other either by clicking or via the arrow key is not fluid at all if one or several large catalogs are selected since each of them blocks the interface for several seconds before other selections are processed.

event rating

since signatures in the data motivating the creation of an event of some kind are highly variable, it is often useful to set some kind of quality rating to an event. Like 0 means "I select it but it's far from the textbook signature, I'm unsure this is XXX" and 10 "textbook like signature, I'm 100% confident this is XXXX".

Selecting events based on their rating, via for example the dynamic filter, allows the user to perform statistics on a more or less conservative approach.

Rating could be for instance represented as 5 stars to check as an additional column in the event list.

catalog tree view?

should users be able to create folders and a kind of tree view for its own local catalogs, as for example exists for distant catalogs on AMDA?

Push catalog to AMDA

Users should be able to push a catalog to its user space in AMDA. This would erase the distant catalog if existing (probably we need a confirmation before doing it?) and create a new one if none existed before.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.