Code Monkey home page Code Monkey logo

sciqlop's Introduction

sciqlop_logo


What Is SciQLop?

SciQLop (SCIentific Qt application for Learning from Observations of Plasmas) is a powerful and user-friendly tool designed for the visualization and analysis of in-situ space plasma data.

Using SciQLop will let you:

  • have a super easy access to tens of thousands of products from the top main data archives in the world,
  • explore multivariate time series effortlessly, with lightning-fast and transparent downloads as you scroll, zoom in, and zoom out,
  • visualize custom products with simple python code executed on-the-fly,
  • easily label time intervals and make or edit catalog of events graphically and rapidely,
  • analyze your data in jupyter notebooks,
sciqlop_logo

Heliophysicists now benefit from decades of space exploration through many spacecraft missions. Exploring this massive amount of data to find events of interest, build catalogs, and conduct statistical multi-mission data analysis can be a daunting task if not having the right tool.

SciQLop aims at being this tool! A simple lightweight yet powerful graphical interface coupled to the limitless options brought by the Jupyter Notebook integration, that focuses on providing users with the easiest possible way to explore, label and analyze huge amounts of data. SciQLop is also the right tool for teaching space physics and in situ spacecraft data handling to students effortlessly.

Main Features

  • Interactive and responsive: SciQLop can handle millions of data points without compromising on interactivity. Users can scroll, zoom, move, and export plots with ease.

    SciQLop smooth navigation
  • User-friendly: Accessing data in SciQLop is as simple as a drag and drop from the tens of thousands of products readily available. Custom user products defined in Python behave exactly the same way and bring infinite possibilities.

    SciQLop drag and drop
  • Jupyter notebook integration: SciQLop can be used as a backend for Jupyter notebooks, allowing users to create and manipulate plots from within their notebooks, define new products and much more.

    SciQLop Jupyter integration
  • Catalogs: SciQLop provides a catalog system that allows users to easily label events in their data or visualize existing catalogs of events.

    SciQLop catalogs
  • Evolving and growing list of examples: SciQLop comes with a growing list of examples that demonstrate how to perform common tasks such as loading data, creating plots, and using the catalog system.

    SciQLop examples

Upcoming features

  • community-driven plugins repository: SciQLop will soon have a plugin system that will allow users to extend the software's capabilities by installing community-driven plugins.
  • catalogs coediting: SciQLop will allow users to coedit catalogs, making it easier to collaborate on event labeling and visualization, thereby also improving reproducibility of space physics studies.

How to install SciQLop

Mac Users

Since SciQLop 0.7.1 we produce a Mac App Bundle that you can download from the latest release page just pick the right architecture for your Mac (ARM64 for Apple M1/2/3 chips and x86_64 for intel ones).

Linux Users

If you are using a Linux distribution, you may not need to install anything, you can just download the AppImage from the latest release and run it (after making it executable).

From sources

Warning: Due to this issue you should not use any * Python* version higher than 3.10.x.

Since SciQLop depends on specific versions of PySide6 you should use a dedicated virtualenv for SciQLop to avoid any conflict with any other Python package already installed in your system.

  • Using releases from PyPi
python -m pip install sciqlop
  • Using the latest code from GitHub
python -m pip install git+https://github.com/SciQLop/SciQLop

Once installed the sciqlop launcher should be in your PATH and you should be able to start SciQLop from your terminal.

sciqlop

or

python -m SciQLop.app

Experimental Python API Examples:

The following API examples are for early adopters and will likely change a bit in the future!

  • Creating plot panels:
from SciQLop.backend import TimeRange
from datetime import datetime

# all plots are stacked
p = main_window.new_plot_panel()
p.time_range = TimeRange(datetime(2015, 10, 22, 6, 4, 30).timestamp(), datetime(2015, 10, 22, 6, 6, 0).timestamp())
p.plot("speasy/cda/MMS/MMS1/FGM/MMS1_FGM_BRST_L2/mms1_fgm_b_gsm_brst_l2")
p.plot("speasy/cda/MMS/MMS1/DIS/MMS1_FPI_BRST_L2_DIS_MOMS/mms1_dis_bulkv_gse_brst")
p.plot("speasy/cda/MMS/MMS1/DIS/MMS1_FPI_BRST_L2_DIS_MOMS/mms1_dis_energyspectr_omni_brst")

# tha_peif_sc_pot and tha_peif_en_eflux will share the same plot 
p2 = main_window.new_plot_panel()
p2.plot("speasy/cda/THEMIS/THA/L2/THA_L2_ESA/tha_peif_en_eflux")
p2.plots[0].plot("speasy/cda/THEMIS/THA/L2/THA_L2_ESA/tha_peif_sc_pot")
p2.plot("speasy/cda/THEMIS/THA/L2/THA_L2_ESA/tha_peif_velocity_dsl")

NOTE: An easy way to get product paths, is to drag a product from Products Tree to any text zone or even your Python terminal.

  • Custom products:
from datetime import datetime

import numpy as np

from SciQLop.backend.pipelines_model.easy_provider import EasyVector, EasyScalar


# The following functions can do anything from loading data from a file to any complex computation, they should not 
# block the GUI since they will be run in background.

def my_vect(start: datetime, stop: datetime) -> (np.ndarray, np.ndarray):
    x = np.arange(start.timestamp(), stop.timestamp(), 0.1) * 1.
    y = np.empty((len(x), 3))
    y[:, 0] = np.cos(x / 100.) * 10.
    y[:, 1] = np.cos((x + 100) / 100.) * 10.
    y[:, 2] = np.cos((x + 200) / 100.) * 10.
    return x, y


def my_scalar(start: datetime, stop: datetime) -> (np.ndarray, np.ndarray):
    x = np.arange(start.timestamp(), stop.timestamp(), 0.1) * 1.
    y = np.empty((len(x), 1))
    y[:, 0] = np.cos(x / 100.) * 10.
    return x, y


my_vector_provider = EasyVector(path='some_root_folder/my_hello_world_vector', get_data_callback=my_vect,
                                components_names=["x", "y", "z"], metadata={})
my_scalar_provider = EasyScalar(path='some_other_root_folder/my_hello_world_scalar', get_data_callback=my_scalar,
                                component_name="x", metadata={})

More examples can be found in the examples folder, they are also available from the welcome screen.

How to contribute

Just fork the repository, make your changes and submit a pull request. We will be happy to review and merge your changes. Reports of bugs and feature requests are also welcome.

Credits

The development of SciQLop is supported by the CDPP.
We acknowledge support from the federation Plas@Par

sciqlop's People

Contributors

itlink-hackathon avatar jeandet avatar meriadegp avatar nicolasaunai avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

sciqlop's Issues

align spectros with time series plots

most pragmatic way : remove colorbar, acceptable for now
other possibility : collapsable colorbar default ON
other : colorbar on top or bottom and possibly collapsable

Partial virtual products

With SciQLop, we have an abstraction for products like the ones provided by speasy (such as Themis A FGM GSM data) and also what we call "virtual products". Virtual products are functions that takes a time range and returns plotable data such as any computation based on a fixed set of other products.

Now we would like to add the possibility to expose generic functions that could work with any compatible products.
This means, we have to expose an API to define and check function <-> product compatibility. This will most likely be based on product unit and shape. We also have to define GUI interactions such as how do we instantiate a partial virtual product?

Closing welcome panel causes crash

  • SciQLop version: c356ca3
  • Python version:3.10
  • Operating System:Mac OS 13.3 (ventura) and above

Description

Closing the welcome panel and adding a new plot panel leads to the app crashing.

Plot templates

When looking at data it is often that we browse the same quantities in the same plot layout. Opening sciqlop, it is a bit annoying to have to re-make that from the data tree all over again.

It would be nice to have the possibility to save a plot panel as a template. Templates would store the variables used, the plot properties (layout, order, which variable is on which plot etc.), not necessarily the time interval. The idea is to be able to save templates and browse them, have favorites...

Orbit viewer

The user can specify the dates from which getting the data by selecting the position of the spacecraft on a GUI.

Main app window resized automatically

  • SciQLop version: c356ca3 (but has appeared since a long time ago)
  • Python version: 3.10
  • Operating System: Mac OS 13.3

Description

When not in full screen:
The main application window is often resized so that the bottom appears hidden behind the dock.
There is perhaps a way to stop it where the dock is.

Log panel window stuck

  • SciQLop version:c356ca3
  • Python version:3.10
  • Operating System:mac os 13.3

Description

Log widget window is stuck.

What I Did

undocked the log widget and cannot grab it anymore, it's stuck, I can only hide it and make it reappear as is:

image

event marker appears only on 1 plot panel

  • SciQLop version: c356ca3
  • Python version: 3.10
  • Operating System:Mac OS 13.3

Description

Catalog events appear only on 1 plot panel

What I Did

  • make 2 plot panels
  • jump on 1 event on panel 1
  • jump on the same event on panel 2
  • still on jump mode on panel 2, switch the view to panel 1 and zoom or pan, the event marker disappears while expected to stay there

Display / edition of event metadata

This issue is about showing the event metadata directly from its colored-span representation on plot.
A possible way is to show it on focus in a contextual window.
Another way could be to display it in a "setting panel" when user clicks on the event.
In the former, information would be just read-only, while in the latter event properties could be edited.

vertical cursors

button to switch into vertical cursor mode
aligned on all plots of the same panel.

truncated plot legend

image

Depending on the monitor size and number of plot in the panel, the legend could be truncated as on the above capture.
Two solutions maybe:

  • change layout from lines to columns (see matplotlib), or "best" if possible
  • allow the legend to be collapsed and scrolled (not great)...

event multi-selection and bulk-edition

This issue is partially for SciQLOP and TsCat_Gui.

The point is to allow the multi-selection of events:

  • from the plots colored spans
  • from the event list on the SciQLOP catalog view panel
  • from TsCat gui.

and be able from any of these selections, to perform bulk editions, like ratings, or whatever.

Bonus: In the plot area in particular, it'd be nice to be able to select multiple events currently on the windows without having to shift/ctrl+click on them, like a lasso kind of selection.

Split logs

At least jupyterlab logs should not be mixed with SciQLop ones, jupyterlab is way too verbose compared to SciQLop.

virtual product registration duplicate paths

  • SciQLop version:
  • Python version:3.10
  • Operating System:fedora

Description

funcs = [mms1_position, mms2_position, mms3_position, mms4_position]


for i,func in enumerate(funcs):
    EasyScalar(path=f"mms_vprods/position_earth_radii_{i}",
        get_data_callback=func,
        component_name=["x","y","z"],
        metadata={},
)

the above code registers 4 virtual products sharing the beginning of their path but duplicate the "root" directory "mms_vprods" in the gui.

image

I would expect 1 directory for all products.

Add a task manager like QTC

Operations like packages install while loading a workspace can be quite long, the user needs to know that dependencies aren't installed yet.

delete virtual products

  • SciQLop version:
  • Python version:
  • Operating System:

Description

I would like to be able to delete virtual products that have been created in the gui.

create/delete/move horizontal and vertical markers

  • in horizontal or vertical cursor mode, a double clic in the plot drops a marker
  • a clic on an existing marker selects it
  • a selected marker(s) can be deleted by pressing suppr
  • a selected marker(s) can be moved
  • multiple markers can be selected (shift+clic)
  • the time (resp. value on Y axis) is written next to the marker as it follows the cursor before being dropped

plots are movable

drag and drop enabled for plots

  • rearrange plots in same (or different ?) panels
  • export plots as image outside sciqlop

App crash

  • SciQLop version: 0.5.4
  • Python version: 3.12
  • Operating System: macOS m2

Description

I tried to drag variables into the window like what the screenshot did, but got no response. Also the app closed whenever I tried to click the plus button or open a notebook and run.

It seems like a Qt/OS problem. But I had no experience with that.

What I Did

RuntimeError: Event loop already running
doh set to ""  --  SystemOnly
js: Language pack 'en_US' not installed!
js: Failed to fetch ipywidgets through the "jupyter.widget.control" comm channel, fallback to fetching individual model state. Reason: Control comm was closed too early
js: Connection lost, reconnecting in 0 seconds.
Compositor returned null texture
Compositor returned null texture
*** Terminating app due to uncaught exception 'NSRangeException', reason: '*** -[__NSArrayM objectAtIndexedSubscript:]: index 0 beyond bounds for empty array'
*** First throw call stack:
(
        0   CoreFoundation                      0x000000018c752800 __exceptionPreprocess + 176
        1   libobjc.A.dylib                     0x000000018c249eb4 objc_exception_throw + 60
        2   CoreFoundation                      0x000000018c6c8fc0 -[__NSCFString hasSuffix:] + 0
        3   libqcocoa.dylib                     0x0000000141476044 _ZN20QCocoaSystemTrayIcon13emitActivatedEv + 274796
        4   libqcocoa.dylib                     0x00000001414760c8 _ZN20QCocoaSystemTrayIcon13emitActivatedEv + 274928
        5   libqcocoa.dylib                     0x0000000141474688 _ZN20QCocoaSystemTrayIcon13emitActivatedEv + 268208
        6   QtWidgets                           0x0000000125ef11a8 _ZN9QListView16selectionChangedERK14QItemSelectionS2_ + 204
        7   QtCore                              0x0000000120eab48c _ZN11QMetaObject8activateEP7QObjectPKS_iPPv + 3808
        8   QtCore                              0x0000000121071e44 _ZN19QItemSelectionModel20emitSelectionChangedERK14QItemSelectionS2_ + 216
        9   QtCore                              0x0000000121071a2c _ZN19QItemSelectionModel6selectERK14QItemSelection6QFlagsINS_13SelectionFlagEE + 636
        10  QtCore                              0x000000012107170c _ZN19QItemSelectionModel6selectERK11QModelIndex6QFlagsINS_13SelectionFlagEE + 68
        11  QtCore                              0x00000001210728cc _ZN19QItemSelectionModel15setCurrentIndexERK11QModelIndex6QFlagsINS_13SelectionFlagEE + 260
        12  QtWidgets                           0x0000000125d4f338 _ZNK9QComboBox5countEv + 828
        13  QtWidgets                           0x0000000125d528b4 _ZN9QComboBoxC1ER16QComboBoxPrivateP7QWidget + 2412
        14  QtCore                              0x0000000120eab224 _ZN11QMetaObject8activateEP7QObjectPKS_iPPv + 3192
        15  QtCore                              0x000000012105a944 _ZN18QAbstractItemModel13endInsertRowsEv + 200
        16  QtGui                               0x000000012455f1a8 _ZN18QStandardItemModel11itemChangedEP13QStandardItem + 844
        17  QtWidgets                           0x0000000125d55e38 _ZN9QComboBox11insertItemsEiRK5QListI7QStringE + 340
        18  QtWidgets.abi3.so                   0x0000000126ac590c _ZL26Sbk_QComboBoxFunc_addItemsP7_objectS0_ + 156
        19  python3.12                          0x0000000100ed0670 cfunction_vectorcall_O + 296
        20  python3.12                          0x0000000100e746e4 PyObject_Vectorcall + 88
        21  python3.12                          0x0000000100f8dcc8 _PyEval_EvalFrameDefault + 51728
        22  python3.12                          0x0000000100e77e8c method_vectorcall + 560
        23  libpyside6.abi3.6.6.dylib           0x0000000104b66750 _ZN6PySide13SignalManager20callPythonMetaMethodERK11QMetaMethodPPvP7_objectb + 536
        24  libpyside6.abi3.6.6.dylib           0x0000000104b66318 _ZN6PySide13SignalManager20SignalManagerPrivate16qtMethodMetacallEP7QObjectiPPv + 760
        25  QtCore                              0x0000000120eab1f4 _ZN11QMetaObject8activateEP7QObjectPKS_iPPv + 3144
        26  libpyside6.abi3.6.6.dylib           0x0000000104b66284 _ZN6PySide13SignalManager20SignalManagerPrivate16qtMethodMetacallEP7QObjectiPPv + 612
        27  libpyside6.abi3.6.6.dylib           0x0000000104b64f78 _ZN6PySide12MetaFunction4callEP7QObjectiP7_objectPS4_ + 1380
        28  QtCore.abi3.so                      0x00000001209924e8 _ZL20Sbk_QObjectFunc_emitP7_objectS0_ + 316
        29  python3.12                          0x0000000100ecfcd8 cfunction_call + 308
        30  python3.12                          0x0000000100e74954 _PyObject_Call + 184
        31  libpyside6.abi3.6.6.dylib           0x0000000104b79454 _ZL18signalInstanceEmitP7_objectS0_ + 728
        32  python3.12                          0x0000000100e84abc method_vectorcall_VARARGS + 204
        33  python3.12                          0x0000000100e746e4 PyObject_Vectorcall + 88
        34  python3.12                          0x0000000100f8dcc8 _PyEval_EvalFrameDefault + 51728
        35  libpyside6.abi3.6.6.dylib           0x0000000104b66750 _ZN6PySide13SignalManager20callPythonMetaMethodERK11QMetaMethodPPvP7_objectb + 536
        36  libpyside6.abi3.6.6.dylib           0x0000000104b698b8 _ZN6PySide16GlobalReceiverV211qt_metacallEN11QMetaObject4CallEiPPv + 672
        37  QtCore                              0x0000000120eab1f4 _ZN11QMetaObject8activateEP7QObjectPKS_iPPv + 3144
        38  QtGui                               0x00000001245cf0a4 _ZN7QAction8activateENS_11ActionEventE + 368
        39  QtWidgets                           0x0000000125d38cf0 _ZNK15QAbstractButton11isCheckableEv + 344
        40  QtWidgets                           0x0000000125d39c48 _ZN15QAbstractButton17mouseReleaseEventEP11QMouseEvent + 172
        41  QtWidgets                           0x0000000125e37d90 _ZN11QToolButton17mouseReleaseEventEP11QMouseEvent + 20
        42  QtWidgets                           0x0000000125c858b8 _ZN7QWidget5eventEP6QEvent + 132
        43  QtWidgets                           0x0000000125e384ec _ZN11QToolButton5eventEP6QEvent + 208
        44  QtWidgets                           0x0000000125c3c594 _ZN19QApplicationPrivate13notify_helperEP7QObjectP6QEvent + 272
        45  QtWidgets                           0x0000000125c3e5c0 _ZN12QApplication6notifyEP7QObjectP6QEvent + 5072
        46  QtCore                              0x0000000120e61474 _ZN16QCoreApplication15notifyInternal2EP7QObjectP6QEvent + 292
        47  QtWidgets                           0x0000000125c3cb34 _ZN19QApplicationPrivate14sendMouseEventEP7QWidgetP11QMouseEventS1_S1_PS1_R8QPointerIS0_Ebb + 808
        48  QtWidgets                           0x0000000125c9a860 _ZNK21QWidgetRepaintManager3rhiEv + 11080
        49  QtWidgets                           0x0000000125c99a6c _ZNK21QWidgetRepaintManager3rhiEv + 7508
        50  QtWidgets                           0x0000000125c3c594 _ZN19QApplicationPrivate13notify_helperEP7QObjectP6QEvent + 272
        51  QtWidgets                           0x0000000125c3d3e8 _ZN12QApplication6notifyEP7QObjectP6QEvent + 504
        52  QtCore                              0x0000000120e61474 _ZN16QCoreApplication15notifyInternal2EP7QObjectP6QEvent + 292
        53  QtGui                               0x00000001242bb9ec _ZN22QGuiApplicationPrivate17processMouseEventEPN29QWindowSystemInterfacePrivate10MouseEventE + 1740
        54  QtGui                               0x0000000124317674 _ZN22QWindowSystemInterface22sendWindowSystemEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE + 408
        55  libqcocoa.dylib                     0x0000000141417d74 qt_plugin_instance + 58768
        56  CoreFoundation                      0x000000018c6ddcfc __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 28
        57  CoreFoundation                      0x000000018c6ddc90 __CFRunLoopDoSource0 + 176
        58  CoreFoundation                      0x000000018c6dda00 __CFRunLoopDoSources0 + 244
        59  CoreFoundation                      0x000000018c6dc5f0 __CFRunLoopRun + 828
        60  CoreFoundation                      0x000000018c6dbc5c CFRunLoopRunSpecific + 608
        61  HIToolbox                           0x0000000196c55448 RunCurrentEventLoopInMode + 292
        62  HIToolbox                           0x0000000196c550d8 ReceiveNextEventCommon + 220
        63  HIToolbox                           0x0000000196c54fdc _BlockUntilNextEventMatchingListInModeWithFilter + 76
        64  AppKit                              0x000000018feb6c54 _DPSNextEvent + 660
        65  AppKit                              0x000000019068cebc -[NSApplication(NSEventRouting) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 716
        66  AppKit                              0x000000018feaa100 -[NSApplication run] + 476
        67  libqcocoa.dylib                     0x00000001414158fc qt_plugin_instance + 49432
        68  QtCore                              0x0000000120e6b160 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE + 540
        69  QtCore                              0x0000000120e61b00 _ZN16QCoreApplication4execEv + 112
        70  QtWidgets.abi3.so                   0x0000000126b872d8 _ZL25Sbk_QApplicationFunc_execP7_object + 60
        71  python3.12                          0x0000000100ed07fc cfunction_vectorcall_NOARGS + 292
        72  python3.12                          0x0000000100e746e4 PyObject_Vectorcall + 88
        73  
libc++abi: terminating due to uncaught exception of type NSException

Selecting events on plot don't work if event extends beyond window

  • SciQLop version: 0.4.1
  • Python version: 3.10
  • Operating System: Linux

Description

Event extends beyond the right side of the window cannot be selected.

image

But if the full event is displayed as
image

then selection works.

I would expect being able to grab events even if they are not fully in the window.
This is not a major issue though.

Plot model

It seems necessary to implement a model to represent SciQLop plots state.
This allows to refactor many bits and simplifies the following operations:

  • represent plot "pipelines" -> we could use a simple tree widget as a start
  • provides a API to edit plot states and properties
  • insert/remove custom filter

speed up events display on data

Currently all events are redrawn when there's any change.
This makes sciqlop very slow on real-world use case, i.e. when having many (>~10events) per window, and most of the time leads to the "SciQLOP does not respond" pop up.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.