• secrets mgmt solution #3

• minimal or 0 autoloading #1

• pip installable packages

• core

  • logs
  • errors #5
  • osinfo
  • executors? local/remote

• data manipulation

  • cache kvstore (use redis)
  • lrucache: use python lru
  • hashers: wrappers around popular hashings md5, sha256,512, blake2
  • serializers: python objects from/to json, yaml, toml
  • idgenerators: guids, randomchars, names, passwds, bytes, capnpids
  • regex: wrap re main functions and provide most used regexes in code
  • tar/zip manipulation
  • datetime: use arrow library facilities
  • jinja2 easier interface

• sals:

  • process: execute binary and get its output
  • tmux
  • bashprofile
  • filesystem
  • hostsfiles
  • btrfs
  • disklayout info about the mounts and lsblk
  • netconfig? too much
  • nettools: all ip common/checking connections
  • ssh/d configs
  • nginx/caddy
  • qemu img
  • ssl/tls to generate certificates
  • rsync
  • nics? not netconfig?
  • openvswitch
  • docker

servers/
TBD

• tools/

  • sync between machines
  • object inspector (won't be needed.)
  • locale
  • memusage
  • performance trace
  • path ? python Path?
  • reporter what exactly?
  • teammgr?
  • timer
  • sandboxer
  • googleslides
  • expect automation
  • dns tools
  • docker

• clients:

  • coredns
  • etcd
  • redis
  • blockchain
  • gdrive
  • git
  • github
  • gitea
  • google_compute
  • digitalocean
  • http (around requests)
  • itsyou.online
  • mail (sendgrid)
  • mongo
  • mysql
  • postgres
  • s3
  • sonic
  • sshkey
  • sshagent
  • syncthing
  • tarantool?
  • trello
  • virtualbox
  • zdb
  • zero-os
  • zerotier? wireguard
  • kraken
  • traefik
  • tfgateway

What are the core functionalities provided by jumpscale?

Problem 1: Code Structure

We can use namespaces the way zope does with zope.interface for example.
Now we can easily have jumpscale.sal , jumpscale.clients easily separated into there own pip packages like any normal python project. (bye bye installtools!)

declaring namespaces

One of two ways

1- remove __init__.py
2- add namespace declaration in __init__.py __import__('pkg_resources').declare_namespace(__name__)

Example structure

.
├── projectclients
│   └── jumpscale
│       ├── clients
│       │   ├── github.py
│       │   ├── gogs.py
│       │   ├── __init__.py
│       ├── __init__.py

├── projectmain
│   └── jumpscale
│       ├── god.py
│       ├── __init__.py
├── projectsals
│   └── jumpscale
│       ├── __init__.py
│       └── sal
│           ├── fs.py
│           ├── __init__.py
└── projecttools
    └── jumpscale
        ├── __init__.py
        └── tools
            ├── __init__.py
            └── sync.py

Problem 2: implicit imports (autoloading subpackages)

jumpscale by design wants the least amount of imports thats why all are registered under j

e.g for sal/__init__.py

import pkgutil

__all__ = []
for loader, module_name, is_pkg in pkgutil.walk_packages(__path__):
    __all__.append(module_name)
    _module = loader.find_module(module_name).load_module(module_name)
    globals()[module_name] = _module

CHECK: lazyloader/import hooks in stdpython
https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader

Problem 3: Singletons

Solved by design using python modules

Problem 4: jslocation

If we open god.py

import jumpscale.sal
import jumpscale.tools
import jumpscale.clients

j = jumpscale

we have handcrafted imports for sal, tools, clients so their subpackages can be autoloaded, but how should it work with packages like digitalme

How to register digitalme in the god object

Do we generate import jumpscale.digitalme? is there a standard python way to do it? a reliable plugin system?

where would its module be registered?

for instance there might be digitalme.tools should it be under j.tools directly or j.digitalme.tools? I prefer the latter for clarity and conflict resolution too

Example usage with jumpscale

~> export PYTHONPATH="projecttools:projectsals:projectclients:projectmain"

In [1]: import jumpscale.god                                                                                     

In [2]: jumpscale.sal.fs.copyfile('a', 'b')                                                                      
copying file

In [3]: jumpscale.tools.sync.sync()                                                                              
sync tool

In [4]: jumpscale.clients.github.get_githubclient?                                                               
Signature: jumpscale.clients.github.get_githubclient(username, password)
Docstring: <no docstring>
File:      ~/wspace/jumpscale-skeleton/projectclients/jumpscale/clients/github.py
Type:      function

In [5]: jumpscale.clients.github.get_githubclient('a', 'bb')                                                     
getting client with a bb

In [6]: jumpscale.clients.gogs.get_gogs('a', 'bbb') # uses jumpscale sal.fs in its code                          
sync tool
getting gogs client with a bbb

Example usage with j

In [1]: from jumpscale.god import j                                                                              

In [2]: j.sal.fs.removefile('a')                                                                                 
removing file

In [3]: j.clients.gogs.get_gogs('a', 'b')                                                                        
sync tool
getting gogs client with a b

                                                                                                                 

jumpscale is meant to be used by single user to when it comes to using facilities that require credentials/secrets e.g github client requires a user and a token, mysql client requires a user and a password but anyhow these secrets shouldn't be exposed as plain text

requirements:

• don't expose secrets in scripts
• user can have multiple configurations (e.g multiple github accounts)
  most likely the way to go is using public key cryptography to encrypt configurations file for users and only decrypt it when they have the private key
• don't consider with shared secrets for groups if it's needed maybe we should consider using vault

one of the pain points is the jumpscale dirs; with the proposed architecture most of them won't be needed at all

• TMPDIR: not needed, python tempfile.
• CFGDIR: should be XDG_CONFIG_HOME or ~/.config/jumpscale
• CODEDIR: not needed anymore with pip packages we can develop install against anywhere in the system.
• LOGDIR: should be default log dir or ~/.logs/jumpscale or keep config and logs under ~/.jumpscale
• HOMEDIR: pathlib.Path.home() "cross platform" or expanduser("~") on unix

for sandboxed systems: pip install X --user is enough to have contained packages relative to user

To be build a new Jumpscale framework that share the same features and ideas, we need to focus on the main core features of jumpscale and build the minimal working version with these feature.
The main components are:

• JS0 loader
  This is the main entry point of the system, it will be responsible of initializing the environment and load all the other components
• JS0 Config Manager
  The config manager will allow users to create different configurations for different services/systems and safely and securely save these configurations for later usage
• JS0 Logger and error handling
  Manages the logs and provide APIs to handle errors and report them.
• JS0 Plugin manager
  Allows the system to be extensible, this to allow different SALS and Clients to be developed independently and work out of the box once installed

things like jshell, synctool, etc can be managed using entrypoints console_script entry https://packaging.python.org/specifications/entry-points/

• all exceptions should be consistent (can be achieved using a base exception) in the way the look
• logged to the filesystem
• logged too into redis with json format so we would be able to utilize redis-desktop-manager