korniichuk / rk Goto Github PK
View Code? Open in Web Editor NEWThe remote Jupyter kernel/kernels administration utility
Home Page: http://www.korniichuk.com
License: The Unlicense
The remote Jupyter kernel/kernels administration utility
Home Page: http://www.korniichuk.com
License: The Unlicense
I usually set ssh to listen to different ports than the default 22, so is there a way to specify the port number?
@korniichuk any thoughts what's going wrong here?
Michaels-MacBook:~ michael$ jupyter notebook
[I 09:56:43.382 NotebookApp] Serving notebooks from local directory: /Users/michael
[I 09:56:43.382 NotebookApp] 0 active kernels
[I 09:56:43.382 NotebookApp] The IPython Notebook is running at: http://localhost:8888/
[I 09:56:43.382 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 09:56:57.407 NotebookApp] Creating new notebook in
[I 09:56:58.257 NotebookApp] Kernel started: 8cab0703-0950-4334-96c6-334f3a3f2d56
[W 09:56:58.362 NotebookApp] 404 GET /kernelspecs/desktop%20ipython%203/logo-64x64.png (::1) 31.53ms referer=http://localhost:8888/notebooks/Untitled3.ipynb?kernel_name=desktop%20ipython%203
Traceback (most recent call last):
File "/Applications/anaconda/bin/rkscript", line 230, in
paramiko_tunnel(v, remote_ports[k], remote_username_at_remote_host)
File "/Applications/anaconda/lib/python2.7/site-packages/rk/ssh/tunnel.py", line 320, in paramiko_tunnel
if not _try_passwordless_paramiko(server, keyfile):
File "/Applications/anaconda/lib/python2.7/site-packages/rk/ssh/tunnel.py", line 118, in _try_passwordless_paramiko
client.load_system_host_keys()
File "/Applications/anaconda/lib/python2.7/site-packages/paramiko/client.py", line 101, in load_system_host_keys
self._system_host_keys.load(filename)
File "/Applications/anaconda/lib/python2.7/site-packages/paramiko/hostkeys.py", line 101, in load
e = HostKeyEntry.from_line(line, lineno)
File "/Applications/anaconda/lib/python2.7/site-packages/paramiko/hostkeys.py", line 331, in from_line
key = RSAKey(data=decodebytes(key))
File "/Applications/anaconda/lib/python2.7/site-packages/paramiko/rsakey.py", line 64, in init
if msg.get_text() != 'ssh-rsa':
File "/Applications/anaconda/lib/python2.7/site-packages/paramiko/message.py", line 186, in get_text
return u(self.get_bytes(self.get_int()))
File "/Applications/anaconda/lib/python2.7/site-packages/paramiko/py3compat.py", line 53, in u
return s.decode(encoding)
File "/Applications/anaconda/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf7 in position 1: invalid start byte
[1437] Warning: unhandled RemoteError: Traceback (most recent call last):
File "", line 1072, in executetask
File "", line 1, in do_exec
File "", line 66, in
File "D:\Users\Michael\Anaconda3\lib\site-packages\traitlets\config\application.py", line 587, in launch_instance
app = cls.instance(**kwargs)
File "D:\Users\Michael\Anaconda3\lib\site-packages\traitlets\config\configurable.py", line 415, in instance
'%s are being created.' % cls.name
traitlets.config.configurable.MultipleInstanceError: Multiple incompatible subclass instances of IPKernelApp are being created.
[W 09:57:08.386 NotebookApp] Timeout waiting for kernel_info reply from 8cab0703-0950-4334-96c6-334f3a3f2d56
[I 09:57:10.257 NotebookApp] KernelRestarter: restarting kernel (1/5)
WARNING:root:kernel 8cab0703-0950-4334-96c6-334f3a3f2d56 restarted
^C[I 09:57:13.542 NotebookApp] interrupted
Serving notebooks from local directory: /Users/michael
1 active kernels
The IPython Notebook is running at: http://localhost:8888/
Shutdown this notebook server (y/[n])? y
[C 09:57:14.778 NotebookApp] Shutdown confirmed
[I 09:57:14.779 NotebookApp] Shutting down kernels
[I 09:57:15.825 NotebookApp] Kernel shutdown: 8cab0703-0950-4334-96c6-334f3a3f2d56
So... I've been racking my brain about this and can't figure it out, please help?
I've seen the video on how to set it up... I am even getting connection errors when I specify a bad name in kernel.json
. For example it asks me for my password if I specify the wrong user... all fine. Then I open up the terminal and do ipython console --kernel fionn
, where fionn
is (from kernel.json
):
{
"argv": [
"rkscript",
"python3",
"{connection_file}",
"[email protected]"
],
"display_name": "Fionn",
"language": "python"
}
I had to change one thing in rkscript
, that is, I'm calling: makegateway("popen//service .profile//ssh=%s//python=%s"...
, without the popen//service .profile//
it wouldn't work cause this machine uses loaded modules and a lot of environment variable setting up. Anyway... one more proof that I am connected properly, because without popen//service .profile
it wouldn't work (so I think it's a good practice to read the .profile
or .bashrc
or both...) But then I go to the console and write import sys; sys.version
and I get 3.4.3
which is the version on my local computer!... the office machine to which I want to connect to and run the kernel has 3.4.0
. What am I missing here?!... Please help! (I've also tried adding popen//service .profile
after ssh=%s
, but that didn't change anything...
edit: oh yeah... to be sure I also checked sys.path
so this returns the local path... instead of the office machine path.
edit: I noticed one more thing. I'm getting: Port forwarding stopped uncleanly: [Errno 98] Address already in use
. Might this mean that the connection isn't properly set up and I end up with the commands being executed on my local machine?
some more details
So the way I connect to my work machine is through VPN and another machine so, my .ssh/config
looks like this:
ControlMaster auto
ControlPath /home/razvan/.ssh/tmp/%h_%p_%r
Host office
HostName IP-MACHINE-1
Host fionn
ProxyCommand ssh razvan@office nc -q0 IP-MACHINE-2 %p
LocalForward 8888 127.0.0.1:8888
And I connect to fionn
to work...
wouldn't it be possible to save kernelspecs in ~/.ipython/kernels
?
I've followed the setup steps, and confirmed that I can log in to my remote host without a password, but when I run Jupyter and switch to the remote kernel, I get the following:
Traceback (most recent call last):
File "/usr/local/bin/rkscript", line 218, in <module>
remote_ports = ch.receive()
File "/usr/local/lib/python3.4/site-packages/execnet/gateway_base.py", line 711, in receive
raise self._getremoteerror() or EOFError()
execnet.gateway_base.RemoteError: Traceback (most recent call last):
File "<string>", line 1039, in executetask
File "<string>", line 1, in do_exec
File "<remote exec>", line 11, in <module>
ImportError: No module named IPython.kernel.zmq.kernelapp
Perhaps a PYTHONPATH issue?
Running Python 3.4 on OX X 10.11 locally.
While trying to use rk, I run into the following problems:
module_location = join(getusersitepackages(), module_name)
Did it work eventually ? Nope. Now I'm facing an strange issue. I can ssh into the remote machine without being asked for passphrase or password since the authentication is based on kerberos. Strangely enough, rk asks me first my passphrase, then my password and both eventually fail. Not sure what's causing this. Here is part of the log:
[I 12:04:27.899 NotebookApp] KernelRestarter: restarting kernel (4/5)
WARNING:root:kernel a251e120-4f06-485e-938f-5e60c1e11505 restarted
/usr/lib/python3.5/getpass.py:92: GetPassWarning: Can not control echo on the terminal.
passwd = fallback_getpass(prompt, stream)
Warning: Password input may be echoed.
remote's password: Traceback (most recent call last):
File "/usr/lib/python3.5/getpass.py", line 70, in unix_getpass
old = termios.tcgetattr(fd) # a copy to save
termios.error: (25, 'Inappropriate ioctl for device')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "~/.local/bin/rkscript", line 230, in <module>
paramiko_tunnel(v, remote_ports[k], remote_username_at_remote_host)
File "~/.local/lib/python3.5/site-packages/rk/ssh/tunnel.py", line 321, in paramiko_tunnel
password = getpass("%s's password: "%(server))
File "/usr/lib/python3.5/getpass.py", line 92, in unix_getpass
passwd = fallback_getpass(prompt, stream)
File "/usr/lib/python3.5/getpass.py", line 127, in fallback_getpass
return _raw_input(prompt, stream)
File "/usr/lib/python3.5/getpass.py", line 149, in _raw_input
raise EOFError
EOFError
[5897] Warning: unhandled RemoteError: Traceback (most recent call last):
File "<string>", line 1072, in executetask
File "<string>", line 1, in do_exec
File "<remote exec>", line 55, in <module>
PermissionError: [Errno 13] Permission denied: '~/.ipython/profile_default/security/kernel-a251e120-4f06-485e-938f-5e60c1e11505.json'
Cheers.
from site import getsitepackages
returns an ImportError
when ran in a virtualenv. It seems to be an open bug since 2012โฆ
Is there an easy workaround, to be able to launch jupyter-notebook with remote kernels, from within a virtualenv?
Thanks
alter@BICB260:~/rk$ pip install rk
Collecting rk
Using cached rk-0.3b1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/qm/b_1n3bn15wb373zvkvd1vcp4yjd86m/T/pip-build-ki70csci/rk/setup.py", line 39, in <module>
long_description = open(join(dirname(__file__), "README.rst")).read(),
File "/Users/balter/miniconda3/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2208: ordinal not in range(128)
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/qm/b_1n3bn15wb373zvkvd1vcp4yjd86m/T/pip-build-ki70csci/rk/
I know what the documentation says about this issue however my remote server does require a password and there is no way I can change that (ssh keys or other auth means are not supported). My users will be ok with it.
Is there any technical reasons (such as this) that apply here and would prevent this from working? I see most of the ssh invocations are happening via pexpect and that should be able to handle the password, however I've never done it. If you think this could work, I'll be happy to work on a PR
Hi,
I tried to run rk install-template
on Windows 7 and it failed with the following error:
C:\Anaconda>rk install-template
Traceback (most recent call last):
File "c:\anaconda\lib\runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "c:\anaconda\lib\runpy.py", line 72, in _run_code
exec code in run_globals
File "C:\Anaconda\Scripts\rk.exe\__main__.py", line 5, in <module>
File "c:\anaconda\lib\site-packages\rk\rk.py", line 7, in <module>
from os import link, listdir, makedirs, remove, strerror
ImportError: cannot import name link
(As you can see, I'm using Anaconda distribution)
Is the utility is compatible with Windows? What would you recommend?
Cheers,
Denis
First of all, great package, that's exactly what I was looking for. I still have an issue with using the remote conda environment: I am not sure the conda environment on the remote is used properly.
Ideally I would like to source it but the only option I found was to indicate the path of the remote conda environment python binary in kernel spec.
So when importing sys in the local notebook using the remote kernel, i get the correct environment python version (3.5):
sys.version
`3.5.1`
But clearly, the environment variables are not right (conda environment was clearly not sourced):
!python --version
Python 2.7.11
Is there a way to properly source the whole remote conda environment instead of just using its associated python binary?
Starting a remote kernel (as per you example) or even running rkscript
manually always results in this error:
Traceback (most recent call last):
File "/root/.local/bin/rkscript", line 110, in <module>
create_messages()
File "/root/.local/bin/rkscript", line 103, in create_messages
config_messages_rel_path = config["config_messages_rel_path"]
File "/root/.local/lib/python2.7/site-packages/configobj.py", line 554, in __getitem__
val = dict.__getitem__(self, key)
KeyError: 'config_messages_rel_path'
Installed rk via pip-2.7 in a FreeBSD 10.3 jail
Just a few sentences would be helpful. I got here by a link, and I'm not sure if it solves the needs I have.
Hi,
I followed the instructions, using iPython 3.1.0. When creating the new "Template" notebook, it will connect for a few seconds, and then begin a loop between "Dead kernel" and "Connected" notifications.
Thank you in advance.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.