Adding extensions to the .NET kernel using nuget packages for F# about interactive HOT 10 CLOSED

Goodness … this is actually quite hard … the goal is to load up and execute an assembly and all of it's dependencies in a process.

It should be noted that nuget packages can:
1. Service multiple target frameworks,
E.g. net472, net48, netstandard1.6, netstandard2.0, netcoreapp1.0, netcoreapp2.1 .... and frameworks as yet unspecified.

2.0 Can target different platforms:
I.e. x86, x64, anycpu32bitpreferred, or anycpu

3.0 Can include OS specific native code dlls
I.e. MacOS, Linux, Windows.

We will need to be crisp, if we are going to support any of these scenarios with extension loading.
For example F# interactive and compiler supports multiple target frameworks, but only anycpu il assemblies. Therefore we do not differentiate, platform specific libraries and OS specific libraries. (* It is amusing that F# considers word size platform a hangover from msbuild in the old days I suppose *)
This behaviour has already shipped for type providers, and has been continues with compiler tools.

Because F# supports multiple platforms the directory structure in a TP looks like:

$PackageRoot\
  lib\
        netstandard2.0\
               MyReferencedAssembly.dll
  typeproviders\
        net462\
               MyReferencedDesignTime.dll
        netstandard2.0\
               MyReferencedDesignTime.dll
        netcoreapp2.1\
               MyReferencedDesignTime.dll

The type provider has a referenced assembly in lib\netstandard2.0. And design time add-ins that can work on net462, netstandard2.0 and netcoreapp2.1 platforms.

However, because of how probing works the same package could be shipped like this:
With the tools for a particular library, being packaged as a subdirectory of the library.

I personally think the below should have been an issue, but we shipped this way for a couple of years now. However, I can see why developers kind of like it. It makes debugging simpler, not requiring the developer to actually build the nuget package.

$PackageRoot\
  lib\
        netstandard2.0\
               MyReferencedAssembly.dll
               typeproviders\
                    net462\
                          MyReferencedDesignTime.dll
                    netstandard2.0\
                          MyReferencedDesignTime.dll
                    netcoreapp2.1\
                          MyReferencedDesignTime.dll

FSharp probing is described here: https://github.com/fsharp/fslang-design/blob/master/tooling/FST-1003-loading-type-provider-design-time-components.md

The one change we are going to make for FSharp compiler extensions is that as well as probing the typeproviders folder we will also probe the tools folder.

F# probing requires the presence of the net462, netstandard2.0 or netcoreapp2.1 sub folders.

Having the TFM (for execution) is a two edged sword, it means we can have platform specific TP's and compiler extensions, however, it means we need to know in advance what TFM (for execution) the executing program supports. It looks to me as if the proposal for .NET kernel is only going to support NetStandard2.0 since you don't appear to probe for specific tfms.

I like the scripting proposal, although we will probably allow, independent scripts per TFM, enabling the flexibility of having a windows initialization script and a netcoreapp2.1 specific initialization script.

from interactive.

cc @eerhardt

from interactive.

We might want to differentiate the types of extensions by using specific paths.

Following the example of Roslyn analyzers (which are found under {package_root}\analyzers\dotnet\cs), we might want to differentiate by language. For example:

{package_root}\interactive-extensions\dotnet\cs -> Assemblies containing extensions for CSharpKernel
{package_root}\interactive-extensions\dotnet\fs -> Assemblies containing extensions for FSharpKernel

But language-agnostic extensions would also be possible:

{package_root}\interactive-extensions\dotnet -> Assemblies containing extensions for CompositeKernel

On another dimension, we've also discussed supporting extension via scripts, which, if found in the conventional package location, would be loaded using #load. This might look something like:

{package_root}\interactive-extensions\scripts\dotnet\cs -> #load {script_path} sent to CSharpKernel
{package_root}\interactive-extensions\scripts\dotnet\fs -> #load {script_path} sent to FSharpKernel

(These are always language-specific).

And on yet another dimension, some scripts might be intended for use only in specific hosts:

{package_root}\interactive-extensions\scripts\dotnet\cs\main.csx -> #load {script_path} sent to CSharpKernel regardless of host
{package_root}\interactive-extensions\scripts\dotnet\cs\jupyter.csx -> #load {script_path} sent to CSharpKernel when host is Jupyter
{package_root}\interactive-extensions\scripts\dotnet\cs\repl.csx -> #load {script_path} sent to CSharpKernel when host is REPL

from interactive.

@tmat

from interactive.

As @KevinRansom mentioned, for now we are supporting netstandard2.0 only and we have merged a first step towards that (#396 ) where we are probing the nuget package for the path interactive-extensions/dotnet/cs and loading extensions if any are present.

from interactive.

Hi, one question on extension, how can install the kernel extensions? If I have a nuget or an assembly with the extension, is there any command to install it and used in a notebook without call #r nuget:xx .?

from interactive.

Currently, the way to install an extension is via #r nuget.

You can see the whole process, including source code, build, and installation, in this notebook: https://github.com/dotnet/interactive/blob/master/NotebookExamples/Extensions.ipynb

Is there an alternative approach that you would find useful?

from interactive.

Currently, the way to install an extension is via #r nuget.

You can see the whole process, including source code, build, and installation, in this notebook: https://github.com/dotnet/interactive/blob/master/NotebookExamples/Extensions.ipynb

Is there an alternative approach that you would find useful?

It is useful but how I can load it like #!time directive that is loaded by the kernel at start time and not by #r directive?

from interactive.

#r nuget has functionality to scan the loaded package for extensions. If an extension dll is found in the correct location in the package, it's scanned for implementations of IKernelExtension and the handling IKernel is passed in. At that point, you can programmatically modify the kernel using the APIs in the Microsoft.DotNet.Interactive package.

You can see how those APIs work here:

from interactive.

Closing this as the current extension mechanism is simpler, by design. The kernel extension is not language or platform specific. The kernel can implement additional requirements via dependent packages if needed. We can revisit the issue as additional scenarios come up.

from interactive.