Code Monkey home page Code Monkey logo

excel-d's Introduction

excel-d

Actions Status Coverage

Excel API bindings and wrapper API for D

This dub package contains D declarations for the Excel SDK as well as a D wrapper API. This allows programmers to write Excel worksheet functions in D.

Motivation and background for the project can be found here. See also the DConf 2017 lightning talk about excel-d.

Run makedocs.sh to generate the documentation Generated documentation - a work in progress - is available at dpldocs.

A working XLL example can be found in the example directory. Running dub build there will create an XLL (myxll32.xll) that can be loaded in Excel making all of the functions in test/xlld/test_d_funcs.d available to be used in Excel cells. The types are automatically converted between D native types and Excel ones. To build the example: dub build -c example [--arch=x86_mscoff|--arch=x86_64].

For this package to build you will need the Excel SDK xlcall32.lib that can be downloaded from Microsoft. Copying it to the build directory should be sufficient (i.e. when building the example, to the example directory). The library file should be useable as-is, as long as on 32-bit Excel dub build is run with --arch=x86_mscoff to use Microsoft's binary format. If linking with optlink, the file must be converted first. We recommend using link.exe to not need the conversion. On 64 bit Excel just use --arch=x86_64 - no questions of different library formats.

As part of the build a .def file is generated with all functions to be exported by the XLL.

Excel won't load the XLL automatically: this must be done manually in File->Tools->Add-Ins. Click on "Go" for "Excel Add-Ins" (the default) and select your XLL there after clicking on "Browse".

The only difference between building for 32-bit or 64-bit Excel is the arch= option passed to dub. A 32-bit XLL will only work on 32-bit Excel and similarly for 64-bit. You will also need the appropriate 32/64 xlcall32.lib from the Excel SDK to link.

Sample code (see the example directory for more):

    import xlld;

    @Excel(ArgumentText("Array to add"),
           HelpTopic("Adds all cells in an array"),
           FunctionHelp("Adds all cells in an array"),
           ArgumentHelp(["The array to add"]))
    double FuncAddEverything(double[][] args) nothrow @nogc { // nothrow and @nogc are optional
        import std.algorithm: fold;
        import std.math: isNaN;

        double ret = 0;
        foreach(row; args)
            ret += row.fold!((a, b) => b.isNaN ? a : a + b)(0.0);
        return ret;
    }

and then in Excel:

=FuncAddEverything(A1:D20)

Future functionality will include creating menu items and dialogue boxes. Pull requests welcomed.

WARNING: Memory for parameters passed to D functions

Any parameters with indirections (pointers, slices) should NOT be escaped. The memory for those parameters WILL be reused and might cause crashes.

There is support to fail at compile-time if user-written D functions attempt to escape their arguments but unfortunately given the current D defaults requires user intervention. Annotate all D code to be called by Excel with @safe and compile with -dip1000 - all parameters will then need to be scope or the code will not compile.

It is strongly advised to compile with -dip1000 and to make all your functions @safe, or your add-ins could cause Excel to crash.

Function spelling

excel-d will always convert the first character in the D function being wrapped to uppercase since that is the Excel convention.

Variant type Any

Sometimes it is useful for a D function to take in any type that Excel supports. Typically this will happen when receiving a matrix of values where the types might differ (e.g. the columns are date, string, double). To get the expected D type from an Any value, use xlld.wrap.fromXlOper. An example:

double Func(Any[][] values) {
    import xlld.wrap: fromXlOper;
    import std.experimental.allocator: theAllocator;
    foreach(row; values) {
        auto date = row[0].fromXlOper!DateTime(theAllocator);
        auto string_ = row[1].fromXlOper!DateTime(theAllocator);
        auto double_ = row[2].fromXlOper!double(theAllocator);
        // ...
    }
    return ret;
}

Asynchronous functions

A D function can be decorated with the @Async UDA and will be executed asynchronously:

@Async
double AsyncFunc(double d) {
    // long-running task
}

Please see the Microsoft documentation.

Custom enum coversions

If the usual conversions between strings and enums don't work for the user, it is possible to register custom coversions by calling the functions registerConversionTo and registerConversionFrom.

Structs

D structs can be returned by functions. They are transformed into a string representation.

D structs can also be passed to functions. To do so, pass in a 1D array with the same number of elements as the struct in question.

Optional custom memory allocation and @nogc

If you are not familiar with questions of memory allocation, the below may seem intimidating. However it's entirely optional and unless performance and latency are critical to you (or possibly if you are interfacing with C or C++ code) then you do not need to worry about the extra complexity introduced by using allocators. The code in the previous section will simply work.

excel-d uses a custom allocator for all allocations that are needed when doing the conversions between D and Excel types. It uses a different one for allocations of XLOPER12s that are returned to Excel, which are then freed in xlAutoFree12 with the same allocator. D functions that are @nogc are wrapped by @nogc Excel functions and similarly for @safe. However, if returning a value that is dynamically allocated from a D function and not using the GC (such as an array of doubles), it is necessary to specify how that memory is to be freed. An example:

// @Dispose is used to tell the framework how to free memory that is dynamically
// allocated by the D function. After returning, the value is converted to an
// Excel type and the D value is freed using the lambda defined here.
@Dispose!((ret) {
    import std.experimental.allocator.mallocator: Mallocator;
    import std.experimental.allocator: dispose;
    Mallocator.instance.dispose(ret);
})
double[] FuncReturnArrayNoGc(double[] numbers) @nogc @safe nothrow {
    import std.experimental.allocator.mallocator: Mallocator;
    import std.experimental.allocator: makeArray;
    import std.algorithm: map;

    try {
        // Allocate memory here in order to return an array of doubles.
        // The memory will be freed after the call by calling the
        // function in `@Dispose` above
        return Mallocator.instance.makeArray(numbers.map!(a => a * 2));
    } catch(Exception _) {
        return [];
    }
}

This allows for @nogc functions to be called from Excel without memory leaks.

Registering code to run when the XLL is unloaded

Since this library automatically writes xlAutoClose it is not possible to use it to run custom code at XLL unloading. As an alternative XLL writers can use xlld.xll.registerAutoCloseFunc passing it a function or a delegate to be executed when xlAutoClose is called.

About Kaleidic Associates

We are a boutique consultancy that advises a small number of hedge fund clients. We are not accepting new clients currently, but if you are interested in working either remotely or locally in London or Hong Kong, and if you are a talented hacker with a moral compass who aspires to excellence then feel free to drop me a line: laeeth at kaleidic.io

We work with our partner Symmetry Investments, and some background on the firm can be found here:

http://symmetryinvestments.com/about-us/

excel-d's People

Contributors

9il avatar aceawan avatar adamdruppe avatar atilaneves avatar dkorpel avatar geod24 avatar john-colvin avatar kaleidic avatar laeeth avatar razvann7 avatar veelo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

excel-d's Issues

@VBA wrapper

Would be great to have an @vba wrapper as a companion project.
VBA types are pretty simple. Return by reference, strings are char* and length; same for arrays.
Same deal:

  1. generate extern(C) wrappers with marshalling of D types
  2. generate .def file
  3. generate VBA declarations referring to the DLL

That's pretty much it for something useful.
see experimental/excelwrap in kalcore for a simple example

use arsd dox generator to generate github docs pages

with a nice splash screen overview.
initially just start generating docs. later we can add code doc comments
github pages - just create a docs folder in master branch and then enable github docs for this repository

What command to issue for compiling?

Sorry for the silly question, but I've just installed D (dmd and ldc) for the first time. My Excel is 64 bit, and I copied in the example folder the right xlcall32 lib. But the command 'dub build --arch=x86_64' , from the example folder, ends with lot of errors, related to function xlld.wrap.wrap.stringAutoFreeOper not callable

  1. should i use dmd or ldc?
  2. should i first issue a command in the root folder? which?
  3. which is the correct command from the example folder ?

How to format Function Documentation

After registering a function some times the text for the function arguments is not pretty:
image

image

I used the following information for Registry:
@register(ArgumentText("Find A subset from the list which sums to specified value"w),
HelpTopic("Find A subset from the list which sums to specified value"w),
FunctionHelp("Find A subset from the list which sums to specified value"w),
ArgumentHelp(["List, contains list where subsets reside"w,"TargetValue, Goal Value for subset"w,"MaxListSize, Maximum number of subset items allowed (this makes a very large difference in calcualtion time)"w]))

I suppose what I need here is documentation on what goes where and how to format it.

Thanks!

UDF in-sheet IntelliSense

I develop Excel-DNA, an Excel integration project similar to excel-d, but for the .NET languages.

One of the extensions I've made is an add-in that allows in-sheet IntelliSense for UDF functions similar to that displayed for the built-in Excel functions while typing a formula. The IntelliSense extension supports integration by other types of add-ins, not only those made with Excel-DNA. In particular, functions defined in VBA or through the native C API using any other language can also display their function descriptions through the Excel-DNA IntelliSense add-in.

For .xll-based add-ins like those made with excel-d, there are two integration options:

  • Next to MyAddIn.xll, place a file called MyAddIn.xll.intellisense.xml with some xml markup providing the function and argument descriptions. The xml content will be something like this
<IntelliSense xmlns="http://schemas.excel-dna.net/intellisense/1.0">
  <FunctionInfo>
   <Function Name="MyDLanguageFunction" Description="A function implemented in D"
              HelpTopic="http://www.bing.com" >
      <Argument Name="FirstArg" Description="Whatever you want to put in here" />
      <Argument Name="AnotherArg" Description="Actually the second arg" />
    </Function>
    <Function Name="AnotherFunction" Description="A function described in XML"
              HelpTopic="http://www.bing.com" >
      <Argument Name="FirstArg" Description="Whatever you want to put in here" />
      <Argument Name="AnotherArg" Description="Actually the second arg" />
    </Function>
  </FunctionInfo>
</IntelliSense>
  • Alternatively, you can create a special (hidden) function in the .xll add-in which is registered as a UDF with Excel and returns an array with the function registration information, as used in the xlfRegister calls to Excel. The registered function name will include a magic Guid that is derived from the .xll path - that's ow the IntelliSense extension find your info function. Details of the magic Guid, format of the table you should return and a sample implementation of the Guid calculation can be found in the RegistrationInfoGuid project.

Finally, after choosing either of the above options, you need to download the ExcelDna.IntelliSense.xll add-in from the GitHub Releases tab. This is the small add-in that provides the actual UI extensions. The (single-file) add-in can be renamed if you want to, and multiple instances (for example those embedded in other Excel-DNA add-ins) will negotiate among themselves to ensure that only one instance (with the latest version) will be active.

If you give the IntelliSense add-in a try and run into any issues with the D integration, I'd be very happy to help.

Question about speed improving

Currently VLOOKUP on Excel works very slow on big data. Is it's possible to re-implement it in excel-d to improve speed?

can't call @system operStringLength from @safe dup

I'm getting some annoying @safe errors:

C:\Users\jcolvin_sym\AppData\Local\dub\packages\excel-d-0.5.3\excel-d\source\xlld\conv\misc.d(82,44): Error: @safe function xlld.conv.misc.dup!(shared(const(GCAllocator))).dup cannot call @system function xlld.conv.misc.operStringLength!(XLOPER12).operStringLength
C:\Users\jcolvin_sym\AppData\Local\dub\packages\excel-d-0.5.3\excel-d\source\xlld\conv\misc.d(122,8):        xlld.conv.misc.operStringLength!(XLOPER12).operStringLength is declared here
C:\Users\jcolvin_sym\AppData\Local\dub\packages\excel-d-0.5.3\excel-d\source\xlld\conv\from.d(143,27): Error: template instance `xlld.conv.misc.dup!(shared(const(GCAllocator)))` error instantiating
C:\Users\jcolvin_sym\AppData\Local\dub\packages\excel-d-0.5.3\excel-d\source\xlld\conv\from.d(215,37):        instantiated from here: fromXlOper!(Any, shared(const(GCAllocator)))
C:\Users\jcolvin_sym\AppData\Local\dub\packages\excel-d-0.5.3\excel-d\source\xlld\conv\from.d(192,46):        instantiated from here: fromXlOperMultiNumber!(cast(Dimensions)1, Any, shared(const(GCAllocator)))
C:\Users\jcolvin_sym\AppData\Local\dub\packages\excel-d-0.5.3\excel-d\source\xlld\conv\from.d(158,70):        instantiated from here: fromXlOperMulti!(cast(Dimensions)1, Any, shared(const(GCAllocator)))
C:\Users\jcolvin_sym\AppData\Local\dub\packages\excel-d-0.5.3\excel-d\source\xlld\conv\from.d(29,31):        ... (1 instantiations, -v to show) ...
tests\it\main.d(116,33):        instantiated from here: fromXlOper!(Any[][], shared(const(GCAllocator)))

unresolved external symbol xlAutoFree12

Hello @atilaneves
I did the same configuration as in the example and got the following linking issue.
Could you please help me to fix it?

qqq.def : error LNK2001: unresolved external symbol xlAutoClose
qqq.def : error LNK2001: unresolved external symbol xlAutoFree12
qqq.def : error LNK2001: unresolved external symbol xlAutoOpen
.dub\build\excel-debug-windows-x86_64-dmd_2074-3FF21CF9F1977BAFD516469820D312C2\qqq.lib : fatal error LNK1120: 3 unresolved externals
Error: linker exited with status 1120
dmd failed with exit code 1120.

Memory leak

@Excel() 
double[] foo()
{
    import core.memory : GC;
    GC.collect;
    GC.minimize;
    log(GC.stats.usedSize);
    auto ret = new double[](10_000_000);
    log(GC.stats.usedSize);
    return ret;
}

re-run in a single cell (so excel throws away old memory) and take a look at excel memory usage in task manager and the logged GC usage values

Support template functions

It should be possible to write:

double foo(T...)(T args) {
}

And have that wrapped by an Excel function that takes, say, 32 XLOPER12* and forwards accordingly, checking which ones are xlTypeMissing

Excel file example for noobs

Hi Atila,

Could you please add a simple example for of an excel file witch use XLL function call with matrix I/O?

Access violation after a while.

I was working on a "simple" regex tool for excel, i thought it would be nice. however after a while I start to get access violations and things stop working nicely. Any hints would be great.

@Register(ArgumentText("Return Regex Matches in String"w),
          HelpTopic("Returns a regex match if it exists, otherwise it will return empty string. Optional parameter allows for different matchs to be returned"w),
          FunctionHelp("Return Regex Matches in String"w),
          ArgumentHelp(["Input string"w,"Regex string"w,"Sub regex match index"w]))
string URegexMatch(string InputStr,string RegexStr,int matchIdx) nothrow {
    import std.regex;
    import std.conv;

    try{
        auto reg = regex(RegexStr);
        try{
            auto m = matchFirst(InputStr, reg);
            if(m.empty) {
                return "#No Matches";
            } else if ( matchIdx >= 0 && matchIdx < m.length ) {
                return to!string(m[matchIdx]);
            } else {
                return "#No matches at Index: "~to!string(m.length)~" matches";
            }
        } catch(Exception e) {
            return "Input Error";
        }
    } catch(Exception e) {
        return "Bad Regex";
    }
}

steal some ideas for useful features from C++ wrapper XLW

Low priority list of enhancements for some day.
I talked to Joshi (who wrote the XLW framework) years back.
https://blog.adaptiverisk.com/xll/excel/2015/01/21/xlw-part5.html

  1. isEscPressed() function to allow aborting long-running functions.
  2. abortOnEscape(true); to check for escape and abort if so so you don't need to check within a loop (which might not even be in your own code). not worth doing immediately if it will be a pain in neck.
  3. threadsafe attribute if we don't already have it. excel automatically will call functions with this attribute on multiple threads. I think it's just a matter of registration options. I don't think it's easy to introspect code and tell if it's thread safe - function author has to tell our library.
  4. volatile attribute - I think this is at registration time. if we don't already have it.
  5. allow use of Matrices rather than [][]. if you can't make it generic matrix function than use mir arrays.
  6. Showcase example showing integration with python using pyd.
  7. Showcase benchmark to do something useful vs python
  8. Showcase REST example - eg pull data from yahoo, quandl or Federal reserve FRED API
  9. Suppose you have a function that normally returns a double - you should be able to return a string as an error message somehow. XLW uses throw to do that - not sure what makes sense for us.
  10. implement some XLW and PyXLL examples to show ease of use.

Example: xlStats. I haven't checked - probably some error in code.
D example

    import std.algorithm:map,sum;
    import std.range:front;

    @Register(ArgumentText("input range to calculate statistics for"),
        HelpTopic("excel-d"),
        FunctionHelp("calculates mean and variance for input array"),
        ArgumentHelp(["input range to calculate statistics for"]))
    auto xlStats(double[][] inTargetRange)
    {
        auto numCells = (inTargetRange.length > 0) ?
                                     inTargetRange.length * inTargetRange.front.length : 0;
        auto means = inTargetRange.map!(row => row.sum).sum / numCells;
        auto sumSquares = inTargetRange.map!( row => row.map!(cell => cell*cell).sum).sum;
        return [means, sumSquares / numCells - means];
    }
`
Is a bit simpler than the C++ XLW example:
````cpp
 LPXLFOPER EXCEL_EXPORT xlStats(LPXLFOPER inTargetRange) {
        EXCEL_BEGIN;
        XlfOper xlTargetRange(inTargetRange);

        // Temporary variables.
        double averageTmp = 0.0;
        double varianceTmp = 0.0;

        // Iterate over the cells in the incoming matrix.
        for (RW i = 0; i < xlTargetRange.rows(); ++i)
        {
            for (RW j = 0; j < xlTargetRange.columns(); ++j)
            {
                // sums the values.
                double value(xlTargetRange(i,j).AsDouble());
                averageTmp += value;
                // sums the squared values.
                varianceTmp += value * value;
            }
        }
        size_t popSize = xlTargetRange.rows() * xlTargetRange.columns();

        // avoid divide by zero
        if(popSize == 0)
        {
            THROW_XLW("Can't calculate stats on empty range");
        }

        // Initialization of the results Array oper.
        XlfOper result(1, 2);
        // compute average.
        double average = averageTmp / popSize;
        result(0, 0) = average;
        // compute variance
        result(0, 1) = varianceTmp / popSize - average * average;
        return result;
        EXCEL_END;
    }
`

[Blocker] No universal conversion to strings

Motivation

A table contains dates, numbers, and strings (headers, and NaNs).
So it should be convertible to a universal type like string[][] or Variant[][].

How to reproduce

Change the function in the example to FuncBob

Note

Text cell formatting does not help.

testLibrary config broken

lots of dependencies on things that are only defined if testingExcelD is set, plus dependencies on unit-threaded

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.