versilov / matrex Goto Github PK

I think this code summarizes my problem:

iex(1)> [a|b]= [1,2]
[1, 2]
iex(2)> a
1
iex(3)> b
[2]

Note that b is between brackets because is a list.

iex(4)> [a|b] = [Matrex.new([[1]])|Matrex.new([[2]])]
[#Matrex[1×1]
┌         ┐
│     1.0 │
└         ┘ |
 #Matrex[1×1]
┌         ┐
│     2.0 │
└         ┘]
iex(5)> a
#Matrex[1×1]
┌         ┐
│     1.0 │
└         ┘
iex(6)> b
#Matrex[1×1]
┌         ┐
│     2.0 │
└         ┘

PROBLEM:

I was expecting b to be a list, and equal to:

iex(6)> b
[#Matrex[1×1]
┌         ┐
│     2.0 │
└         ┘]

(Note the extra brackets.)

I'm a complete newb, running an IElixir kernel in JupyterLab notebook, tried out the Boyle package matrex_samples found at Notebook with Boyle examples with usage of Matrex library

All works fine until I get to calling heatmap()

I followed instructions for Ubuntu and got no errors when Boyle compiled the Matrex package in it's environment.

Any help greatly appreciated.

inspect can be used in multiple occasions, not only the terminal, so it is best to not rely on :io.columns and instead use the width option given to opts. So instead of:

    def inspect(%Matrex{} = matrex, opts) do
      {:ok, columns} = :io.columns()
      Matrex.Inspect.do_inspect(matrex, columns, 21)
    end

One should do:

    def inspect(%Matrex{} = matrex, opts) do
      columns = opts.width
      Matrex.Inspect.do_inspect(matrex, columns, 21)
    end

Notice that width may be infinity. In this case you may want to handle that by setting a default value, such as 80 or whatever.

PS: note there is a Matrix.Inspect.inspect function but that is exactly the same as calling IO.inspect(matrex). :)

The proposed solution to the error: native/src/matrix_dot.c:5:10: fatal error: 'cblas.h' file not found

is

open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

however, that directory does not exist on macOS 10.15. The solution here did work for me.

export CPATH=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/

Honestly though this is just the first working result on google, not sure if it makes sense to add to the README or not.

I suggest deleting the boobs function. Other then a vessel for joke delivery
but this is nonsense.

@doc """
Function of a surface with two hills.
"""
@spec boobs(float, float) :: float
def boobs(x, y) do
  x = (x - 40) / 4
  y = (y - 40) / 4

  :math.exp(-:math.pow(:math.pow(x - 4, 2) + :math.pow(y - 4, 2), 2) / 1000) +
    :math.exp(-:math.pow(:math.pow(x + 4, 2) + :math.pow(y + 4, 2), 2) / 1000) +
    0.1 * :math.exp(-:math.pow(:math.pow(x + 4, 2) + :math.pow(y + 4, 2), 2)) +
    0.1 * :math.exp(-:math.pow(:math.pow(x - 4, 2) + :math.pow(y - 4, 2), 2))
end

I like jokes probably more than the next person unless they offend or cause
potential harm. Also I am just as guilty for this kind of thing, because we all
are idiots. MISTAKES MADE. LESSONS LEARN. WE GROW, IF GIVEN ROOM.

Formally and moreover:

... this form of code and practice may cause potential harm to some all
participants in OSS. In a larger overarching context, condones and perpetuates
this insidious behavior in a larger community. Dudes, basically ...

Not my intention to be self-righteous judge and high-horse executioner. But there
going to be a large audience that'll be reading this (long and overly complex story).
They may not be so lenient in thought and/or action. Bottom-line, better to save
your ass. So to speak.

iex(24)> Matrex.new([[4_000_022.6578]])
#Matrex[1×1]
┌           ┐
│4000022.75 │
└           ┘

Passing a fairly large number causes the number to be rounded off - this gives me wrong values when I perform operations on these matrices.

Is this a known limitation? Are there any work-arounds for this to work with larger numbers?

Thanks in advance.

/cc @goodhamgupta

Brilliant library. Of course Numpy has a 2 decades of accumulated functionality on top, so there's still a lot of stuff python-side that I'd want to use.

How can Matrix talk to Numpy efficiently, via, say erlports? Erlports sends python back as follows:

Erlang/OTP 21 [erts-10.0.8] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]
Interactive Elixir (1.7.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> {:ok, pid} = :python.start()                   
{:ok, #PID<0.176.0>}
iex(2)> xx = :python.call(pid, :py, :eig, [10])        
{{:"$erlport.opaque", :python,
  <<128, 2, 99, 110, 117, 109, 112, 121, 46, 99, 111, 114, 101, 46, 109, 117,
    108, 116, 105, 97, 114, 114, 97, 121, 10, 95, 114, 101, 99, 111, 110, 115,
    116, 114, 117, 99, 116, 10, 113, 1, 99, 110, 117, 109, 112, 121, ...>>},
 {:"$erlport.opaque", :python,
  <<128, 2, 99, 110, 117, 109, 112, 121, 46, 99, 111, 114, 101, 46, 109, 117,
    108, 116, 105, 97, 114, 114, 97, 121, 10, 95, 114, 101, 99, 111, 110, 115,
    116, 114, 117, 99, 116, 10, 113, 1, 99, 110, 117, 109, 112, ...>>}}
iex(3)> xx = :python.call(pid, :py, :eig_msgpack, [10])
<<146, 133, 164, 116, 121, 112, 101, 164, 60, 99, 49, 54, 164, 107, 105, 110,
  100, 160, 162, 110, 100, 195, 165, 115, 104, 97, 112, 101, 145, 10, 164, 100,
  97, 116, 97, 218, 0, 160, 61, 85, 216, 39, 118, 191, 18, 64, 0, 0, 0, 0, ...>>
iex(4)> Msgpax.unpack!(xx)
[
  %{
    "data" => <<61, 85, 216, 39, 118, 191, 18, 64, 0, 0, 0, 0, 0, 0, 0, 0, 152,
      188, 99, 203, 39, 212, 235, 191, 0, 0, 0, 0, 0, 0, 0, 0, 176, 49, 23, 6,
      60, 177, 190, 63, 82, 69, 170, 45, 5, 196, 229, 63, ...>>,
    "kind" => "",
    "nd" => true,
    "shape" => '\n',
    "type" => "<c16"
  },
  %{
    "data" => <<24, 3, 175, 142, 209, 167, 205, 191, 0, 0, 0, 0, 0, 0, 0, 0, 78,
      49, 82, 243, 187, 186, 202, 191, 0, 0, 0, 0, 0, 0, 0, 0, 24, 28, 80, 170,
      100, 115, 198, 191, 68, 9, 19, 108, 248, 3, 178, ...>>,
    "kind" => "",
    "nd" => true,
    "shape" => '\n\n',
    "type" => "<c16"
  }
]

As you can see, calling eig, which doesn't use msgpack, just returns a binary blob for Numpy. However if we msgpack the numpy arrays first, then we get a more structured return, which might be useful. The "data" field could go into a matrex matrix? How would one go about doing that?

Here by the way is the Python code:

from __future__ import print_function
import numpy as np
import pdb
import IPython
import string
import msgpack
import msgpack_numpy as m
m.patch() # patch msgpack to do numpy

def eig(n):
    np.random.seed(8472)
    xx = np.random.rand(n * n).reshape(n, n)
    yy = np.linalg.eig(xx)
    return yy

def eig_msgpack(n):
    return msgpack.packb(eig(n))

def dicadd(dict):
    return {string.join(dict.keys()): sum(dict.values())}

And here are my mix.exs deps:

  defp deps do
    [
      {:erlport, "~> 0.10.0"},
      {:benchwarmer, "~> 0.0.2"},
      {:msgpax, "~> 2.0"}
    ]

Please tag the releases in github with "v#{@version}" before/after mix hex.publish, using:

git tag v0.6.7
git push --tags

hexdoc automatically creates links on the right side of modules/functions headers ("</>") referencing the current version and without tags these links are broken.

Right now the library uses "regular" NIFs. In general, for playing nice with the soft-realtime guarantees of the VM regular NIFs should execute in under 1ms. When they execute longer, it might throw off the load balancing of the schedulers (OS threads actually executing Erlang code) and lead to what is known as "scheduler collapse".

Fortunately, there are tools to prevent that.
If the job can be easily chunked the best solution is to periodically call enif_consume_timeslice to check if code should yield and call enif_schedule_nif to yield and allow the VM to execute other tasks, if necessary.
If chunking is not an option, the NIFs can be marked to execute on "dirty" schedulers - separate OS threads where execution can take however long it can and does not affect how regular schedulers work. The downside is that it incurs a context switch to a different OS thread - if the computation is complex enough, that overhead should not be noticeable.

After briefly looking at the code, it seems to be it would be good for the library to mark most functions as dirty (especially the element-wise operations or dot-product). A possible approach would also be to provide two implementations - dirty and not and call one of them depending on the size of the matrix. This might bring in more complexity then desired, though. An example of a library with a more intricate scheduler handling is enacl, which depending on the complexity runs functions on regular or dirty schedulers.

Awesome work!

Would you mind benching against my library?

https://www.hex.pm/packages/graphmath

We have somewhat different use cases, but I'd be curious to see how pure elixir stands up for the things tested. Thank you!

@michalmuskala can you have a look at new structure of the Matrex.Array object in this branch: https://github.com/versilov/matrex/blob/array/lib/matrex/array.ex

I've added types, arbitrary dimensions and strides. These are now stored in Elixir structure, and data field contents only pure data, without dimensions. Exactly like you suggested in the post on ElixirForum.

I plan to merge this new structure into the Matrex structure.

Thanks in advance!

Hi,
Thank you so much for this wonderful library!! Would it be possible to get the diagonal of a square matrix and apply exponential functions to it with the current implementation?

Hi @versilov,

I'm trying to round off numbers in a matrix. It works when my matrix contains only numbers.

matrix = Matrex.new([[1.9893432, 2.989899]])
Matrex.apply(matrix, &Float.round(&1, 2))
#Matrex[1×2]
┌                 ┐
│    1.99    2.99 │
└                 ┘

However, the same function breaks when the matrix contains NaN's/Infinity.

Matrex.new([[1.9893432, 2.989899]])
|> Matrex.divide(0)
|> Matrex.apply(&Float.round(&1, 2))

** (FunctionClauseError) no function clause matching in Matrex.apply_on_matrix/3

    The following arguments were given to Matrex.apply_on_matrix/3:

        # 1
        <<0, 0, 128, 127, 0, 0, 128, 127>>

        # 2
        #Function<6.128620087/1 in :erl_eval.expr/5>

        # 3
        <<1, 0, 0, 0, 2, 0, 0, 0>>

    Attempted function clauses (showing 2 out of 2):

        defp apply_on_matrix(<<>>, _, accumulator)
        defp apply_on_matrix(<<value::float()-little()-size(32), rest::binary()>>, function, accumulator)

    (matrex) lib/matrex.ex:693: Matrex.apply_on_matrix/3
    (matrex) lib/matrex.ex:676: Matrex.apply/2

I've also tried using the Matrex.apply function on each element as follows:

Matrex.new([[1.9893432, 2.989899]])
|> Matrex.divide(0)
|> Matrex.apply(fn val, _row, _col -> Float.round(val, 2) end)

** (FunctionClauseError) no function clause matching in Matrex.apply_on_matrix/6

    The following arguments were given to Matrex.apply_on_matrix/6:

        # 1
        <<0, 0, 128, 127, 0, 0, 128, 127>>

        # 2
        #Function<18.128620087/3 in :erl_eval.expr/5>

        # 3
        1

        # 4
        1

        # 5
        2

        # 6
        <<1, 0, 0, 0, 2, 0, 0, 0>>

    Attempted function clauses (showing 2 out of 2):

        defp apply_on_matrix(<<>>, _, _, _, _, accumulator)
        defp apply_on_matrix(<<value::float()-little()-size(32), rest::binary()>>, function, row_index, column_index, columns, accumulator)

    (matrex) lib/matrex.ex:721: Matrex.apply_on_matrix/6
    (matrex) lib/matrex.ex:690: Matrex.apply/2

I've also tried the array branch for the same:

Matrex.new([[1.9893432, 2.989899]])
|> Matrex.divide(0)
|> Matrex.apply(fn val, index -> Float.round(val, 2) end)

** (FunctionClauseError) no function clause matching in Matrex.apply_on_matrix_float32/5

    The following arguments were given to Matrex.apply_on_matrix_float32/5:

        # 1
        <<0, 0, 128, 127, 0, 0, 128, 127>>

        # 2
        #Function<12.128620087/2 in :erl_eval.expr/5>

        # 3
        {1, 1}

        # 4
        {1, 2}

        # 5
        ""

    Attempted function clauses (showing 2 out of 2):

        def apply_on_matrix_float32(<<>>, _, _, _, accumulator)
        def apply_on_matrix_float32(<<value::float()-little()-size(32), rest::binary()>>, function, pos, shape, accumulator)

    (matrex) lib/matrex.ex:1833: Matrex.apply_on_matrix_float32/5
    (matrex) lib/matrex.ex:759: Matrex.apply/2

Using a custom round function and checking for the type of each element also doesn't seem to work

defmodule Custom do
  def round(value, precision) when is_float(value), do: Float.round(value, precision)

  def round(value, _precision) do
    value
  end
end

Matrex.new([[1.9893432, 2.989899]])
|> Matrex.divide(0)
|> Matrex.apply(fn val, index -> Custom.round(val, 2) end)

** (FunctionClauseError) no function clause matching in Matrex.apply_on_matrix_float32/5

    The following arguments were given to Matrex.apply_on_matrix_float32/5:

        # 1
        <<0, 0, 128, 127, 0, 0, 128, 127>>

        # 2
        #Function<12.128620087/2 in :erl_eval.expr/5>

        # 3
        {1, 1}

        # 4
        {1, 2}

        # 5
        ""

    Attempted function clauses (showing 2 out of 2):

        def apply_on_matrix_float32(<<>>, _, _, _, accumulator)
        def apply_on_matrix_float32(<<value::float()-little()-size(32), rest::binary()>>, function, pos, shape, accumulator)

    (matrex) lib/matrex.ex:1833: Matrex.apply_on_matrix_float32/5
    (matrex) lib/matrex.ex:759: Matrex.apply/2

Please let me know if I've missed out on any details! Thanks!

The atoms NaN is represented internally as Elixir.NaN that's why it is most commonly reserved for module names. The recommendation would be use explicit atoms such as: :nan | :inf | :neg_inf or even :NaN | :Inf | :NegInf.

I am using Matrex in the context of AoC for the second year now, and it is not the first time I am confronted with the need to get the neighbours of a given position in the matrix.

That is, given a valid position {x,y}, calculating the list of [{x-1, y-1}, {x-1, y}, {x-1, y+1}, {x, y-1}, {x, y+1}, {x+1, y-1}, {x+1, y}, {x+1, y+1}] (or a subset of this, including only valid points inside the bounds of the matrix, of course).

Would a PR implementing Matrex.neighbours/3 as

@spec neighbours(Matrix.t(), non_neg_integer(), non_neg_integer())

something that this project would be interested in? If so, I offer to implement such PR.

An alternative would be Matrex.neighbours/4,

@spec neighbours(Matrix.t(), non_neg_integer(), non_neg_integer(), boolean())

with an optional argument (default true) to consider the diagonals (or not).

Foremost, thanks for this amazing project!

I noticed that on the benchmark section on the readme there are only comparisons between Elixir/Erlang libraries, which are not that fast/used.

Would be possible to benchmark against Numpy ? So we get a perspective against the most used library on this domain and how well we can expect matrex to perform?