omohokcoj / filterable Goto Github PK

Hi,

I added the following code for filtering on a boolean column archived

defmodule MyApp.Registrations.RegistrationFilters do
  use Filterable.DSL
  use Filterable.Ecto.Helpers
  import Ecto.Query, warn: false

  field(:archived)
end

when I try the following, it works:

{:ok, query, filters} = RegistrationFilters.apply_filters(query, %{archived: true})

with result being:

{:ok,
 #Ecto.Query<from r0 in MyApp.Registrations.Registration, as: :registration,
  where: r0.archived == ^true, order_by: [desc: r0.inserted_at]>,
 %{archived: true}}

but if I run the same with filtering on false, the clause does not get added in query. It does show up in filters though:

{:ok, query, filters} = RegistrationFilters.apply_filters(query, %{archived: false})

returns:

{:ok,
 #Ecto.Query<from r0 in MyApp.Registrations.Registration, as: :registration,
  order_by: [desc: r0.inserted_at]>, %{archived: false}}

Are we supposed to use some different format for the boolean values?

Best regards,
Gorav

Should be changed to something useful ;)

Thanks,
Philipp

This error happens when I try to cast a value that doesn't match any of the atoms in a list.

filterable do
   field :status, cast: {:atom, [:accepted, :pending, :rejected]}
end

[info] GET /v1/trees
[debug] Processing with PetreeApiWeb.TreeController.index/2
  Parameters: %{"status" => "asdf"}
  Pipelines: [:api]
[info] Sent 500 in 1617ms
[error] #PID<0.466.0> running PetreeApiWeb.Endpoint (connection #PID<0.465.0>, stream id 1) terminated
Server: 127.0.0.1:4000 (http)
Request: GET /v1/trees?status=asdf
** (exit) an exception was raised:
	** (FunctionClauseError) no function clause matching in Filterable.Params.cast_error_message/1
		(filterable 0.7.3) lib/filterable/params.ex:229: Filterable.Params.cast_error_message([value: "asdf", cast: {:atom, [:accepted, :pending, :rejected]}])
		(filterable 0.7.3) lib/filterable/params.ex:191: Filterable.Params.cast/3
		(filterable 0.7.3) lib/filterable/params.ex:18: Filterable.Params.filter_value/2
		(filterable 0.7.3) lib/filterable.ex:107: anonymous fn/4 in Filterable.filter_values/3
		(filterable 0.7.3) lib/filterable/utils.ex:8: anonymous fn/3 in Filterable.Utils.reduce_with/3
		(elixir 1.10.4) lib/enum.ex:3686: Enumerable.List.reduce/3
		(elixir 1.10.4) lib/enum.ex:2161: Enum.reduce_while/3
		(filterable 0.7.3) lib/filterable.ex:93: Filterable.apply_filters/4
		(petree_api 0.1.0) lib/petree_api_web/controllers/tree_controller.ex:16: PetreeApiWeb.TreeController.index/2
		(petree_api 0.1.0) lib/petree_api_web/controllers/tree_controller.ex:1: PetreeApiWeb.TreeController.action/2
		(petree_api 0.1.0) lib/petree_api_web/controllers/tree_controller.ex:1: PetreeApiWeb.TreeController.phoenix_controller_pipeline/2
		(phoenix 1.5.6) lib/phoenix/router.ex:352: Phoenix.Router.__call__/2
		(petree_api 0.1.0) lib/petree_api_web/endpoint.ex:1: PetreeApiWeb.Endpoint.plug_builder_call/2
		(petree_api 0.1.0) lib/plug/debugger.ex:132: PetreeApiWeb.Endpoint."call (overridable 3)"/2
		(petree_api 0.1.0) lib/petree_api_web/endpoint.ex:1: PetreeApiWeb.Endpoint.call/2
		(phoenix 1.5.6) lib/phoenix/endpoint/cowboy2_handler.ex:65: Phoenix.Endpoint.Cowboy2Handler.init/4
		(cowboy 2.8.0) /app/deps/cowboy/src/cowboy_handler.erl:37: :cowboy_handler.execute/2
		(cowboy 2.8.0) /app/deps/cowboy/src/cowboy_stream_h.erl:300: :cowboy_stream_h.execute/3
		(cowboy 2.8.0) /app/deps/cowboy/src/cowboy_stream_h.erl:291: :cowboy_stream_h.request_process/3
		(stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3

Hey @omohokcoj - really enjoying my time with filterable, has been much smoother than trying to work with another DSL on top of SQL.

Is it possible to use filterable on top of a query rather than just a schema? My use case is that I'd like to get all my join tables and computed columns up front and then filter on those, rather than duplicating the joins in the filterable macro.

i.e. right now, the order of operations is
Fic.apply_filters(conn) |> Fic.get_all_join_tables
and I'd like to do
Fic.get_all_join_tables |> Fic.apply_filters(conn)

My get_all_join_tables query looks like this:

def get_all_join_tables(query) do
   from f in query,
        left_join: reviews in assoc(f, :reviews),
        group_by: f.id,
        preload: [
          :submitter,
          :author,
          :genres,
          reviews: :submitter,
        ],
        select: %{
         f |
          review_count: fragment("count(?) as review_count", reviews.id),
          review_avg: fragment("coalesce(?::float, 0) as review_avg", avg(reviews.rating))
        }
 end

And one of my filterable blocks look like this...

@options top_param: :search, cast: :integer
    filter review_avg(query, value, _conn) do
      from f in query,
           left_join: reviews in assoc(f, :reviews),
           having: avg(reviews.rating) > ^value
    end

Ideally, I wouldn't have to left_join reviews again and could build off the work I did on the get_all_join_tables.

Any suggestions? Thx again for open sourcing this.

Hello,

When using top_param I noticed that the filter_values does not retain the same level of nesting. Example

@options top_param: :q, cast: &String.downcase/1
  filter name(list, name), do: ...

When running this filter I get back

{:ok, list, %{name: "search". ... }%}

This feels a bit inconsistent, I rather have it return back the original nesting of top_param

{:ok, list, %{q: %{name: "search"}, ...}%}

This way it's much easier writing an UI and when you want to indicate which filter fields are used (especially when using multiple search fields)

Please fix your readme. I tried your Phoenix controller and Separate module both of them don't compile.

Phoenix: 1.5.4
Elixir: 1.10.4

can I use filterable to convert the ? = pattern to SEO friendly / ?

Hello,

When implementing a frontend to filterable that allows to search / order / paginate on Lists and ecto records, I noticed that there is no way to find out the total amount of records when a search + ordering is performed.

This is needed when rendering pagination helpers, you don't want to show a next link when you know there won't be any records to show on the next page.

It would be helpful if this gets returned in the response in apply_filters.

Example

Note that I'm using a regular list instead of ecto models

defmodule Filter do
  use Filterable.DSL

  @options param: :q, cast: &String.downcase/1
  filter name(list, value) do
    list |> Enum.filter(&(String.contains?(String.downcase(&1["name"]), value)))
  end

  # Basic copy past of what paginateable generates + adaption for Lists
  @options param: [:page, :per_page], default: [page: 1, per_page: 10], cast: :integer, share: false
  filter paginate(list, %{page: page, per_page: per_page}) do
    list |> Enum.slice((page - 1) * per_page, per_page)
  end

  @options param: [:sort, :order], default: [order: :desc], cast: :atom, share: false
  filter sort(list, %{sort: field, order: :asc}) do
    list |> Enum.sort_by(&(&1[Atom.to_string(field)]), &<=/2)
  end

  filter sort(list, %{sort: field, order: :desc}) do
    list |> Enum.sort_by(&(&1[Atom.to_string(field)]), &>=/2)
  end
end

Propose

Add a new @option flag that indicates that this filter is the pagination filter and should be run at the end. This way you can store the total amount of items currently returned by other filters. Then this value can be returned when apply_filters is run.

{:ok, result, filter_values, total} = Filter.apply_filters(items, params)

As documented in Erlang, an atom is never garbage collected. One should therefore take extreme precaution when converting a binary to an atom from uncontrolled input.

Using the following filter,

@options cast: :atom
filter sort_by(query, value, conn), do: #...

When called with a random value, each random value will be converted to an atom, regardless of whether or not it is indeed a valid value for this parameter.

A default BEAM configuration will have a limit of 1,048,576 atoms. Since BEAM applications tend to have long up times, such a limit could be reached quite easily simply from API consumers mistakes.

Suggested Solution

Since sometimes, not checking for the validity of the value when casting to an atom is sometimes desired, allow it through an option like cast: :atom_unchecked.

To allow one to validate the value before it is cast, cast: {:atom, [:value1, :value2, value3]} could be used.

To prevent breaking changes to users of this library, we could keep the present cast: :atom option, but display a deprecation warning until the next major release. After displaying the warning, it can simply forward it's arguments to cast: :atom_unchecked.

Hi, I am trying to use filterable in a phoenix project, I have installed version 0.6.0. When i compile the project I got the next Error:

== Compilation error in file lib/project/movement/movement_filters.ex ==
** (CompileError) lib/project/movement/movement_filters.ex:5: undefined function filterable/1
    (stdlib) erl_eval.erl:677: :erl_eval.do_apply/6
    (elixir) lib/kernel/parallel_compiler.ex:198: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/6

The code of movement_filters is a copy of the code in the documentation

defmodule PostFilters do
  use Filterable.DSL
  use Filterable.Ecto.Helpers

  filterable do
    filter title(query, value, _conn) do
      query |> where(title: ^value)
    end

    @options cast: :integer
    filter stars(query, value, _conn) do
      query |> where(stars: ^value)
    end
  end
end

Thanks.

https://github.com/omohokcoj/filterable/blob/master/lib/filterable/params.ex#L11 - this could cause BEAM atoms limit overflow.

kudos to @brodeuralexis for reporting this issue.

After considerable trial and error, these are the correct instructions for this to work:

// patient_filters.ex

defmodule PatientFilters do
  use Filterable.DSL
  use Filterable.Ecto.Helpers
  use Filterable
  import Ecto.Query

  field(:name)
  field(:gender)

  paginateable(per_page: 10)

  filterable do
    @options param: :name
    filter name(query, value) do
      query |> where([u], ilike(u.name, ^"%#{value}%"))
    end

    @options param: :gender
    filter gender(query, value) do
      query |> where(gender: ^value)
    end
  end
end

And in controller:

  def index(conn, _params) do
    with {:ok, query, filter_values} <- PatientFilters.apply_filters(Patient, _params),
         patients <- Repo.all(query),
         do: render(conn, "index.json", patients: patients, meta: filter_values)
  end

Hello,

I'm having a difficult time grasping when a filter is triggered when giving some params.

Given following search params

search = %{q: %{name: "playpass", barcode: "1234"}, sort: "name", order: "desc", page: 1, per_page: 6}
list = [%{"barcode" => "915580318","name" => "PlayPass Scanout"}, ...]

With the following filters

@options param: [q: :name], cast: &String.downcase/1
filter name(list, %{name: name}) do
  IO.inspect "SEARCHING NAME"
  list  
end

  @options param: [q: :barcode], cast: &String.downcase/1
  filter barcode(list, %{barcode: barcode}) do
    IO.inspect "SEARCHING barcode"
    list
  end

When running the filters, nothing is printed in the console, but I expect both to be run. So some questions that pop up that I couldn't find in the readme:

1. Is the filter name of any importance? Does it need to match the param name? Or can it be anything? Does it need to be unique?
2. How do @options relate to the filter name and the params? I do notice that the original nesting is not kept, meaning everything is flattend
3. How to deal with nested search like in the above example? It's quite common to have a form for searching so everything is then namespaced

Thanks! I really like this project since it does not assume anything about your framework or how you want to filter data

We are currently using Filterable in our app successfully with controllers and separate filter modules. We're at a point now where this would also be really useful when fetching data through Phoenix channels for async requests.

Is this possible now or are there plans to add it in the future?