elixir-crawly / crawly_ui Goto Github PK

We should have a second table on the homepage with 5 most recent completed jobs there.

It should be possible to export items to CSV and to have a file as the result.

Please be aware that due to the size of the data it might be required to stream the resulting file.

Currently, our search is pretty basic and slow. Also not user friendly at all :(. I wonder if we could improve the case with the help of live view.

I see it in the following way:

• Search needs to support fields autocompletion [e.g. if I start to type ti: it should suggest title]
• Search should support operators like: "AND", "OR" and "NOT" (Ideally also >, < and others)
• Search should not be very slow :). Maybe we need some sort of benchmark to test it

It should be possible to export items to JSON and to have a file as the result.

Please be aware that due to the size of the data it might be required to stream the resulting file.

We should add a menu bar, since now we start to have many pages to navigate to. Ideally something pretty. Not a button anymore

i.e

Home | Jobs | Spiders | Nodes | Schedule Spider
And home is the current index, jobs/spiders/nodes lists all of the said field, and schedule spider as it is now

We need to have a page with a list of all spiders discovered by the crawlyui

Currently, the main template has two closings tags:

If the job is stopped we should display an average speed instead of 0.

Some example is here:

http://crawlyui.com/logs/267/list

The count estimate for this query shows 320. The actual amount of rows in the table is 4. As a result, we have long pagination without items.

Spider logs must be fetched from crawly.

Hi,

I try to compile locally the latest master :

commit 3b95de73299c4f62b84fa6f41418e2ffd0885dca (HEAD -> master, origin/master, origin/HEAD)
Author: Oleg Tarasenko <[email protected]>
Date:   Mon Dec 21 12:02:09 2020 +0100

    Speedup logs

$ iex -S mix phx.server`

`== Compilation error in file lib/crawly_ui_web/views/job_view.ex ==
** (SyntaxError) lib/crawly_ui_web/templates/job/spider.html.leex:26: unexpected token: end

My local env :

Erlang/OTP 22 [erts-10.7.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Elixir 1.9.4 (compiled with Erlang/OTP 20)

Spiders show send a stop message to UI, in order to avoid displaying 'node_down' which does not reflect the reality

Currently, spider names on UI are way too long. For example on the main page, we have Elixir.Spiders.Walmart as the spider. Lets do the following: Elixir.Spiders.Walmart -> Walmart