Sidebar list, data storage, and filesystem integration about webscrapbook HOT 43 CLOSED

@danny0838
You could use native messaging with small native helper written in portable language to interact with file system. Several application-launching addons (e.g. External Application Button ) do just that.

from webscrapbook.

The whole point of old Scrapbook was the sidebar, with handy tree browser. Now you told it is impossible. So what is the usefulnes of new addon now?

from webscrapbook.

Maybe I should have not said "impossible". But the fact is that WebExtension has very very much restriction than the legacy XUL/XPCOM framework. And we have to face the fact or maybe develop a workaround.

from webscrapbook.

I second this:
Local server approach: Sounds good, but pay attention to common vulnerabilities, like ../ on path.

But yeah, careful with security breach if this is this way we will now editing captures.
Therefore, for cloud approach... I don't agree, my datas ARE not always suffisely commonly to left on a cloud.
I mean, I use ScrapBook X at work, so, my data are better safe on my external backups than a cloud.
Even for private cloud.

Please, give up on Cloud, or make Cloud AND Local server the wo possibility to managing Captures !

Thanks for your reading @danny0838 !

from webscrapbook.

In terms of GUI, sidebar is doable in Chrome, either as a frame (e.g. vTabs), or as an undecorated window (e.g. TabOutliner) that is handled by the window manager or its substitutes like AHK. Personally, I found vTabs rather slow, but it could be caused by other peculiarities of its implementation.
Or, alternatively, one could implement much of its functionality in a popup (e.g. uMatrix, Conex).
And then there is the native client way, which is similar to the detached undecorated window approach, but more powerful in terms of searching, indexing and tagging the snippets db.

from webscrapbook.

This may be impossible due to the restriction of the current WebExtension framework:

1. Sidebar is only supported in Firefox, not Chrome.

2. Writing to the local filesystem is not supported, so management (probably including modifying content or metadata) may be impossible.

3. Firefox does not allow reading from file:// scheme, so we would have difficulty reading stored data.

from webscrapbook.

As a summary, there are currently following approaches for managing web page captures:

1. Archive file approach: Save each web page as a nice archive file, and let the file manager or other application to manage them.

2. Indexer approach: Provide an indexer that scans a user provided directory with captured web pages and generates a static site that can list or search them.

3. Browser database approach: Capture web pages directly into the database system in the browser so that the browser can access all saved files, and therefore the user can view, list, search, or even edit them. We probably also need a functionality to import data to and to export data from the database.

4. Native messaging approach: Write a small app that allows the browser to access the filesystem via native messaging, and thus allows any manipulation of the captured web pages.

5. Local server approach: Write a small app that hosts a local server with web API, through which the browser could access the filesystem data and thus the captured web pages can be manipulated.

6. Cloud service approach: Implement the functionality to allow the browser to save files to and load files from a cloud storage service, such as Dropbox, Google Drive, Box, etc., and thus the captured web pages can be manipulated.

Currently we already have the 1, and the 2 is under development. As for 3~6, we may only implement one of them as they share almost the same purpose and effect. We need further investigation to decide which to implement.

from webscrapbook.

Browser database approach: may become a browser burden for large amounts of data, use IndexedDB needed.
Native messaging approach: no experience.
Local server approach: Sounds good, but pay attention to common vulnerabilities, like ../ on path.
Cloud service approach: Sounds like this should as an enhancement feature.

from webscrapbook.

Nice discussion going on here, I temporarily don't use Scrapbook as (reorganization, and I needed to clear my bookmarks), but here are a few thoughts:

A combination of Archive file approach, Indexer approach and Local server approach looks like the best solution: Items are accessible through the filesystem (portable), can be searched/sorted through a statically prebuilt index, and more advanced features can be implemented in a local or public server/web interface component, consuming data from the filesystem (archives + metadata)

Browser database approach doesn't seem very portable, risk of tying the storage to an interface that will be deprecated in the future (same thing for Native Messaging), etc. Thing like IndexedDb or LocalStorage are also temporary in nature, and can be easily lost (browser migration, clearing caches, etc..).

The filesystem storage method is already compatible with cloud file synchronization systems (dropbox/gdrive/nextcloud...) so there would be no work required for that feature.

from webscrapbook.

With similar idea, Archive file approach + Indexer approach + Local server approach is currently more preferable, majorly due to several issues of other approaches:

• Browser database approach:

  1. The implementation for browser databse API is different across browsers and unstable. For example, indexedDB does not work in a private window in Firefox and is separated between normal and incognito window in Chrome, and blobs cannot be stored in chrome.storage in Chrome. There are tons of issues to simply support the I/O for all of our targeted browsers.
  2. For a cross-browser user, it's probably too annoying to export, import, export, import, ..., forever, just to access the same data in different browsers.
  3. There is still much limitation for the archive page viewer to losslessly view a web page, but we have to depend on it if all captures are a archives stored in the browser database.

• Native messaging approach:

  1. The installation is quite complicated. And the users may have problem just to install it.
  2. The implementation is different across browsers, and we need to write different codes.
  3. Firefox addon cannot handle the file:// protocol. We'd have difficulty to get data shown in the browser even if we can I/O them via native messaging.

In comparison, browsers are natively the expert sending and receiving network request, and it would be a piece of cake to "control" the local file system via network request and a cooperative local server. Besides, the local server can be turned into a cloud server without too much change. Additionally, it would be a nice additional feature that complements the Archive file approach and Indexer approach, as @nodiscc says.

We probably are heading for this strategy. We'd have to seek for a good framework for such local server. Is there a living project doing such a "manipulate local file system with a local server" thing? (node.js and python based are more preferred)

from webscrapbook.

Ah yeah @danny0838 if the Local Server act like a Private Cloud server, it will be ok for me too !
(What I mean by saying "NO CLOUD please" was only Public/Distant (third party Cloud/not-own) services !)

If it's like a Media Server (DLNA like but Cloud protocol) it is OK.
Much wanted feature, I restarted my Firefox today, 57 installed.. And.. I cryied.
I mean.. my beloved extensions all destroyed :/

ScrapBook X miss me, very miss me, but, I'm really proud of you to not give up your users!

from webscrapbook.

Very nice discussion! I am connect too. I had read all proposition and I think this:

1. Screening is one addition to capture
   1.1 It is necessary to add tags to search for saved pages
   1.2 Need to be able to add a save folder
   1.3 It is better to keep the page in the archives as a single file (if it's just html it's very crappy with backups, as many small files)
   1.4 It is necessary to delete superfluous or to save only that that is necessary.
2. Viewing and searching for accumulated content is the second addition or script for converting to a frame
   2.1 Need a search
   2.2 Structure of folders
   2.3 Reference to the source of the page
   2.4 Page Editor

I prefer local storage. I store files on a flash drive and use it both at home and at work. However, sometimes I think about Googledrive, but there are many small files and it is not possible now to save new notes there.
Recently, he climbed to the vivaldi browser and used scrapbook only a tree frame. I am comfortable.
Thanks @ danny0838 for reading and for excellent work!
P.S. Sorry for my English :(

from webscrapbook.

@danny0838

I'd like to defend native messaging approach, since I offered it.

The installation is quite complicated. And the users may have problem just to install it.

It's certainly less complicated than server installation. At least you don't have to configure firewall.

The implementation is different across browsers, and we need to write different codes.

You mean that manifest locations are different? Because API itself seems the same to me.

get data shown in the browser even if we can I/O them via native messaging.

Yeah, I had something like that in mind.

local server

Do you intend to write your own HTTP server or just run your Python/Node modules on top of existing one? Which one? Would it be cross-platform? What about difficulty of installation?

from webscrapbook.

@kirill9000

1. As far as I could remember, the firewall configuration can be passed for localhost server access.

2. Besides, it seems that the passed arguments are slightly different across browsers, and version. Maybe I should have added that we have to write different code to install the native manifest on different OS if we want to automate it.

3. The file: protocol issue is actually the most challenging. If we cannot find a good solution, native messaging approach is basically not viable.

4. We don't set a clear restriction here. It could either be developed on an existing framework or purely made on our own. We would make it as cross-platform as possible. As for the difficulty of installation... I don't think it could be any more difficult than installation of the app for native messaging.

from webscrapbook.

Sidebar list fuction is crucial and constituted the main functionality of the extension. So that it should be preserved. Have you thought about using a cloud storage directly and bypassing the local storage entirely? Basically saving a page as single html in a folder of the cloud provider and listing that folder contents in the sidebar of Firefox. Individual single html files can be still reached and downloaded or desktop client of the cloud provider would sync it automatically to a folder on the client computer so that no functionality is missing from a user point of view. I don't know how that would effect editing capabilities or the responsiveness of the extension though.

from webscrapbook.

@danny0838

passed arguments are slightly different across browsers

Yeah, I have seen it, but don't know why would you even need to use them.

The file: protocol issue is actually the most challenging. If we cannot find a good solution, native messaging approach is basically not viable.

I think, it can be avoided altogether. Just encode path as parameter to you moz-extension page (like you do now in viewer) and pass to native app which will read it and return it's content.

It could either be developed on an existing framework or purely made on our own.

Server approach may offer more functionality, but I just doubt that it'll be easier to develop and use. The decision is obviously yours.

from webscrapbook.

In 0.26.0 the site indexer can output a server script, which can be run to host a server as long as Python 3.* is installed on the system. Configure of the server can be set using the config.ini. The option can be checked in the options page.

from webscrapbook.

As a summary, there are currently following approaches for managing web page captures:

Those are indeed all good choices. Personally, I hope for a modular solution, that would give the user the freedom to have it the way he needs, be it creating his own indexing and management solution or using one of the available ones, e.g. zipgrepping a FUSE mounted folder or whatever.

from webscrapbook.

Re local server and file system integration -- my use case:

I have a local Lighttpd instance running that I use to serve up local content. I've added a document location mapping for the WebScrapBook directory and can get to the map.html and frame.html pages through a local HTTP URL. Getting to the saved content requires changing permissions for the content folders that WebScrapBook creates. This is a manual step after indexing. Would adding pre-defined custom commands at the end of indexing be viable?

from webscrapbook.

@danny0838 Do you have more details about the side indexer's server script?

A simple Python web app on localhost can be built with Bottle (single file microframework with built in template engine).

from webscrapbook.

Hi, folks:
A similar project: scrapbookq. It seems to store pages via a local server.

from webscrapbook.

@konstruktiv A short introduction is available here. Is there still something you'd like to know?

from webscrapbook.

It works now. (I was using < 0.26.0)

from webscrapbook.

Concerning local server, cloud and partially native messaging approaches.

Have you thought if WebDAV server can be used as a storage backend? Actually I have almost no experience with WebDAV, just some experiments but from the first glance it looks promising as a kind of common denominator. Of course, it may be not trivial to setup and configure such server. On the other hand there is no perfect way for scrapbook withing the limits of WebExtensions.

Please, correct me if I am wrong, but e.g. ScrapbookQ native messaging application has built-in HTTP-server and custom file-saving protocol. Personally. I would prefer to configure http-server myself instead of getting an instance that even is not limited listened IPs to loopback interface and has no authorization facilities.

WebDAV is an extension of HTTP, so likely a thin wrapper around fetch API should be enough inside browser add-on (mostly PUT and GET methods, may be some dance with locks).

Nginx and apache web servers have dav modules. I have seen implementations of WebDAV in python and go however I have not tried them. Likely that means that native messaging application or local server can be developed with support of WebDAV protocol, so provide access to filesystem for browser extension.

Custom configuration of web-server (I am playing with nginx) allows to separate read-write WebDAV access for add-on and read-only plain HTTP to view saved pages. So JavaScript code, that for some reason may remain in the saved page, has less chance to change saved data, including other pages.

Access to a cloud with WebDAV support in general should be enough for displaying saved pages but I would recommend at least a special account that does not contain sensitive private data.

It may be overkill, but I have even tried to isolate pages saved from different domains. Page has to be requested from domain prefixed by its origin domain (e.g. github.com.data.srcapbook.localdomain). Simple python script is queried by ngx_http_auth_request_module through uWSGI interface. Ubuntu has preinstalled dnsmasq, so adding scrapbook.localdomain DNS zone required a single line in a conf file.

If you fill that there is a better place to discuss this approach (e.g. a dedicated issue), please suggest it.

I consider WebDAV as a good candidate for srcapbook storage backend. The protocol is more or less standard. I hope, implementation will not be hard (comparing to original XUL extension). For advanced users it provides access and privileges control or even a kind of private cloud. Likely existing components will allow to write custom python native messaging application that acts as WebDAV and HTTP server for those who do not like to configure a general-purpose web-server.

from webscrapbook.

Any updates on the issue?

from webscrapbook.

WebScrapBook 0.36.* is a quick-and-dirty version implementing basic sidebar and backend server integration (cooperating with PyWebScrapBook).

There are still many details to improve, including but not limited to:

1. Enhanced sidebar organization: current commands implementation is rather crude to ensure mobile support. We may inplement a real context menu and drag-and-drop manipulation in the future.
2. Sidebars syncing: when there's a change to backend data that is not from the sidebar, the sidebar database does not update, and the user has to refresh the sidebar manually. If the user changes something before refreshing, previous change to the metadata and TOC are overwritten, which is dangerous and needs fixing.
3. Search implementation: currently the sidebar does not support search functionality.
4. Recycle bin: we may implement a "recycle bin" feature so that a mistake deletion can be undone, for better data security.
5. Format and highlight tools for HTML page editing.
6. Update modified timestamp when a page or note is edited.
7. Edit content of an archive file directly.

Somethings we may change in the future:

1. Data scheme

Current data scheme, which looks like:

<scrapbook>/data/<data files>
<scrapbook>/tree/<index files>

may be change to:

<scrapbook>/.wsb/tree/<index files>
<scrapbook>/<data files>

so that everything generated by the backend is uniformly saved in the .wsb directory.

This will make serving a directory more clear and enables the support of a verbatim data structure (i.e. organize data files using native filesystem directories and files).

Unfortunately there would be an issue for downward compatibility. We may need a migration tool for this.

2. Server side indexer

We are planning to rework the indexer at the server side, which is more natural for a client-server application and would allow the indexer to be run from the CLI.

Current extension indexer may be removed in the future as its functionality is rather limited. A side effect is that the indexer will be no longer available without installation of the backend, and mobile support of the site indexer is no longer available. This would be a shock for those who wants data organization without a backend application.

(The current indexer, when run for the backend, is a lot faster on Firefox Quantum than on Chromium. Does anyone know why?)

3.. ScrapBook folder

Currently WebScrapBook can save captured data to (1) the backend, (2) scrapbook folder, and (3) as single file. We are going to integrate (2) and (3) by allowing the user to specify the default name of a capture, such as WebScrapBook/{{ID}}, or WebScrapBook/{{YEAR}}/{{MONTH}}/{{PAGE-TITLE}} under the default download folder. A cost to this, however, is that the user will no longer be able to specify an explicit save path for individual capture. This is also dependent of 1. as the current save path is still WebScrapBook/data/{{ID}}.

4. Removal of single page capture

We may remove the support of capturing a web page directly into (scripted) single HTML, as the single HTML format is rather arbitarary and is not a good idea for long-term data archival.

We would probably implement a conversion tool for the backend toolkit so that the user can convert captured data into single HTML (and other) format like the legacy ScrapBook X File Converter did. But they may be available as a CLI tool only, and there will be extra steps for the user to get a single HTML capture of a certain web page.

Further feedbacks are welcome.

from webscrapbook.

您好，我感觉webscrapbook首要需要恢复的功能就是scrap x的左侧栏文件夹树状结构定制功能，以及右键菜单定制默认或目标存储文件夹功能，这会大大增加便捷性。我看您在某个贴里提到正在致力于恢复左侧栏功能，期待同时也能加入右键菜单的联动功能。

其他功能比如：更便捷的设置和更改撷取网页资料的存储文件夹地址功能；左侧栏更丰富的定制和搜索功能等如果能恢复当然更好。

另外，有两个问题：

第一个是：我的“打开剪贴簿”按钮一直是灰色，且无法点击，即便我撷取了网页后也是如此，不知为何？

第二个问题是：当我同时使用下载界面增强扩展，如下图的“Download Manager”和把webscrapbook的资料保存格式设置为“文件夹”后，在撷取网页的过程中“Download Manager”界面会瞬间出现大量的文件下载通知，并造成浏览器的卡死。而这种现象在scrap x里是没有的，即在scrap x下这两类扩展之间没有冲突，不知道这种情况能否消除。

再次感谢您，

祝一切顺利！

from webscrapbook.

@minshifu

0. 側欄功能從 0.36 版開始就有實做並且持續更新。右鍵選單直接選擇儲存位置牽涉到許多複雜的問題，短期內大概還不會實做。

1. 「打開剪貼簿」功能要安裝並設定好後端伺服器才能使用。可參見相關選項及說明：

2. Chromium 系瀏覽器若勾選了「下載每個檔案前先詢問儲存位置」就會有這個問題，建議不勾選。（Firefox 系瀏覽器的運作方式不同，即使勾選了該項目，擴充套件若設定不詢問則以擴充套件的設定優先，所以不會每個檔案都跳視窗。）

這些問題點小問號都有提示，可善加利用。

from webscrapbook.

您好，感谢指教。我昨天折腾了一顿，但是还是没完全搞明白。比如：

1）按照您网上的相关提示我导入了原先的scrapbook x 文件夹内容，点击frame.html 后可以在侧栏显示树状结构图，但点击每个链接都没有内容可以显示，如下图所示，不知为何。

2）是否每次都要手动开启“backend server” ? 否则每次都显示如下图的提示。

3）比如我想设定所有资料存储在D:\backup\firefox 这个文件夹下面的话。cd /path/to/scrapbook 这条命令不知道具体如何设置，因为我安装Python 启动backend server 后显示如下图的提示

4）如果我现在想利用的文件夹路径是D:\backup\firefox\scrapbook，以下“剪贴簿文件夹”具体该如何设置，恕我没有太看懂小问号的提示内容。

from webscrapbook.

1) 產生索引下載壓縮包以後，要解壓縮放回原來的剪貼簿資料夾。

假設原來的剪貼簿資料夾內容有：

<path>/WebScrapBook/data/...

加上產生的索引會變成：

<path>/WebScrapBook/data/...
<path>/WebScrapBook/tree/...

然後再開啟 WebScrapBook/tree/frame.html 或WebScrapBook/tree/map.html 就可以了。

其實這些說明都有寫到，不曉得為什麼這麼多人不理解（不少人問過同樣問題）：

2) 是的，要使用後端就是每次都要開啟。Windows 下可以把 .wsb/serve.py 的捷徑放到啟動資料夾，就可以在每次登入時自動啟動（或用其他方式設定成系統服務也可以）。

3) Windows 的 CMD 如果目標磁碟機和目前磁碟機不同，要先切換磁碟機，例如：

C:\Users\Test>d:

D:\>cd "D:\backup\firefox"

D:\backup\firefox>wsb config -ba
......

D:\backup\firefox>wsb serve
......

還有一個更快的方法是在資料夾上 Shift+右擊，然後選擇「在此處開啟命令視窗」，就會直接切換到該目錄下：

這裡還有個問題，如果還沒建立索引，還是會出現那樣的訊息，解決方法就是先建立索引（到建立站台索引直接點擊「索引伺服器資料」即可）。

4) 既然要用後端，直接把「儲存擷取資料至」設定成「後端伺服器」就好。

如果不用後端又想下載到該目錄，就需修改預設下載資料夾（比如預設下載資料夾設定成 D:\backup\firefox、剪貼簿資料夾設定成 scrapbook），或者用建立符號連結的方式。

from webscrapbook.

感谢感谢，不过我一直都有把产生索引后下载的压缩包里的内容放回原来的剪贴簿资料夹里（也删除了原始的scrapbook.rdf），然而每次点击 frame.html 或 map.html 后虽然侧栏会显示原始的树状图结构，但是链接都是空的，无法正常显示原始的储存内容，而且也无法对树状结构图进行编辑，如下图。

而点击“打开剪贴簿”按钮后，左侧栏虽然显示了原始添加的网页链接，点击后也可以正常显示原始储存的网页内容，但是却失去了原始的文件夹树状结构，只剩下一堆网页链接的集合，如下图。

所以第一种情况是保留了原始的文件夹结构，但链接无效；而第二种情况是保留了原始的链接和网页内容，但是失去了原始的文件夹树状图结构。

from webscrapbook.

第一個問題，我想你還是沒有把路徑弄對。你把哪個資料夾放到索引器建索引，索引檔就應該要解壓縮放到哪個資料夾。

如果你是把 WebScrapBook 的剪貼簿資料夾放到索引器建索引，若瀏覽器預設下載資料夾是 C:\Users\Administrator\Downloads，而剪貼簿資料夾是 WebScrapBook，產生的索引檔應該是放到 C:\Users\Administrator\Downloads\WebScrapBook\tree\frame.html。

而如果你是從舊版的 ScrapBook 目錄建立索引，假如舊版 ScrapBook 目錄是放在 C:\Users\Administrator\Documents\ScrapBook，那麼產生的索引檔應該是放到 C:\Users\Administrator\Documents\ScrapBook\tree\frame.html 等等，依此類推。

至於第二個問題，如果前一個問題有處理好，這裡應該就能讀到和前者一樣的目錄樹。

from webscrapbook.

奇怪的是，我一开始就是按照您说的来做的，但是还是反复出现上贴里的问题，是否正常的情况下点击frame.html 后左侧栏显示的内容既包括文件夹树状结构，也包括有效的链接是么？

from webscrapbook.

當然，frame.html, map.html 等檔案都是記錄相對連結，路徑沒放對就會找不到。如果你還是搞不清楚，需要我們協助除錯，那就要麻煩你提供完整資訊了（預設下載資料夾路徑、剪貼簿資料夾路徑、拖放到索引器的資料夾為何，下載的索引檔解壓縮到哪裡等等）。

from webscrapbook.

您好，我把过程录成了视频，麻烦看下哈：

导入scrapbook x的文件夹资料.zip

设置后端服务器资料.zip

from webscrapbook.

第一個影片：應該是把解壓縮出來的 tree 資料夾放回原剪貼簿目錄，不是把 tree 裡面的檔案放回原剪貼簿目錄。

第二個影片：你似乎沒有執行過 wsb config -ba，導致一些相關路徑沒有初始化。先執行 wsb config -ba 再執行 wsb server，把之前產生的錯誤的 tree/ 目錄刪除，之後再匯入應該就不會有問題了。

from webscrapbook.

灰常感谢哈，问题已经解决。现在就是希望能把左侧栏的功能和体验继续完善了。如您上次所说，目前右键菜单尚无法支持。而左侧栏现在的不便之处主要在于：1）新的网页资料只能添加在最后，而不能指定目标文件夹。2）另外，对于文件夹和所存储的网页无法实现拖动功能。

from webscrapbook.

很遺憾的，無論是右鍵選單還是拖放功能，在多工及外連伺服器的條件下都不是很好辦，要支援手機更難。目前既然已經有下拉選單可以做到同樣的事（我知道沒那麼方便，但至少功能是有的），可能要等其他主要功能（比如搜尋）解決了才會開始處理。

from webscrapbook.

好的，如此看来firefox quantum似乎没有以前的版本对扩展的支持性能那样强大啊，真不知道这种改变对firefox来说是否真的是进步。firefox的最大优势之一就是丰富的扩展，但他们官方对扩展的扶持力度并不怎么大，或者至少方法不是很得当，完全是无政府状态，放任自流的风格。

from webscrapbook.

Hello,
I can't install PyWebScrapBook on xUbuntu18.04

$ pip install webscrapbook
Collecting webscrapbook
Could` not find a version that satisfies the requirement webscrapbook (from versions: )
No matching distribution found for webscrapbook

some help please ?
Sorry for my English

from webscrapbook.

Hello,
I can't install PyWebScrapBook on xUbuntu18.04

$ pip install webscrapbook
Collecting webscrapbook
Could` not find a version that satisfies the requirement webscrapbook (from versions: )
No matching distribution found for webscrapbook

some help please ?
Sorry for my English

This should be posted in the PyWebScrapBook project.

One cause of the error is using python2. You need python3.

from webscrapbook.

from webscrapbook.

@minshifu 拖放移動擷取項目的功能已在新版完成。

此問題已基本解決，因此關閉。個別細節問題可再另開 issue 討論。

from webscrapbook.