danny0838 / webscrapbook Goto Github PK

Hello,

I'm using scrapbook X (Putt Firefox on hold for 56.x) and as i read all the text, english is not my native language, it seems to me that there's little left going from X to Web version.

I understand that there are things not possible but as it seems the basic functions are gone to.
Little explanation:
As i say i use scrapbook X.
Every month I export de entire dir.
Doing that i rename de map / folder from datetime stamp to date (Year&Month) a hyphen domainname incl. subdomain a hyphen and dayHourMinute and seconds (Example; 201801-sub.domain.ext-01010101) to another folder called Year & subfolder month on my server (For humans easer to read and find if neccassary).
Thru a series of macros (Excel) i generate the above and a index.html file in the root of this folder and a CSV or TXT file for import in a database, currently MS ACCES but soon a PHP / MYSQL one.

For this i need the scrapbook.rdf file and for security reasons the index.dat with data as stored in Scrapbook X (id like 20171205203037, create 20171205203037, modify 20171205203037,
title Smallenbroek | Facebook, chars UTF-8, icon H3nktOa7ZMg.ico, source https://nl-nl.facebook.com/smallenbroek and comment)

Both they are gone,
GUI is very rudimentary,
The site indexer shows no, or I can't find it, usefull data,
and the view archive page makes a zip file (having trouble to get usefull data from it) that is, for me, not functional.

Is it possible to restore these two basic but importent elements?

Gr.

Theo Smallenbroek

The site indexer looked at the 4 files I selected to test it and finished up saying done but nothing happened after that. No index of the files.

Whilst (not?) creating the index it took the htz files and copied something to the downloads folder as zip files but did not remove these again when it had finished maybe because the process terminated incorrectly? Should there be 4 different zip files or am I not understanding what is supposed to happen?

Closing this because I don't think this does what I was expecting it to do.

I installed the add-on but as of now it seems like an alpha version, i can't edit webpages before saving them, there's no menu, webpages have to be manually saved to a specific folder (having the scrapbook folder in profiles was perfect) and so on.

Are things going to change before the release of firefox 57 ? and will we be able to import our scrapbookX pages to webscrapbook ?

Hi,

Web ScrapBook 0.26.2 / Firefox 58.0.2 (x64) / Windows 7SP1 (x64)

I've just installed Web ScrapBook (WSB) in order to save webpages without organizing them in mind. I was running the Save Page WE Firefox extension and it appears that WSB with the option Save captured data as -> Scripted Single HTML renders pages perfectly contrarily to the other extension. I mean complex pages because simple ones never pose an issue.

So: a Web page, a click on WSB toolbar button, and there we go, no fuss, complete page saved.
From there on I linger to understand the pertinence of saving pages in HTZ and MAFF packages. What's the point? You'll need WSB to open them, you won't be able to send such a packaged file to someone who doesn't run WSB... I must be missing something, but what?

Thanks.

All is said in title I think.

Any changelog would be great for know new fixes/features/caveats..

If it already exists, please give me a link.
If not, please add to the wiki (or a changelog.txt) file.

Thanks!

Hi
The View archived page have problem at Firefox 58. It can't shown saved page's images.
HTZ file or MAFF file have same result.
After decompress the HTZ file, the images was saved success.
And use Chrome to open the saved HTZ file, the images can be showed without problem.

In version 0.3.0 we use .htz extension as the zipped package of a captured web page. We also implemented a viewer that can directly load a .htz file in the browser, including the direct method (directly open the htz file in Chrome; the user must check "allow file url access") and the indirect method (open the viewer from the toolbar dropdown list and then pick an .htz file). Unfortunately, the viewer requires the requestFileSystem API, which is currently not available in Firefox.

Is there other way to implement the .htz viewer using currently available WebExtension APIs in Firefox? One idea may be the technique that EPUBReader use, but it seems that its source code is obfuscated and is not available for us to study.

We currently consider the best cross-platform HTML archive format to be zip based. Besides current .htz one, another zip based approach is MAFF that MAF addon uses, which, unfortunately, seems to be unmaintained. Besides zips, there are still many types of HTML archive, such as: .mhtml, .warc, .webarchive, or so. Is there other recommended format?

if there was only one add-on could be used, that will be scrapbook[x].
then the firefox updates to 'quantum', and the lovely scrapbook x could not work.
luckily we have this one, and thanks to you.

there's a small problem with it:
[env = Firefox Quantum, nightly build, 58.0b4 (x64), windows 10 x64]
if a url could not be opened quickly according to the connection quality, the saving opreation of WEBSCRAPBOOK will get the user a long time to wait before the save_dialog_box shows up to chose a path where to save the MAF doc.
and what's worse, during this waiting moment, the WEBSACAPBOOK was UNAVAILABLE, all options except the last two could not be used.
there are 2 advices:

1. change the operation logic: show the save_dialog_box firstly, and gain the data secondly.
2. if the webpage was already opened, (it seemed naturally of this occasion), just gain the data from the cache(why not?)

best regards.

Firefox 57+ (64-bit), WS v0.21.3, previous versions' status unknown

Trying to export settings using the Export button at the bottom of the Options page always fails, a blank moz-extension://####/core/options.html page is displayed and as far as I can see no backup file is created anywhere.

Please add a menu like Scrapbook x, I spent a lot of time to save and organize my webpages in scrapbook x, but in Firefox 57 there is no way to import them or see them like a menu sidebar.

Hi,
I use firefox 58.0a1 with the last version of webscrapbook (0.14.2) under Kubuntu 17.10.
If I define a folder to save (Options -> Capture - Saving -> Scrapbook Folder -> webscrapbook or ~/webscrapbook), I can't find this folder in my home folder. In fact, I have no scrapbook save.
If I define an other place, like "/media/svg/webscrapbook", it doesn't work too.
Thanks

Just like the "Open all pages in tabs" in old ScrapBook, now WebScrapbook can only open a file at one time. For the tree view we may handle it manually now (to change the filename, to put it into folder, ... etc.) if there are so many restrictions in WebExtension. However if it can open all the pages in a folder by dragging the folder it will be a lot convenient (to me).

You can not change the location where you save the captures.
When you put another location, replace the symbols: o .. with _ and use the desktop directory as a base.

... Not sure if it's clear.

But take an example:
In ScrapBook X we can drag & drop the "favicon" icon to ScrapBook X panel to.. capture it.
When we keep pushing ALT key, it create a bookmark instead!

.... In WebScrapBook.. It no working like this, so, it's this feature I ask to:

• 1. Dragging a "favicon" icon to the "icon" of WebScrapBook will trigger the opening of the WebScrapBook menu.
• 2. In this menu, since it's the same as the actual WebScrapBook menu, it's then giving us choices to: Capture TAB, capture TAB (Bookmark) or Capture TAB (source)
• 3. For capturing then, just release the "favicon" dragged on one of these choice.

And then, the TAB is captured like we would want: (physical), (bookmark) or (source) !

Thanks to considering this, as I save MANY often numberous of TABS in old-forme ScrapBook X like that.
It save me many precious seconds, that can become minutes/hours saved !

(the old panel miss me, since now we can't organize our capture in just.. drag & drop... but that's a thing... Firefox 57 is responsible of.. I hate it (Firefox 57))

scrapbook / webscrapbook ist a very good tool, I need and use every day. I'm very thankful that you took on the task to transport scrapbook to the new firefox. The proble is, up to now webscrapbook lacks many features (or I haven't discovered them yet) the old one had. Webscrapbook today is not really fit to be used. We need urgently the possibility to save and see and use what has been saved immediately.
The multiple steps we need now are disfunctional. Same goes for the former possibility to organize the saved file in folders before saving them.
Is there any hope you will find the time and the technical possibility to implement these functions again?
Grateful nevertheless
Robin

Please implement keyboard shortcut support for actions in the dropdown menu using the onCommand API:

https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/commands/onCommand

Hi, Danny & all,

Please forgive if I'm asking dumb questions or should have been able to figure out answers from what's already been written. My thanks to anyone who can answer my questions.

I'm one of those people who are so behind the times that I was still using Scrapbook by Gomita -- with lots of bookmarks saved. And Firefox set to automatically update. And not tuned in enough to know the Firefox 57/Quantum was coming. Then yesterday, Firefox updated. And I'm without a bunch of loved addons, most alarmingly Scrapbook.

As you can see if you look at the reviews under Scrapbook at https://addons.mozilla.org/en-US/firefox/addon/scrapbook/reviews/ and on Mozilla help https://support.mozilla.org/en-US/questions/1188657 I'm not the only one. The suggested work-arounds look difficult, and it does seem that, sooner or later, one should move to the new Firefox.

I've gotten to the point of figuring out that Web Scrapbook is the successor to Scrapbook X, which was the successor to Scrapbook. And I've even figured out where my Scrapbook data is. I've also looked a bit at the Github page on Scrapbook X.

It looks to me like Web Scrapbook is the logical extension to use to replace Scrapbook and I'm ready to install it. But I have a question and a suggestion.

Question: If I install it, can I bring in the data from Scrapbook? If so how?

Suggestion: The "Find a Replacement" on the Firefox 57 list of legacy extensions currently links to the MS OneNote Clipper add on -- which is not really the same as Scrapbook. Is there some way you can get Mozilla to change that and link to Web Scrapbook.

Finally, I am sure that I am not the only person who would be glad to contribute to support an extension to replace Scrapbook. You should add a donate link to the Web Scrapbook page.

Hello,

I would like to have the titile of a page in the filename. I did it by modifiying capturer.js in this way:

Replacing the line
filename = timeId + ".htz";
with
filename = timeId + " " + (data.title ? data.title : scrapbook.urlToFilename(sourceUrl));
filename = scrapbook.validateFilename(filename, options["capture.saveAsciiFilename"])+".htz";

and
filename = timeId + ".maff";
with
filename = timeId + " " + (data.title ? data.title : scrapbook.urlToFilename(sourceUrl));
filename = scrapbook.validateFilename(filename, options["capture.saveAsciiFilename"])+".maff";

Please add this to the official version.

Thank you.

Hi Danny Lin, I would like to request a new feature for Web Scrapbook: Save dynamic webpage processed by server-side with dynamic script (asset/resources, AJAX, JavaScript, XHR, and Websocket for debug test offline) and keep dynamic html/scripts to html file, example is https://kiwiirc.com/client , saving dynamic webpage but dynamic scripts not loaded. Example in this video: http://android2.altervista.org/Save-processed-webpage.mp4

Is it planned to implement multi-scrapbook feature with different profile directories as it was done in ScrapBook X?

OK.

Well, I used (just before making the update of Firefox 57) File Converter
(which is an extension of ScrapBook X), but as ZIP file not MAFF or HTZ.

ZIP are ok-readable in WebScrapbook aswell it says.
In fact, it working, but, what not working is.. WHEN "Directory Name" option has been selected when Converted through File Converter.

WHY?
Because that:

Web ScrapBook search for a direct root file named as index.html in the ZIP.

BUT..

Option "Directory Name" .. adds a sub-folder in the ZIP file named the same name the ZIP file is named (I didn't really know that, but I can't rollback.. so this issue can be fixed quickly by letting WebScrapBook reading index.html like that:

1. In ZIP it would search a root file named "index.html", if it isn't found, it will then use the ZIP NAME as sub-folder and then in this SUB-Folder search for a file named index.html !

ACTUALLY WORK:

20140210163203.zip
  |- index.html
  |- some other files... (such as resources: images, css, js, etc..)

ACTUALLY DON'T WORK:

20140210163203.zip
  | 20140210163203
         |- index.html
         |- some other files... (such as resources: images, css, js, etc..)

Thanks for considering this issue for making the thing smooth for everyone..

If a stylesheet is in another domain form the current web page, extension scripts cannot read the stylesheet rule from them. As a result, if "save current used" is set for "background images" and "fonts", background images or fonts referenced by the stylesheet are never saved.

Software: Chromium 63, Linux, WebScrapbook from Google Store

On this page, I saved the following text selection:

s a folder, a single HT

The resulted archive is 2.2MBs large and contains all sorts of image resources, most notably "press-header.jpg" that is 1.0MBs in size. Curiously enough, it is not referenced in the selection's HTML source code.
That makes saving a selection ineffective, slow and rather cumbersome.

As a temporary solution and/or an alternative method of saving selections I suggest you take a look at this Chrome addon, the github link

Hereby attached the archive(note I had to change the extension due to github's limitations):
GitHub - danny0838_webscrapbook_ A browser addon that captures the web page faithfully with highly customizable configurations. This project inherits from ScrapBook X.zip

Ok. Again, more a question than an issue...
Anyway, I think, it's an important reguards question on visual side of WebScrapbook.

Well, today, I was aware of how will be the UI of the new version based on WebExtension of Scrapbook, named as WebScrapbook.

I mean, it is a final UI ?

Is it impossible to have side panels like we had in Scrapbook X ?
It's great for organizing with a side bar panel (always opened in my Firefox, at work/at home !
I can think to live without but it would be a pain to organize/accessing!

I KNOW you do your best and must develop the new version based on what WebExtension authorize.. so there is no any workaround possible?
Since it seems not as you said it here: #2 (comment)

The fatal day is coming...
I'm worry about the futur of addons with WebExtension, but more worrying about the UI of WebScrapbook, how it would be without sidebar!?

Explications on what I reffering as sidebar:

In Scrapbook X we had that sidebar: https://addons.cdn.mozilla.net/user-media/previews/full/146/146924.png?modified=1416635343
In WebScrapbook the sidebar seems gone and as UI we have that instead: https://addons.cdn.mozilla.net/user-media/previews/full/190/190179.png?modified=1508913059

Thanks for answer, and keep up your great work, again, do your best!

Firefox 57 with newest webscrapbook.
Say, I have archived webpage opened in viewer, I right click on some image to save it and instead it tries to open the link to article behind that image. Right clicking on other elements (like links) caused that problem too.

仅概览中缺失，附加组件详情中不会。加 "48": "core/scrapbook_128.png", 可解决。
自动缩放就可以，提供两份图片则或许有细微性能提升，减少缩放一个1KB 128px的图片。
https://developer.mozilla.org/en-US/Add-ons/WebExtensions/manifest.json/icons

When clicking on webscrapbook icon there is presented a menu that have few entires, that are not clear (at least I did not find info about them), like:
Save tab (source) - seems same to "Save tab"
Save tab (bookmark) - saves bookmark?
Save following tabs - saves current tab? bug?

Can you some info about those somewhere? Or maybe add tooltips for menu entries.

I don't now how to put an enhancement flag on this but....

Could the file name (htz etc) or folder name take the name of the tab? It worked once 'out of the box' but once I directed it it's default directory all I get is the date and time as a file or folder name.

nodes in web pages are not properly hidden in the viewer as they should be when saved as MAFF file.0

Example URL: http://www.ebay.com/itm/182829397448

When saved as single HTML, node is properly hidden:

When saved as MAFF, node is not hidden. It seems to be a bug of the viewer, as the node is not shown if you browser the file using the old MAFF extension in Firefox 56:

几乎不会用到右键扩充菜单，需要保存网页的时候会直接从工具栏扩展程序按钮点击保存。

Are there any unit tests for this?

I know you aren't done building out the features for WebScrapbook, but I'm curious, is it technically possible to implement an AutoSave feature, or do the APIs block such behavior?

Thanks for the great extension!

This is a very important feature in ScrapBook. It will be great and I'll appreciate very much if this feature can be implemented

Are you planning to add dedicated "Capture Page as" window, as in Scrapbook to do custom/advanced saving (multilink saving, depth, folder, images, files, extensions) aka

P.S. Rough TODO page for this addon would be nice.

Hi Danny Lin, I would like to request a new feature for Web Scrapbook: web page recording (web page with all assets, css and JavaScript recorded during the session and saved to html file). Its possible add this feature in Web Scrapbook?

https://stuk.github.io/jszip/

And even allow convert existing items, allowing the configuration of compression levels and other parameters.

There is possibility for the Web Scrapbook to work on mozilla.com pages.

For normal work on addons.mozilla.org pages

1. in the address bar: about:config
2. right-click to create a logical
3. setting name: privacy.resistFingerprinting.block_mozAddonManager
4. value: true

But "Capture tab" menu items of Web Scrapbook are disabled and I can not capture these pages. Can you create option to enable them or replace this functionality by some notification?

UPD
Also I suggest you add this note to your addons.mozilla.org page as author of Screengrab! did.

It is more a question than an issue indeed, I'm thinking (and I think I'm not the only one who thought that..), to the best process to recover partially (it will be awesome if it will be fully !) all our Scrapbook X/Scrapbook Plus/Scrapbook base data and then making them working in WebScrapbook ?

By recovering, I mean recover:

• Scrapbook X (and co) organization (seperators, view folders, structure)
• Which items are LINK (simple link bookmarked), and which items are FILE (simple saved page with its assets).
• our Notes (Scrapbook X note)
• our added custom icons to folder in view panels.
• just... like it was before Firefox 57 (per example: exactly how it was in Scrapbook X).

Since it is actually not a database (not only I mean):

The actual important files of all Scrapbook X versions (including old formers Scrapbook Plus/Scrapbook base) are stored in some physicals folders here:

Web pages saved and their assets:
C:\Users\{windowsUsername}\AppData\Roaming\Mozilla\Firefox\Profiles\{firefoxSessionName}\ScrapBook\data

Scrapbook X (.rdf) structure/organization/treeview Auto Backup:
C:\Users\{windowsUsername}\AppData\Roaming\Mozilla\Firefox\Profiles\{firefoxSessionName}\ScrapBook\backup

Scrapbook X (.rdf) used structure/organization/treeview (when viewing in Firefox):
C:\Users\{windowsUsername}\AppData\Roaming\Mozilla\Firefox\Profiles\{firefoxSessionName}\ScrapBook\scrapbook.rfd

Scrapbook X (.rdf) cache:
C:\Users\{windowsUsername}\AppData\Roaming\Mozilla\Firefox\Profiles\{firefoxSessionName}\ScrapBook\cache.rdf

Scrapbook X (.txt) Folders main id:
C:\Users\{windowsUsername}\AppData\Roaming\Mozilla\Firefox\Profiles\{firefoxSessionName}\ScrapBook\folders.txt

Scrapbook X (.txt) Folders main id:
C:\Users\{windowsUsername}\AppData\Roaming\Mozilla\Firefox\Profiles\{firefoxSessionName}\ScrapBook\folders.txt

So all thoses files that are important to Scrapbook X to restoring/importing/making of use of our existing content in Scrapbook X to WebScrapbook..

I use Scrapbook for .. years now... (and I'm not the only one I think) who love organizing Scrapbook data like I want ! My treeview is perfectly sorted!
I will be sad if all my hard work on sorting/organizing/treeview work would be gone....

Thanks by advance for your answer @danny0838.
I know it's not an easy task to making Scrapbook X working in Firefox 57 under the limitation (absurd) of WebExtension...

It says addon is corrupted after downloading... Using FF android v49

Hello friend. I have a question about folder tree. May be I am old-fashioned, but for comfortable information searching is a tree structure. I not saying about old interface like scrapbook X, but may be add text-box for typing folder name in a save process? It just for generation tree.

I cannot even save bookmarks. There is definitely activity, the "busy" cursor shows for half a second or so, but that is it.

我一直用scrapbook + scrapbook MAF creator来保存maf格式文件，查看是使用文件管理器totalcommander的一个插件decMaffWLX来查看https://totalcmd.net/plugring/MaffWlx.html。
最近试用webscrapbook，发现一个问题：
使用webscrapbook保存的maff文件在使用decMaffWLX查看的时候，网页中的图片不能显示，其他文字、样式都正常。
然后把这个maff文件解压，解压得到的目录再手动压缩为zip文件并改后缀为maff，然后再使用decMaffWLX来浏览，图片就可以显示了。
这是否说明webscrapbook的maff格式和scrapbook MAF creator的有所区别？
同样的情况在htz中也存在，解压htz，把得到的所有文件重新打包为zip并改后缀，就正常了。
作者能不能帮忙看看这个问题？ 谢谢！

另外，现在默认保存目录是%USERPROFILE%\Downloads\下面，这个目录如果能自定义就好了。

Mozilla Archive Format and UnMHT support saving and viewing MHTML format, both of them are wide spread until FF 57.0 released. But Web ScrapBook doesn't support it. MHTML is an open standard formatted in RFC 2557.

Web ScrapBook 是目前 （firefox 57+）唯一能准确保存网页原始状态的插件。相比 UnMHT 而言，存储的文件会大很多。保存为脚本化单一 HTML 文件虽然能缩小体积，但仍然比 MHT 格式大很多。
希望能像 Save Page WE 一样采用 URI 编码保存 css 和 js，base64 编码保存图片等资源，以减小保存文件的体积。

Is it possible to display the original URL when opening an archive file, just like the Mozilla Archive Format extension did? And may I request for this feature if it is able to implement?

Thanks for your great work.

Some suggestions for MAFF creation improvements in presumably increasing difficulty of implementation.

1. Add the option to archive all files with no compression, like MAF (the extension) has with the maf.advanced.maff.compression;none preference.

2. Modify the output directory structure by putting only index.html and index.rdf directly below the main timestamp directory and all other files inside a subdirectory. When opening manually extracted archives in a offline browser, it is difficult to find the index.html in pages with many other saved files.

PAGETITLE.maff
	| TIMESTAMP/
		| index.html
		| index.rdf
		| SUBDIR/
			| file.css
			| file.png
			| ...

3. Add the ability to save multiple tabs within a single MAFF archive, in such a way that if manually extracted the resulting directories recreate the ordering of each tab as displayed in Firefox when alphabetically sorted.

Firefox tab bar: Tab-1 | Tab-9 | Tab-4

TITLE.maff
	| TIMESTAMP_001/
		| index.html
		| index.rdf
		| DIR/
			| tab-1.css
			| tab-1.png
			| ...
	| TIMESTAMP_002/
		| index.html
		| index.rdf
		| DIR/
			| tab-9.css
			| tab-9.png
			| ...
	| TIMESTAMP_003/
		| index.html
		| index.rdf
		| DIR/
			| tab-4.css
			| tab-4.png
			| ...

4. If suggestion 3 is implemented, make links between tabs work offline. For example, if Tab-1's document links to Tab-4, after saving both tabs to the same MAFF archive the offline copy of Tab-1 should link directly to the offline copy of Tab-4.

Keep the manifest-firefox.json will bring trouble for debugging add-on, select it will load the manifest.json of the folder instead of the manifest-firefox.json. This may cause the temporary ID to be used, then bug 1323228.

I guess you want to avoid Chrome warning the "applications", it seems unavoidable (unless a gulpfile to workflow), but the warning is harmless.

Hello.
I have old (really old) folders with Scrapbook data.

May I import it to webscrapbook?