Code Monkey home page Code Monkey logo

tumblthree's Introduction

TumblThree Logo

TumblThree - A Tumblr Blog Backup Application

Build status GitHub All Releases (archived repo) Github Releases (current repo)
first-timers-only PRs Welcome contributions welcome

TumblThree is a free and open source Tumblr and Twitter blog backup application. It downloads photo, video, audio and text posts from a given Tumblr or Twitter blog. Tweet
It is the code rewrite of TumblTwo, using the Win Application Framework (WAF) and C# with WPF and the MVVM pattern.

TumblThree Main UI

Features

  • Source code at GitHub (written in C# using WPF and MVVM)
  • Multiple concurrent downloads of a single blog and/or different blogs
  • A clipboard monitor that detects blogname.tumblr.com urls in the clipboard (copy and paste) and automatically adds the blog to the bloglist
  • A settings panel (change download location, turn preview off/on, define number of concurrent downloads, set the imagesize of downloaded pictures, set download defaults, enable portable mode, etc.)
  • An option to skip the download of a file if it has already been downloaded before in any currently added blog
  • Preview of photos & videos
  • Download of Gif instead of WebP/Gifv images from Tumblr
  • File rename functionality
  • โญ Download of Twitter blogs
  • Image viewer with slideshow mode ๐Ÿ†•
  • Automated update process ๐Ÿ†•
  • Group blogs into collections ๐Ÿ†•
  • Choice of download format for .pnj links ๐Ÿ†•
click for more
  • Internationalization support (several languages available)
  • A download queue
  • Autosave of the queuelist
  • Save, clear and restore the queuelist
  • Uses Windows proxy settings
  • A bandwidth throttler
  • An option to download an url list instead of the actual files
  • Set of a start time for a automatic download (e.g. during nights)
  • Change of blog settings of multiple selected blogs at once
  • Uses SSL connections
  • Taskbar buttons and key bindings

Backup/download of (click names to expand)

  • Blogs / Hidden blogs
    • Download of photo, video (only tumblr.com hosted), text, audio, quote, conversation, link and question posts
    • Download meta information for photo, video and audio posts
    • Download inlined photos and videos (e.g. photos embedded in question&answer posts)
    • Download of all image sizes possible (SVC, API only for newer blogs, higher resolution not possible for old blogs)
    • Support for downloading Imgur, Gfycat, Webmshare, Uguu and Catbox linked files in tumblr posts
    • Download of safe mode/NSFW blogs
    • Allows to download only original content of the blog and skip reblogged posts
    • Can download only tagged posts
    • Can download only specific blog pages instead of the whole blog
    • Allows to download blog posts in a defined time span
    • Can download hidden blogs (login required / dash board blogs)
    • Can download password protected blogs (of non-hidden blogs)
  • Liked/by and Likes
    • A downloader for downloading "liked by" (e.g. https://www.tumblr.com/liked/by/wallpaperfx/) and "likes" (e.g. https://www.tumblr.com/likes) photos and videos instead of a tumblr blog (login required)
    • Download of all image sizes possible (SVC, API only for newer blogs, higher resolution not possible for old blogs)
    • Allows to download posts in a defined time span
  • Tumblr searches
    • A downloader for downloading photos and videos from the tumblr search (e.g. http://www.tumblr.com/search/my+keywords)
    • Download of all image sizes possible (SVC, API only for newer blogs, higher resolution not possible for old blogs)
    • Allows to download posts in a defined time span
  • Tumblr tag searches
    • A downloader for downloading photos and videos from the tumblr tag search (e.g. http://www.tumblr.com/tagged/my+keywords) (login required)
    • Download of all image sizes possible (SVC, API only for newer blogs, higher resolution not possible for old blogs)
    • Allows to download posts in a defined time span
  • Twitter Blogs
    • Download of photo, video and text posts of blogs
    • Download meta information for photo and video posts
    • Allows to download only original content of the blog and skip reblogged posts
    • Can download only tagged posts
    • Allows to download blog posts in a defined time span

Download

Latest versions can be found here.

Please keep in mind that probably only the latest version is functioning properly since the platforms evolve and from time to time change their data structures which makes changes in TumblThree necessary again. So update your application regularly.

Application Usage and Getting Started

Read our wiki page about Application Usage.

The default settings should cover most users. You should only have to change the download location and the kind of posts you want to download. You can find more information in our wiki Getting Started and Insights.

Feedback and Bug Reports

If you like TumblThree, give it a star star (at the right upper corner of the page)!

You are welcome to participate in Discussions (our forum). If you don't have a GitHub account yet, please sign up for one, it is free.
We appreciate it, if you send us your feedback or file a bug report. In case of an error, preferably just fill out a Bug report.

In case you don't like to register an account for some reason, but still want to provide feedback or a bug report, use the following web form: tumblthreeapp.github.io/TumblThree/feedback.html

How to Build the Source Code to Help Further Development

Contributing to TumblThree

We like the all-contributors specification. Contributions of any kind are welcome! If you've ever wanted to contribute to open source, and a great cause, now is your chance!

Contributors โœจ

Last but not least see also the list of contributors who participated in this project.

tumblthree's People

Contributors

amigre38 avatar arrowsmith001 avatar deege216 avatar dependabot[bot] avatar elipriaulx avatar emphasia avatar gitter-badger avatar johanneszab avatar khintze1 avatar khodyn avatar kulesy avatar rexius41 avatar salrana avatar shadowblade72 avatar spl1ne avatar thomas694 avatar yfdyh000 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tumblthree's Issues

Downloading only images with a specific minimum resolution

First thanks for this great application. I use it often to download images from my selected tumblrs.

Often downloaded images are of a low quality and I delete these usually. Would it not be possible to put a selection in the configuration section where it will be possible to specify the minimum required resolution? Pictures with a resolution below this setting will then not be downloaded. If this will be possible, then I will set it to accept only tumblr pictures with a resolution of 1080.

Minimise existing presence.

The existing TumblThree repository should direct users here. Updating the Readme, closing the issues, and making it obvious in the releases that this is the new home of TumblThree will help reduce discontinuity and confusion in the future.

Documentation

Find a home and process for creating documentation. This doesn't need to entail creating documentation, but at the least, determine how documentation will be available in the future, how people can contribute, and create issues to create documentation that can be absorbed into future milestones.

Image downloading not working properly anymore/Local files being deleted?

Hello. Up until about 4 months ago everything worked fine.
However, now the app won't download latest images anymore. Some blogs (urmeimu.tumblr.com) are skipped completely, not a single image downloaded.

For some other blogs (yaoi-sei.tumblr.com) it won't download any images newer than 4 months at all.

However, it adds the info about the skipped images to a text file called "texts" instead.

For a ton of other blogs it downloads newer images, but not all, a lot of them are missing if I compare it.

Dunno what exactly broke 4 months ago. But it really needs a fix.

Also, I did a test run and copied a folder with 60k pics. Then ran a full scan with the index file intact (only readded the blog in the app). Later I copied the old folder into it. This SHOULD have resulted in an error, as in the 60k pics already did exist before. However, I now have 120k pics in the folder, because I only got an error for about 660 duplicate images (???). So I wonder if pics are being deleted or renamed from the local folder even when the index file is intact. At first glance I also don't seem to have a whole lot of duplicates, so it really seems like images were deleted from the local folder merely by running a new full scan.
Another folder is missing a few files after running a new full scan. I also checked a few random pics, and it seems that locally stored files were touched during the scan (the size and date changed, also the name). Locally stored files should never be touched.

I'm worried that when doing a full re-scan, it checks every local file and only keeps the images that are still online, removing all that were deleted by Tumblr. If this is true, it's a huge issue because sometimes URLs glitch and you need to readd them.

Edit: I just verified that there ARE indeed images missing after a full re-scan, using the same folder and same index file. I have no idea why this is, but I compared the folders and some images that were copied BEFORE the re-scan were GONE afterwards. This really needs to be fixed!

No .EXE

My apologies if this is a humiliating newb question, but having downloaded the zip, I can't find any EXE. Should the folder within the zip called "TumblThree-master" somehow be converted to an EXE file? Thanks in advance.

Enhance code base to allow new website crawler

Currently there is quite an effort necessary to support a new website in TumblThree (for example reddit or instagram).

Roughly, the following steps have to be done:

  • Implement the ICrawler interface and override the Crawl method to start the crawler and a DownloadBlogAsync task from the IDownloader interface. Here, most of the code should be equivalent.
  • The url validator needs adjustments to detect proper urls.
  • Add a BlogType.
  • Add a Crawler to the CrawlerFactory.
  • Add a BlogType to the BlogFactory.
  • You might have to to add a new DetailsView.cs if you want different checkboxes or statistics.

It might be worth streamlining these tasks. At least for a simple RegEx grepper there should only be a few lines of code necessary (i.e. the url, the name of the new downloader/crawler and the RegEx).

Data loss occurs when "access denied" videos replace local files if index files are not present to force skipping.

So I think i've discovered some possibly unwanted behavior. I wanted to see if it was safe to reload blogs without their index files and re-scan. I found that it is NOT safe and data loss WILL occur in specific circumstances and it could get much worse if Tumblr changes how pictures are pointed to.

So here is what I did. I found a NSFW blog that I know for a fact has "access denied" videos on it and that I had a local version of from months ago where the videos were still working. Then I ran the following tests:

Test 1: Placed old downloaded files along with old index files into new directory then ran the re-scan.
Result: Index files were updated to reflect the current post count. Which is a bit annoying and misleading but the local files were not touched. Ran multipar and verified all files were intact and not touched other than index files which got updated.

Test 2: Placed old downloaded files into folder WITHOUT index files and did a full blog re-download.
Result: Any videos currently listed in the blog with "access denied" attempted re-download but failed and local copies were DELETED.

I'm not sure whether this only happens with videos or if it can happen with images as well.

However, if tumblr started to change access to full URL images on a widespread scale it could be very bad. Someone trying to re-download blogs without index files could suffer catastrophic data loss. Millions of backed up files gone in an instant because no index files to tell the program not to touch anything in the local folders. Even with index files i'm not comfortable updating my blogs right now just because how the software handles local file disputes.

I'd recommend some sort of software update adding a feature to "backup old files" with different file sizes so that error pages, access denied pages or any other kind of replacement image/video never replace local files. Then the user can choose which file they want to keep, assuming anything is saved at all. In this case nothing is actually saved and so the only file left would be the working one. Though it might end up re-named in the process that's still better than outright deletion.

New files will be downloaded?

  1. Since tumblr introduced the new naming conventions ThumblTree misses lot of files of new posts.
    Older post are still working.
    When TumblrThree will be fixed to handle new naming, it will be download now skipped files?
  2. Last file remains opened by TumblrTree therefore I can't delete it unless I close TumblrTree.

Coding Standard

Lets find a common ground regarding the coding standards and conventions. We can set up Resharper, Stylecop, or analyzers to check and enforce these, and work progressively towards cleaning things up as we go.

Meta only saved/available for blogs

For Tumblr Tag Search/Likes, it seems not possible to download meta.

Is that intentional?

I can only enable meta downloads in the "Details" tab when the site is a Tumblr blog.

COMException in OnClipboardContentChanged (maybe Remote Desktop/RDP/mstsc related?)

Original issue TumblThree/#379 by @aziraphale.

Hiya,

  • TumblThree version: 1.0.8.68 (the current latest)
  • OS: Windows 10 Pro x64 (10.0.17134.345 according to cmd.exe!)
  • Event Viewer saved crash events: TumblThree crash - maybe RDP-and-clipboard-related.zip

First of all, great work on this application! No, it's not perfect in a few areas, but it's incredibly functional and very useful once you figure out the quirks, so for something that appears to have been developed almost entirely in one person's free time, I'm impressed! ๐Ÿ˜ธ

Of course, I'm writing this because the program crashed on me yesterday, in a way that was quite annoying, but which I'm hopeful can be fixed nice and easily - perhaps just by adding a try/catch block, I dunno!

The reason it was rather annoying is because the crash was "at random" while processing the queue, so I didn't notice it until quite some time later, and once I did notice it I had to delete and re-add the unfinished blogs in order to get their blog data to be repopulated (which I think is a known bug). Also I don't recall seeing any kind of "this program has crashed" dialog box when it crashed - I just suddenly noticed that TumblThree was no longer running!

Please see the attached Windows Event Viewer saved event files (zipped) for details of the two crashes I experienced. They both seem to be pretty much identical, occurring during the TumblThree.Applications.Controllers.ManagerController.OnClipboardContentChanged method.

I had previously had this application running for quite a while without it crashing, and the only notable difference that I can think of and which could be at all relevant is that at the time of both crashes, I was logged into the PC running TumblThree via Microsoft's Remote Desktop (aka RDP/RDC/mstsc/etc.) from my PC at work (the RDP session was minimised, which is why I didn't immediately notice the crash).

My Remote Desktop session was configured with clipboard sharing enabled, and my completely wild guess is that, at a point where I copied something to the clipboard on my work PC (the Remote Desktop client), RDP synchronised that clipboard content with my home PC (the Remote Desktop host), and then RDP's attempt to write to the clipboard interfered with TumblThree's attempt to read from the clipboard.

Hopefully the actual content of my clipboard isn't relevant, as I couldn't exactly tell you what that would have been! Given what I'm normally doing at work, it's likely to have been either text copied from JetBrains' PhpStorm (which also copies a rich-text version of copied code, so that I can paste code samples complete with colouring into HTML contexts such as emails), or some random image data. I couldn't tell you more accurately than that!

Thanks again for this tool, and good luck fixing this! ๐Ÿ˜บ

Use HttpClient instead of HttpWebRequest

The code is full with error prone exception handling caused by the HttpWebRequest api. Additionally the async variant of the GetReposone method, GetResponseAsync() is broken as it never timeouts if the underlying connection drops and lead me to implement the TaskTimeout Extension.

From the MSDN docs (the GetResponeAsync() method uses BeginGetResponse() internally (thanks for not telling MS!)):

The Timeout property has no effect on asynchronous requests made with the BeginGetResponse or BeginGetRequestStream method.

The newer interface HttpClient was designed with async in mind and should be used instead. It also doesn't use Exceptions to handle non-successful web responses (i.e. status codes 300-499).

See also these issues for more: TumblThree/#116, TumblThree/#214.

Rethink the "database"/data structure

Currently, for each blog there are several files saved in a folder called \Index that contain all the relevant blog information and settings used by TumblThree as well as the downloaders state.

Initially I don't wanted to have the overhead of a real database and for a simple application this seem sufficient. But maybe there is a more convenient way to handle this which also offers transactionality. Currently those files (as well as the settings) are quite seldom updated/saved (i.e. settings: Only if the application is shutting down; blog: if the download is finished or canceled). Maybe there is a way to improve the download state consistency in case of failures.

Images saving as string of numbers instead of usual tumblr_XXX

Tumblr seems to have changed how images are stored on their end, so instead of gifsets being saved by the program as:

tumblr_nmgkwurThj1sm6x1so1_500
tumblr_nmgkwurThj1sm6x1so2_500
tumblr_nmgkwurThj1sm6x1so3_500

It has been saving them as stuff like:

0c4f154aa27dd3268b23e6de64621d3141d293d6
0e4d30a891f94ccd9561a06d2aae4f9abcf6db4c

Even when they are the same gifset. Can anyone tell me if I can fix it on my end so the image sets at least stay together after they're saved?

Kickoff Meeting

Conduct a 'kick-off meeting' to discuss project direction, expectations, etiquette, contact details if necessary, and generally get to know all interested parties.

Readme

The ReadMe should be reviewed to highlight the current state. This should probably be tackled after many of the other tasks this milestone have been completed.

This also might include:

  • Reviewing the licence
  • Reviewing the code of conduct
  • Reviewing the contributing guide

Sometimes app stops downloading blogs

Sometimes when switching from blog that finished downloading to next one in queue app simply fails to stasrt it. It doesn't appear any single particular blog causes this. There are no error messages nor entries in event log.

Add Blog button doesn't do anything

I'm pasting the URL into the textbox, and clicking Add Blog, but nothing happens. I also tried clicking the Crawl button, but that also doesn't seem to do anything.
I'm running Windows 10, and the latest build under Releases, if that helps.

Use json.NET for all json related things instead of the DataContractJsonSerializer and/or the JsonReaderWriterFactory.

The usage of json.NET instead of the DataContractJsonSerializer/JsonReaderWriterFactory should make the life easier with wonky, incomplete data coming from the Tumblr api/SVC (Tumblr website service) as it can be setup more fault tolerant. Additionally, it's easier to use.

The main reason TumblThree currently doesn't use it, is because I wanted to learn the .NET framework and C# by developing TumblThree.
As such, I wanted to use as few third party libraries as possible with the additional goal of implement as much as possible by myself using the raw .NET framework.

In retrospect and having "finished" my initial goal, I think json.NET would be a great library to use.

Use of old API doesn't display high resolution images

Yeah, the API limits are pretty high now for v2. I'm not sure how the v1 requests look, as I was only looking at v2 and v2npf (neue post format).

It looks like most of Tumblr is switching to this neue (new) post format, which changes how they're displaying images.

As stated in the original issue, the image format changed from _1280 to embedding the resolution into the URL, as well as having a different URL for each resolution. I've included two API calls below. One with the NPF flag for requests and one without. From what I've seen, it's hit or miss as to which blogs have switched to the new format. Some still use the _1280, while others are using the new format.

You'll see that with the npf=true you have different resolutions available.
Drill down to Response > Posts > 0 > Content > 0 > Media > {1,2,3,4,5,6,7}

Without npf=true the media is contained in the "body" tag and only shows up in 640x960 resolution.
Drill down to Response > Posts > 0 > Body

With NPF - http://api.tumblr.com/v2/blog/crazycatswiftie89.tumblr.com/posts?npf=true&id=186983242460&api_key={Insert your API key here}
Without NPF - http://api.tumblr.com/v2/blog/crazycatswiftie89.tumblr.com/posts?id=186983242460&api_key={Insert your API key here}

If this is confusing, let me know and I'll try to explain it better.

Originally posted by @ShadowBlade72 in #48 (comment)

Build System

I'm not sure what our options are, but it would be great to get some CI up and running. If we could just tag or merge to create releases, and get some feedback about code coverage, rule violations etc as we create PRs, then we can set a good precedence for a quality product and minimise the labour involved in doing so manually.

[Feature Request] Tumbex support

Tumbex can still access NSFW images by accesing the media.tumblr.com URL, could it be possible to add support for it in TumblThree?

Current Release.

Create a new release of TumblThree, in its current state such that there is no need for consumers to interact with the existing TumblThree repository.

Photos not being downloaded

It appears that some photos are not being downloaded. This looks to be my fault due to refactoring. I renamed some properties, and although retaining the data contract, I didn't realise we were forming property names with string fragments and accessing them via reflection.

This should be a matter of fixing the string formation, or replacing the reflection with direct property access (which will be refactor-safe). We should also verify and address all other uses of GetProperty.

I will likely have a PR for this tonight.

Establish gitter or slack presence for communication.

To improve the coordination and communication between (potential?) contributors, it would be worth while having somewhere that can track long standing conversations in public view, for the sake of discussing direction and coordination.

Rebrand

I'm not sure if there is any sentimental attachment to the current logo and style, but irrespective it would be useful to lock in logos, colours, and general stylistic preferences among those involved early to minimise double handling of the more 'touchy feely' areas. Maybe a style guide or something? Some general consensus of how things should look, feel, and interact with users.

Settings never saved

Hello I'm on 1.0.10.1008
Tumblethree is on a computer , blogs on an external hard drive
I recovered blogs from another computer saved with a previous tumblthree who crashed on this previous computer ( lost path ans settings on half the bolgs and some general settings

The problem is continuing with the newly installed tumblthree on this new computer : new settings are not saved : new path new blog settings( download options) I have to reset manually everything any time I run tumblthree

It seems the the new tumblethree is keepeing the old troubleshootings evenif it's not the same tumblethree version the same computer and the same harddrive for storage???
I can't find a clear path to a settings file which save the erronous informations to destroy them

Thanks for your work

Support for personal tumblr likes

Original issue TumblThree/#337

TumblThree should be able to download personal tumblr likes via the https://www.tumblr.com/likes url. Currently, it can only download likes using the publicly visible likes page at https://www.tumblr.com/liked/by/blogname. The capabilities of this page however were changed at the 17th December 2018. Thus, enabling the personal likes page would result in more downloadable content.

The costs for this feature should be minimal as it probably only requires to change the UrlValidator (might relate #19) allowing it to add the personal likes page and possibly the "likes-url" builder within the TumblrLikedByCrawler.cs.

Error 429

I have constantly en error 429 limit exceeded. no matter what I do it gives me the same error. I tried to make a number of API connections lower, but it didn't work

Crash during Check Blog Online Status

Manually migrated issue #392:

Exception info from Event:
Application: TumblThree.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.NullReferenceException
at TumblThree.Applications.Crawler.AbstractCrawler.HandleUnauthorizedWebException(System.Net.WebException)
at TumblThree.Applications.Crawler.TumblrBlogCrawler+d__9.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at TumblThree.Applications.Controllers.ManagerController+d__61.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at TumblThree.Applications.Controllers.ManagerController+<>c__DisplayClass60_0+<b__0>d.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at TumblThree.Applications.Controllers.ManagerController+d__60.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at TumblThree.Applications.Controllers.ManagerController+d__59.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at System.Waf.Applications.AsyncDelegateCommand+d__18.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate)
at System.Windows.Threading.DispatcherOperation.InvokeImpl()
at MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at MS.Internal.CulturePreservingExecutionContext.Run(MS.Internal.CulturePreservingExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Windows.Threading.DispatcherOperation.Invoke()
at System.Windows.Threading.Dispatcher.ProcessQueue()
at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef)
at MS.Win32.HwndWrapper.WndProc(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef)
at MS.Win32.HwndSubclass.DispatcherCallbackOperation(System.Object)
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate)
at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(System.Windows.Threading.DispatcherPriority, System.TimeSpan, System.Delegate, System.Object, Int32)
at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr, Int32, IntPtr, IntPtr)
at MS.Win32.UnsafeNativeMethods.DispatchMessage(System.Windows.Interop.MSG ByRef)
at System.Windows.Threading.Dispatcher.PushFrameImpl(System.Windows.Threading.DispatcherFrame)
at System.Windows.Application.RunDispatcher(System.Object)
at System.Windows.Application.RunInternal(System.Windows.Window)
at TumblThree.Presentation.App.Main()

To answer questions from original issue.
When I added blog (https://jofamo.tumblr.com/) it was normally accessible. Sometimes later it got deleted. Currently no other blog with custom domain got deleted. If needed I can locate some of them for future reference.

Tumblr v1 API has new day limits

Tumblr has imposed new day limits on v1 APIs. Each version of API (XML vs JSON) has its own limit. Precise info is not available so it is not clear how many requests can be made and when it resets.

Note: I have rewrote original bug report.

TumblrThree cannot import datas from flagged posts "as NSFW"

Since january 3rd, TumblThree is no longer able to find any data from flagged as explicit posts, it looks like these posts don't exist anymore. The total number of posts appearing in [number of downloads] column before crawling is correct, but these posts are missing, no infos neither in images, images_url or texts files after backup. Please, do you know a solution? Thank you for your help.

Redesign the user interface (GUI)

Original issues TumblThree#332, TumblThree/wiki

Simplify application usage by re-organizing its layout or even redesigning the application user interface.

Possible enhancements (by @gpriaulx):

  • Try to contextualise everything visually. Easier said than done I guess...
  • Get rid of textual labels and instead rely on tooltips and layout.
  • Show less stuff by default.
  • Show more visual feedback.
  • Reduce the start/stop/pause/resume concept to just pause/resume. It's either downloading stuff from the queue, or it isn't. Only show the relevent action. EG Pause OR Resume.
  • Move the user account to be a 'sessional' thing rather than a 'settings' thing. Still persist it of course.
  • Restore previous state with respect to pause/resume, otherwise if new launch with no items just be in crawl mode already such that as soon as something is queued it 'just works'.

Something like this I guess - please forgive my terrible mock and (lack of) design skills:
49936487-305d5d80-ff28-11e8-9836-8a931e7ba8ba

[Feature Request] Support for Tumblr archives/alternatives

Before Tumblr's shitty youth protection policy became active some people exported their accounts to platforms like 2mblr.

While their Tumblr page seems to be deleted (and therefore can't be downloaded via TumblThree) the files are still on their servers. 2mblr for example doesn't mirror images, it just scraped the media.tumblr.com urls, which are still accessible.

[request] jar version of tumblthree

Hi there! Is there any chance to release the *.jar version of tumblthree?

I've been using tumblthree for several years and this this is the best image grabber ever! Thank you for supporting this project for so long! I moved to os X and found no chance to open *.exe file there.

Images with non "tumblr_" urls don't seem to be downloading from blogs

I'm not sure if it's specifically related to #19 or #27, but I've never noticed the url looking exactly like this before

https://66.media.tumblr.com/cd813bc6ec731b7057f6f250ae13d587/tumblr_pusoqc8FmZ1uw3f6jo1_1280.jpg
https://66.media.tumblr.com/cfafe79f6a7713098cea73103d5bd647/171e761be99622dd-92/s1280x1920/cd36d02b9327562f651244930596cdaeb2871b6a.png

the first link works fine, but the second link, doesn't show up in the files despite being the newest image posted on the blog. am I doing something incorrect? i've tried with another blog and i've been unable to get the files that have that same type of url

File rename functionality

Original issue TumblThree/#193

Okay, since so many people keep asking this, the next thing I'll implement is a file renamer. Since I don't have that much time for the next two months, I'll lay out how I think we should do this. Thus, if anyone wants to implement something, feel free to do it:

I think we should inject it into the AbstractDownloader and rename the files right before we download them. Or we implement it into the TumblrPosts classes and add a rename method for all of the different post classes. Then all we have to do is to call post.rename().

Injecting/renaming it right before we download them and not during the crawl would have the benefit that we can compare the original found file name against the databases. Thus, all databases stay valid, and changing the rename pattern doesn't redownload the same file again.

We'll have to send the whole post or all eventually useful information down the IProducerConsumer queue, thus we'll have to modify the TumblrPosts so that it contains reblog, tag, date, title, etc. information. Everything someone might need.

Then we could add a new text box into the details view where we allow some predefined pattern for file renaming. For example something like:

%d-%p-%h__fixedstring

would create outputs like

20181001-title-tumblrhash__fixedstring.png

so that %d stands for the postdate, %t for all tags, %h for the tumblr hash, %r for reblog, %p for the post title, %k for the keyblog key. Or similar.

I'm not sure if it's possible to add a counter for all downloaded files that starts at the first post of the blog and rises, beginning with zero, but the tumblr post id is unique and rises and is mapped to each post. Thus, it will not start at zero, but at some offset and has holes, but you can eventually sort the files using the post id. It think having a counter that starts at zero is hard to do with all the async code, and if some file is inaccessible, or someone only wants to count pictures, but not videos, it's getting really messy to implement this.

If anyone has any ideas of how to do this better, feel free to comment.

Retarget

The existing minimum requirements and current tooling should be reviewed. Lets have a clear vision of the minimum user requirements we will support from now on, what platform version of .Net we shall target, how the build and release process will work etc.

The solution and projects should all be upgraded and legacy support dropped.

Deploy TPL Dataflow/Actor

In the AbstractDownloader.cs and *Crawler classes, use something like TPL Dataflow/Actor model instead of creating a huge list of Tasks for tracking and semaphores for pressure management. Next to being less error prone (improper exception handling in the downloader/crawler might lead to a stall) it should be more convenient to use.

Initially, TumblThree was one of my toy projects to "learn a new language", a try to build a nice open source application without spaghetti code (i.e. usage DI, MVVM, "clean code", etc) and once in my life setup a new project from scratch. As such, I wanted to use as few third party libraries as possible with the additional goal of implement as much as possible by myself using the raw .NET framework.

Vision

We need to have a clearly defined scope of what TumblThree is, what TumblThree wants to be, and what the expectation of the core team are regarding it. Lets agree on consensus from the start, and work together towards something awesome!

[Resolved]Application Crashes Immediately

So, I'm not even sure whats going on, few months ago the application ran smooth now, i went into the folder, double tapped the app the app loads app, white screen and crashes.

Tried downloading again and unpacking in another folder and still same thing happen, has anyone experienced it before?

Issue Resolved, Probably something I did with my system files caused the problem (pc reset fixed it)

Installation files zipped under TumbleThree sub folder, not TumblThree

The installation zip file of the last few versions of TumblThree has the installation files packed under the sub folder TumbleThree, not under TumblThree as before. This means that if you unpack the installation files in the same folder as before, then a new installation folder will be created and the old files will not be overwritten. The blog folder will then also be changed. Is this done on purpose? If not, then it will be great if this can be fixed since the installation process will be easier.

In case you do not want to change the installation folder, is it possible for the users to manually install the files in the old TumblThree folder? Will all the old files be compatible with the new versions?

not downloading webp images and webm videos

webp and webm are becoming common and should be added. Not sure if it's here :

R:\z\TumblThree-1.0.11.2\src\TumblThree\TumblThree.Applications\Parser\ImgurParser.cs(28) : public Regex GetImgurImageRegex() => new Regex("(http[A-Za-z0-9_/:.]i.imgur.com/([A-Za-z0-9_])(.jpg|.png|.gif|.gifv))");
R:\z\TumblThree-1.0.11.2\src\TumblThree\TumblThree.Applications\Downloader\AbstractDownloader.cs(33) : private readonly string[] suffixes = { ".jpg", ".jpeg", ".png" };

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.