[FR]: Opt-In NoHardlinking detection outside of root directory about qbit_manage HOT 27 CLOSED

Hijacking a GitHub is not a valid way for support.
Sharelimits have no relevance to the noHL tagging

Based on your QBm paths - the fact you have /downloads at all means you lied and never followed trash guides at all.

visit the QBm channel on the Notifiarr discord or use Discussions for support

Can all but guarantee you failed to configure your mounts / ownership / permissions correctly and you don't actually have hardlinks.

from qbit_manage.

QBm Cross seed is just .torrent files, no? For inject

from qbit_manage.

@bobokun, qbit_manage is now detecting the hardlinks in the root dir and removing the noHL tag! Thank you so much!

from qbit_manage.

Hi, sorry for reopening this, I'm having the same issue. I'm also on develop branch and added the ignore_root_dir parameter to my config (see below). My goal is to have every noHL file cleaned up as hardlink deletion is managed by jorren92/Maintainerr which rules include my share time limits. Although, for now its features does not include deletion from download client, hence going through qbitmanage. But I don't get why HL aren't detected, my libraries patterns respect Trash guides as follows :

-downloads
  --series
  --anime
  --tv 
-data #where sonarr and radar import and HL files.
 --series
 --anime
 --tv

The config file :

commands:
  # The commands defined below will IGNORE any commands used in command line and docker env variables.
  dry_run: true
  cross_seed: false
  recheck: false
  cat_update: false
  tag_update: true
  rem_unregistered: true
  tag_tracker_error: false
  rem_orphaned: true
  tag_nohardlinks: true
  share_limits: true
  skip_qb_version_check: false
  skip_cleanup: true

qbt:
 ####

settings:
  force_auto_tmm: true # Will force qBittorrent to enable Automatic Torrent Management for each torrent.
  tracker_error_tag: issue # Will set the tag of any torrents that do not have a working tracker.
  nohardlinks_tag: noHL # Will set the tag of any torrents with no hardlinks.
  share_limits_tag: ratio # Will add this tag when applying share limits to provide an easy way to filter torrents by share limit group/priority for each torrent
  ignoreTags_OnUpdate: # When running tag-update function, it will update torrent tags for a given torrent even if the torrent has at least one or more of the tags defined here. Otherwise torrents will not be tagged if tags exist.
    #- noHL
    #- issue
    #- cross-seed
  - issue
  - cross-seed
  share_limits_min_seeding_time_tag: MinSeedTimeNotReached
  share_limits_min_num_seeds_tag: MinSeedsNotMet
  share_limits_last_active_tag: LastActiveLimitNotReached
  cross_seed_tag: cross-seed
  cat_filter_completed: true
  share_limits_filter_completed: true
  tag_nohardlinks_filter_completed: true
  force_retag_all: false
directory:
  # Do not remove these
  # Cross-seed var: </your/path/here/>  # Output directory of cross-seed
  # root_dir var: </your/path/here/>  # Root downloads directory used to check for orphaned files, noHL, and RecycleBin.
  # <OPTIONAL> remote_dir var: </your/path/here/>  # Path of docker host mapping of root_dir.
  # remote_dir must be set if you're running qbit_manage locally and qBittorrent/cross_seed is in a docker
  # remote_dir should not be set if qbit_manage is running in a container
  # <OPTIONAL> recycle_bin var: </your/path/here/>   # Path of the RecycleBin folder. Default location is set to remote_dir/.RecycleBin
  # <OPTIONAL> torrents_dir var: </your/path/here/>  # Path of the your qbittorrent torrents directory. Required for `save_torrents` attribute in recyclebin
  # <OPTIONAL> orphaned_dir var: </your/path/here/>  # Path of the the Orphaned Data folder. This is similar to RecycleBin, but only for orphaned data.
  #cross_seed: /your/path/here/
  root_dir: /downloads
  recycle_bin: /downloads/.RecycleBin
  torrents_dir: /qbittorrent/data/BT_backup
  orphaned_dir: /downloads/orphaned_data
  cross_seed:
  remote_dir: /downloads/
cat:
  # Category & Path Parameters
  # <Category Name> : <save_path>  # Path of your save directory.
  movies: /downloads/movies
  tv: /downloads/tv
  anime: /downloads/anime
  software: /downloads/software
  comics: /downloads/comics
  books: /downloads/books
  music: /downloads/music
  anime_movies: /downloads/anime
  power-seed: /downloads/power-seed
  scepticotheque: downloads/scepticotheque

cat_change:
  ####
tracker:
  ####
nohardlinks:
  # Tag Movies/Series that are not hard linked outside the root directory
  # Mandatory to fill out directory parameter above to use this function (root_dir/remote_dir)
  # This variable should be set to your category name of your completed movies/completed series in qbit. Acceptable variable can be any category you would like to tag if there are no hardlinks found
  movies:
    ignore_root_dir: true
  anime:
    ignore_root_dir: true
  tv:
    ignore_root_dir: true

And the result is that almost every torrent in those categories are being tagged in the dry run :

|====================================================================================================|
|                                            Finished Run                                            |
|                                     Total Torrents Tagged: 838                                     |
|                                  Total noHL Torrents Tagged: 828                                   |
|                                   Total Share Limits Updated: 5                                    |
|                                         Run Time: 0:00:28                                          |
|              Current Time: 01:09 PM | 1 day until the next run at 2024-05-07 01:09 PM              |
|====================================================================================================|

What am I doing wrong ?
Thanks for everything, btw !

from qbit_manage.

@Nesego , Please take a look at the following in order to set up the folder structure properly https://trash-guides.info/Hardlinks/Hardlinks-and-Instant-Moves/

Once you have completed that guide then please following this guide to set up qBittorrent properly.
https://trash-guides.info/Downloaders/qBittorrent/Basic-Setup/

from qbit_manage.

Noted.

from qbit_manage.

I fail to see how share limits doesn't accomplish this. Please explain in detail what you've tired and why it doesn't work?

cross-seeded torrents already have a cross-seed tag and you can use tags and categories for filtering accordingly.

from qbit_manage.

The specific functionality that I want is for cross seeds to be removed as soon as the file from which they’re seeded from is deleted.

Eg:
I download torrent1 to build ratio, so it not hardlinked outside the root dir
Cross-seed then grabs torrent2 and torrent3 that are just torrent1 from other trackers.
The only way to detect wether torrent1 is deleted is to check if torrent2 and torrent3 are hardlinked to something, but since torrent1 is still only in rootdir then torrent2 and 3 are tagged with noHL and automatically deleted.

There is, to my knowledge, no share limits rule to detect whether the original file has been deleted. However with the option to refine how hardlinks are checked this functionality would be available and would greatly improve my torrenting flow as well as the flows of other autobrr+cross-seed users.

from qbit_manage.

• torrent 1: hardlinks outside of linkdir 0, hardlinks inside 1. Share limits can seed to a ratio and for at least X time or to a specific seed time for this tracker or for this tracker with no hardlinks or for this tracker with a specified tag
• torrent 2 and 3 added: hardlinks outside of linkdir 0, hardlinks inside 1. Share limits can seed to a ratio ratio and for at least X time or to a specific seed time for these tracker or for this tracker with no hardlinks or for this tracker with a specified tag

again, please provide in detail...with specific configurations... that prove they not work and it is not possible with current settings. noHL have nothing to do with share limits which can use any tag or category combination to seed to a seed time or seed ratio.

any torrent that QBm detects with no upload but 100% complete should be getting the cross seed tag

from qbit_manage.

With what you’ve put above there is nothing that would remove the cross-seeded torrents when torrent1 reaches its share limits and is removed.

My current setup has share limits for the ratio-focused torrents based on seed time and share ratio. Once those are met, the torrent is removed, and you are exactly right that qbit perfectly removes that torrent. I then have a share limit for torrents with the cross-seed AND noHL tags to instantly remove them. Torrents that do not have the noHL tag do not have seeding limits and are seeded until the noHL tag is removed.

The problem is that there is no way for the ratio-building torrents to be managed by qbit manage while also having the noHL tag added to cross-seeded torrents when the original is removed.

In case 1 (in the original issue post I made), the noHL tag will instantly be added to the cross seeded torrents and thus my setup will instantly remove them. Obviously not an ideal setup.

In case 2, the noHL tag will not be added on the torrent originally, as the ratio-building torrent exists outside the root directory so the cross-seeded torrents will have hardlinks outside the root directory. When the original is removed the noHL will also be removed. However, since the ratio-building torrents are located outside of the root dir qbit manage will never automatically delete them. This would use up all of the data on my harddrive without manual intervention.

from qbit_manage.

Closing as Op has failed to provide the required information - config details - despite multiple requests. Repeating the same thing in slightly rephrasing is not helpful.

the noHL tag will instantly be added to the cross seeded torrents and thus my setup will instantly remove them. Obviously not an ideal setup.

QBm would never remove noHL unless you tell it to. Create better share limits. Again you have repeatedly refused to provide the details for why share limits fail to accomplish this. Tag and cat filtering is supported which would solve this.
there is absolutely nothing that would trigger a noHL to be instantly removed ever other than poor configuration and QBm doing exactly the user configured.

the noHL tag will not be added on the torrent originally, as the ratio-building torrent exists outside the root directory so the cross-seeded torrents will have hardlinks outside the root directory. When the original is removed the noHL will also be removed. However, since the ratio-building torrents are located outside of the root dir qbit manage will never automatically delete them. This would use up all of the data on my harddrive without manual intervention

not a valid use case for QBm to manage files outside of the download dir

from qbit_manage.

you can just disable linkDir (undefined) and itll reference the original file for the injected torrent.
when they are deleted they should error when pieces are requested. delete those torrents in error with the cross-seed tag from your client.

from qbit_manage.

Closing as Op has failed to provide the required information - config details - despite multiple requests. Repeating the same thing in slightly rephrasing is not helpful.

"My current setup has share limits for the ratio-focused torrents based on seed time and share ratio. Once those are met, the torrent is removed, and you are exactly right that qbit perfectly removes that torrent. I then have a share limit for torrents with the cross-seed AND noHL tags to instantly remove them. Torrents that do not have the noHL tag do not have seeding limits and are seeded until the noHL tag is removed." - my previous message.

Bar literally pasting my config file, how does this not suffice for config details? This described exactly what the various share limits that are relevant to this situation are. My config purposefully removed torrents tagged both with the cross-seed and noHL tags. The issue is that because qbit manage only considers hardlinks outside the root directory, so qbit automatically tagged such cross seeded torrents with the noHL tag, and thus the share limits I intentionally configured will automatically remove them.

This share limit configuration works in all other situations, an example of which I'll discuss:

1. I have a file in my media directory that I have not yet watched.
2. Cross seed grabs a torrent containing that file, creates the hardlinks for the torrent data and begins seeding. The torrent is hardlinked outside of the root directory and so does not get the noHL tag.
3. I watch that file in jellyfin, and automatically delete it. Qbit manage then does not detect any hardlinks outside of the root directory, tags the cross seeded torrent with the noHL tag, so my share limits automatically remove it.

In effect, this is a sharelimit for cross seeded torrents that removes the torrents the moment the original file is removed. The point I have tried to make is that this does not work when the original torrent is not hardlinked outside the root directory, as in the case of ratio-focused torrents.

I apologize if my responses have seemed like rephrasings, I have tried to expand on what I mean in each successive message. If it would be beneficial, I can literally copy and paste the relevant sections of my config file here, though if this is what you expected it may be opportune to explicitly say that. As I referenced at the beginning of this message, details about my configuration were in my previous message.

What other information would be beneficial for you so that I can help describe the situation in more complete detail?

(Also, while I'm guessing the issue template auto-assigns the owner of the repo to the issue, if @bobokun does have time to look at this issue I would be very appreciative 😄)

from qbit_manage.

you can just disable linkDir (undefined) and itll reference the original file for the injected torrent.

when they are deleted they should error when pieces are requested. delete those torrents in error with the cross-seed tag from your client.

I like this workaround! I would hope that some config in qbit manage would be created to help automate this, but in the interim this is a very nice manual workaround. Thanks!

from qbit_manage.

Bar literally pasting my config file, how does this not suffice for config details?

Because what you think you are configured to do, and what your config says can be two entirely different things. The latter gives concrete proof of the behavior you should see, not your interpretations in narrated form.

While you may be right about your specific config in this particular instance, not saying you are wrong - I don't know obviously and I'm a cross-seed dev and Deluge user, so take that into account - as a standard for users with issues, if you give us your config, we can immediately say "yep this" or "nope he's wrong and misunderstands" - instantly.

I like this workaround! I would hope that some config in qbit manage would be created to help automate this, but in the interim this is a very nice manual workaround. Thanks!

Your welcome. Hope it's sufficient to meet your goals. :)

from qbit_manage.

Because what you think you are configured to do, and what your config says can be two entirely different things. The latter gives concrete proof of the behavior you should see, not your interpretations in narrated form.

While you may be right about your specific config in this particular instance, not saying you are wrong - I don't know obviously and I'm a cross-seed dev and Deluge users, so take that into account - as a standard for users with issues, if you give us your config, we can immediately say "yep this" or "nope he's wrong and misunderstands" - instantly.

Ooo that's a good point! Here're the relevant parts of my config then!

# This is an example configuration file that documents all the options.
# It will need to be modified for your specific use case.
# Please refer to the link below for more details on how to set up the configuration file
# https://github.com/StuffAnThings/qbit_manage/wiki/Config-Setup

commands:
  # The commands defined below will IGNORE any commands used in command line and docker env variables.
  dry_run: false
  cross_seed: false
  recheck: false
  cat_update: true
  tag_update: true
  rem_unregistered: false
  tag_tracker_error: true
  rem_orphaned: false
  tag_nohardlinks: true
  share_limits: true
  skip_qb_version_check: false
  skip_cleanup: false

qbt:
  # qBittorrent parameters
  host: localhost:8081

  user:
  pass:
settings:
  force_auto_tmm: true # Will force qBittorrent to enable Automatic Torrent Management for each torrent.
  tracker_error_tag: issue # Will set the tag of any torrents that do not have a working tracker.
  nohardlinks_tag: noHL # Will set the tag of any torrents with no hardlinks.
  share_limits_tag: ~share_limit # Will add this tag when applying share limits to provide an easy way to filter torrents by share limit group/priority for each torrent
  ignoreTags_OnUpdate:
  - cross-seed
  - noHL
  - ratioUP
  - PrivateTracker
  - autobrr
  share_limits_min_seeding_time_tag: MinSeedTimeNotReached
  share_limits_min_num_seeds_tag: MinSeedsNotMet
  share_limits_last_active_tag: LastActiveLimitNotReached
  cross_seed_tag: cross-seed
  cat_filter_completed: true
  share_limits_filter_completed: true
  tag_nohardlinks_filter_completed: true
  force_retag_all: true
directory:
  # Do not remove these
  # Cross-seed var: </your/path/here/>  # Output directory of cross-seed
  # root_dir var: </your/path/here/>  # Root downloads directory used to check for orphaned files, noHL, and RecycleBin.
  # <OPTIONAL> remote_dir var: </your/path/here/>  # Path of docker host mapping of root_dir.
  # remote_dir must be set if you're running qbit_manage locally and qBittorrent/cross_seed is in a docker
  # remote_dir should not be set if qbit_manage is running in a container
  # <OPTIONAL> recycle_bin var: </your/path/here/>   # Path of the RecycleBin folder. Default location is set to remote_dir/.RecycleBin
  # <OPTIONAL> torrents_dir var: </your/path/here/>  # Path of the your qbittorrent torrents directory. Required for `save_torrents` attribute in recyclebin
  # <OPTIONAL> orphaned_dir var: </your/path/here/>  # Path of the the Orphaned Data folder. This is similar to RecycleBin, but only for orphaned data.
  cross_seed: /home/media/CrossSeeds/
  root_dir: /home/media/Torrents/
  recycle_bin: /home/media/.TorrentsRecycleBin
  torrents_dir: [qbit folder]
  orphaned_dir: /home/media/Orphans
  remote_dir: /home/media/Torrents/
cat:
  Torrents: /home/media/Torrents/
  ratioUP: /home/media/Torrents/RatioUPTorrents
  cross-seed-data: /home/media/Torrents/CrossSeedLinks/

  # Category & Path Parameters
  # <Category Name> : <save_path>  # Path of your save directory.
cat_change:
  # removed

tracker:
  # removed, but importantly every private tracker gets the PrivateTracker tag
  # The "other" key is a special keyword and if defined will tag any other trackers that don't match the above trackers into this tag
  other:
    tag: [other]

nohardlinks:
  # Tag Movies/Series that are not hard linked outside the root directory
  # Mandatory to fill out directory parameter above to use this function (root_dir/remote_dir)
  # This variable should be set to your category name of your completed movies/completed series in qbit. Acceptable variable can be any category you would like to tag if there are no hardlinks found
  Torrents:
  ratioUP:
  cross-seed-data:


share_limits:
  # Control how torrent share limits are set depending on the priority of your grouping
  # Each torrent will be matched with the share limit group with the highest priority that meets the group filter criteria.
  # Each torrent can only be matched with one share limit group
  # This variable is mandatory and is a text defining the name of your grouping. This can be any string you want
  noHL_Private:
    priority: 110
    include_all_tags: [noHL]
    include_any_tags: [PrivateTracker, cross-seed]
    exclude_any_tags: [tags related to specific private trackers]
    max_seeding_time: 8640
    cleanup: true
  noHL_[Specific tracker]:
    # like above but for specific trackers
  noHL_CrossSeed:
    priority: 100
    include_all_tags: [noHL, cross-seed]
    categories:
    - cross-seed-data
    max_seeding_time: 1 # this is what should be responsible for automatically removing cross-seeded torrents when the noHL tag is applied
    cleanup: true

  noHL_Public:
    priority: 111
    include_all_tags: [noHL, other]
    exclude_all_tags: PrivateTracker
    max_ratio: 1
    max_seeding_time: 1440
    cleanup: true
  Hardlinked:
    priority: 9999
    exclude_all_tags: noHL
    cleanup: true

  ratioUP:
    priority: 10
    categories:
    - ratioUP
    max_seeding_time: 7200
    min_seeding_time: 7200
    max_ratio: 1
    cleanup: true

  ratioUP_[For specific tracker]:
    - # again, like above but for specific trackers
  default:
    priority: 10000
    cleanup: true
recyclebin:
  # Recycle Bin method of deletion will move files into the recycle bin (Located in /root_dir/.RecycleBin) instead of directly deleting them in qbit
  # By default the Recycle Bin will be emptied on every run of the qbit_manage script if empty_after_x_days is defined.
  enabled: true
  # <OPTIONAL> empty_after_x_days var:
  # Will automatically remove all files and folders in recycle bin after x days. (Checks every script run)
  # If this variable is not defined it, the RecycleBin will never be emptied.
  # WARNING: Setting this variable to 0 will delete all files immediately upon script run!
  empty_after_x_days: 0
  # <OPTIONAL> save_torrents var:
  # If this option is set to true you MUST fill out the torrents_dir in the directory attribute.
  # This will save a copy of your .torrent and .fastresume file in the recycle bin before deleting it from qbittorrent
  save_torrents: true
  # <OPTIONAL> split_by_category var:
  # This will split the recycle bin folder by the save path defined in the `cat` attribute
  # and add the base folder name of the recycle bin that was defined in the `recycle_bin` sub-attribute under directory.
  split_by_category: false

orphaned:
  # Orphaned files are those in the root_dir download directory that are not referenced by any active torrents.
  # Will automatically remove all files and folders in orphaned data after x days. (Checks every script run)
  # If this variable is not defined it, the orphaned data will never be emptied.
  # WARNING: Setting this variable to 0 will delete all files immediately upon script run!
  empty_after_x_days: 1
  # File patterns that will not be considered orphaned files. Handy for generated files that aren't part of the torrent but belong with the torrent's files
  exclude_patterns:
  - '**/.DS_Store'
  - '**/Thumbs.db'
  - '**/@eaDir'
  - /data/torrents/temp/**
  - '**/*.!qB'
  - '**/*_unpackerred'

from qbit_manage.

A simple solution would be to move your cross_seed directory to somewhere inside your root_dir.
In your example it would look something like:

  cross_seed: /home/media/Torrents/CrossSeeds/
  root_dir: /home/media/Torrents/
  recycle_bin: /home/media/.TorrentsRecycleBin
  torrents_dir: [qbit folder]
  orphaned_dir: /home/media/Orphans
  remote_dir: /home/media/Torrents/

from qbit_manage.

It is not really recommended to put linked cross-seeds inside your data directory if using data matching. Just as a note.

from qbit_manage.

oh your right, I got confused. I meant cross seed data, but I see it's already in the root folder. cross-seed-data: /home/media/Torrents/CrossSeedLinks/

from qbit_manage.

Yea that's not recommended setup...but the consequences generally are just that you could have duplicate searches. Not the worst, but kind of something you'd avoid if you can.

from qbit_manage.

@ddxtanx

Can you please test in the latest develop branch?
See below for the new configuration:

from qbit_manage.

@bobokun

Just tested this out, set ignore_root_dir: false in the cross-seed-data config, but it is still marking those torrents as noHL. When running with -thnl -ll trace, all of the logs for the cross-seeded torrents have the output below

| Checking file inum: [inum]                                                                       |
| Checking file size: [size]                                                                     |
| Checking no of hard links: 2                                                                       |
| Checking inode_count dict: 2                                                                       |

but the torrents are still tagged with noHL in qbittorrent.

Are there any other logs/stuff that I could put here to help w/ debugging?

from qbit_manage.

Also, thank you so much for implementing this feature!

from qbit_manage.

@ddxtanx can you please try again in develop branch? I've put some more information in the trace logs.

from qbit_manage.

@bobokun

And it just successfully detected when an original torrent was removed, retagged the cross seeded torrent, and removed that too perfectly!

from qbit_manage.

Okay, sorry for the "hijack", thought it was legit. Although I'm not lying about Trash guides, don't presume of my intentions, I just probably did it wrong. Not everybody has your knowledge.

from qbit_manage.

Okay, sorry for the "hijack", thought it was legit. Although I'm not lying about Trash guides, don't presume of my intentions, I just probably did it wrong. Not everybody has your knowledge.

The knowledge is available on the guides, that is their purpose. Looking at your paths it is immediately apparent that you either misread them drastically or did not follow them at all.

Either way, github etiquette is a thing, especially on larger user-base projects....seeking support in an issue for misconfigurations on your part before normal channels (in this case discord) is, generally, kind of a no-no - on top of posting in a thread barely adjacent to the issue you are having.

from qbit_manage.