Code Monkey home page Code Monkey logo

szurubooru-auto-tagger's Introduction

szurubooru-auto-tagger

IQDB and Danbooru based auto tagger for szurubooru2.

Notice

I'm no longer actively maintaining this project. I'll try to fix critical bugs and merge pull requests that fixes bugs but no new features will be added. This tool should continue to work until external APIs change too much to keep up.

I think this project deserves a rewrite to fix bad design decisions, but I'm no longer using szurubooru, so I won't be doing that.

How this works

Auto tagger searches for all posts having trigger tag (auto_tagme by default), then sends post images to IQDB in order to find matching image on Danbooru. When match is found, Danbooru tags, post safety and notes are retrieved and set for Szurubooru post. New tag is added to mark post as being managed by auto tagger (auto_tagged by default). Later auto tagger may update tags on such posts. After posts tagging is finished, new tags that were created are collected and their data is updated from Danbooru tags. Auto tagger will update tag category but it can also obtain tag aliases, implications and suggestions. In case when no match is found, post are tagged with tagme so you can tag them manually. If you want you can also setup tag and tag category remapping.

Usage

  1. Make sure you have Java 11 or newer installed.
  2. Get jar from Releases page or build it yourself.
  3. Create separate account for auto-tagger and give it high enough privileges.
  4. On Szurubooru create tag categories: general, artist, character, copyright, meta. Note: you can remap categories in config file. Default category should be set to general or your custom one.
  5. Create config.yaml file. Either override what you need from config.default.yaml or copy that file and modify it.
  6. Backup your stuff and run auto-tagger: java -jar auto-tagger.jar --config /path/to/config/file.yaml

By default only new posts will be tagged, you can specify different task using --task switch. Use --help to see list of tasks. Except standard tasks used for post tagging, you can also use BatchUpload to quickly upload all images from directory. Note that uploaded images will be moved to uploaded subdirectory to simplify upload resuming.

Help message

Result of java -jar auto-tagger.jar --help

Szurubooru auto tagger. Usage: [-c config-file-path] [-t task task-argument1 task-argument2 task-argument3 ...]
config-file-path: (-c, --config) Path to configuration file. If not specified config.yaml is used or config.default.yaml if former does not exist.
task: (-t, --task) Optional, task that auto tagger will perform. By default 'NewPosts' is used. This parameter can be: 
	NewPosts - Updates new posts (having config.triggerTag)
	ExistingPosts - Updates already tagged posts (having config.managedTag)
	NewTags - Updates tags that weren't ever updated
	ExistingTags - Updates existing tags
	Posts - Updates specified posts, you must specify post ids: Posts <postId1> [postId2] [postId3] ...
	Tags - Updates specified tags, you must specify tag names: Tags <tagName1> [tagName2] [tagName3] ...
	Notes - Updates specified post notes only, you must specify post ids: Notes <postId1> [postId2] [postId3] ...
	BatchUpload - Upload all image files from given directory. You must specify path to source directory: BatchUpload <path>. Warning: Uploaded images will be moved to 'uploaded' subdirectory to simplify upload resuming.
	BatchDownload - Download all images matching search query. You must specify query and optionally output directory: BatchDownload <searchQuery> [outputPath]. 
task-arguments: Optional, only needed if tasks requires passing argument. Must be specified as last config parameter

Developing

./gradlew run # run from sources
./gradlew jar # create fat jar under build/libs
./gradlew zip # create dist zip under build/distributions

szurubooru-auto-tagger's People

Contributors

kotcrab avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

szurubooru-auto-tagger's Issues

Exceptions being thrown when attempting to update existing tags.

I have a local server set up running Szurubooru using Docker and am trying to run this on a separate Windows 10 machine. I am attemping to update existing tags to get their categories automatically. When attempting to use Danbooru authenticated I get the following error (everything in square brackets has been redacted):

PS C:\Users\[W10user]\Downloads\szurubooru-auto-tagger-1.2\bin> .\szurubooru-auto-tagger -t ExistingTags
Szurubooru auto tagger
Using config from C:\Users\[W10user]\Downloads\szurubooru-auto-tagger-1.2\lib\config.yaml
Exception in thread "main" javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at sun.security.ssl.Alerts.getSSLException(Unknown Source)
at sun.security.ssl.Alerts.getSSLException(Unknown Source)
at sun.security.ssl.SSLSocketImpl.recvAlert(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:563)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:540)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:227)
at com.kotcrab.szurubooru.tagger.RestClient.executeSafely(RestClient.kt:71)
at com.kotcrab.szurubooru.tagger.RestClient.get(RestClient.kt:37)
at com.kotcrab.szurubooru.tagger.RestClient.get$default(RestClient.kt:36)
at com.kotcrab.szurubooru.tagger.Danbooru.getPost(Danbooru.kt:70)
at com.kotcrab.szurubooru.tagger.Danbooru.isAuthorized(Danbooru.kt:40)
at com.kotcrab.szurubooru.tagger.AutoTagger.(AutoTagger.kt:59)
at com.kotcrab.szurubooru.tagger.MainKt.main(Main.kt:55)

Looking at that I figured I would give it a shot unauthenticated and here is what I got from that:

PS C:\Users\[W10user]\Downloads\szurubooru-auto-tagger-1.2\bin> .\szurubooru-auto-tagger -t ExistingTags
Szurubooru auto tagger
Using config from C:\Users\[W10user]\Downloads\szurubooru-auto-tagger-1.2\lib\config.yaml
Booru connectivity looks ok
Reading tag map...
Obtaining tags.json...
Exception in thread "main" com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 12 path $
at com.google.gson.JsonParser.parse(JsonParser.java:65)
at com.google.gson.JsonParser.parse(JsonParser.java:45)
at com.kotcrab.szurubooru.tagger.RestClient.get(RestClient.kt:37)
at com.kotcrab.szurubooru.tagger.RestClient.get$default(RestClient.kt:36)
at com.kotcrab.szurubooru.tagger.Szurubooru.getTags(Szurubooru.kt:54)
at com.kotcrab.szurubooru.tagger.AutoTagger.(AutoTagger.kt:69)
at com.kotcrab.szurubooru.tagger.MainKt.main(Main.kt:55)
Caused by: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 12 path $
at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1567)
at com.google.gson.stream.JsonReader.checkLenient(JsonReader.java:1416)
at com.google.gson.stream.JsonReader.doPeek(JsonReader.java:546)
at com.google.gson.stream.JsonReader.peek(JsonReader.java:429)
at com.google.gson.JsonParser.parse(JsonParser.java:60)
... 6 more

Here is my config.yaml (indentationlooks weird here, but it is correct in the file):

triggerTag: auto_tagme
managedTag:
errorTag: auto_tagger_error
noMatchTag: tagme
batchUploadTag: auto_tagme

checkBooruConnectivity: true
storeSourceUrl: false
updateImageRating: true
updateImageNotes: true
createCommentWhenBiggerImageFound: true

singleInstance:
enabled: true
port: 54212

tags:
obtainAliases: true
obtainSuggestions: false
obtainImplications: true
tagMapFile: tagMap.yaml
tagEscaping:
escapeCharacters: '/+'
escapeWith: _

remapCategories:
- from: general
to: default
- from: artist
to: creator
- from: character
to: hero
- from: copyright
to: series

szurubooru:
apiPath: http://192.168.1.129:6666
dataPath: http://192.168.1.129:8121
username: auto-tagger
password: auto-tagger

danbooru:
anonymous: false
username: [username]
apiKey: [API key]
hourRequestLimit: 3000

Java heap out of memory error

I encounter java.lang.OutOfMemoryError: Java heap space error when running the program for the second time.

Command: java -jar szubooru-auto-tagger-1.3.jar --config config-szu-tag.yaml

Szurubooru auto tagger Using config from config-szu-tag.yaml Booru connectivity looks ok Reading tag map... Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.base/java.util.regex.Matcher.<init>(Matcher.java:248) at java.base/java.util.regex.Pattern.matcher(Pattern.java:1133) at com.esotericsoftware.yamlbeans.tokenizer.Tokenizer.fetchMoreTokens(Tokenizer.java:312) at com.esotericsoftware.yamlbeans.tokenizer.Tokenizer.peekNextToken(Tokenizer.java:120) at com.esotericsoftware.yamlbeans.tokenizer.Tokenizer.peekNextTokenType(Tokenizer.java:125) at com.esotericsoftware.yamlbeans.parser.Parser$20.produce(Parser.java:320) at com.esotericsoftware.yamlbeans.parser.Parser.getNextEvent(Parser.java:80) at com.esotericsoftware.yamlbeans.parser.Parser.peekNextEvent(Parser.java:91) at com.esotericsoftware.yamlbeans.YamlReader.readValueInternal(YamlReader.java:270) at com.esotericsoftware.yamlbeans.YamlReader.readValue(YamlReader.java:152) at com.esotericsoftware.yamlbeans.YamlReader.read(YamlReader.java:103) at com.esotericsoftware.yamlbeans.YamlReader.read(YamlReader.java:90) at com.kotcrab.szurubooru.tagger.AutoTagger.readTagMap(AutoTagger.kt:82) at com.kotcrab.szurubooru.tagger.AutoTagger.<init>(AutoTagger.kt:66) at com.kotcrab.szurubooru.tagger.MainKt.main(Main.kt:55)

Running the program with more memory does not fix the issue. Tried this with ~2.5gb (maximum I can allocate at my Raspberry Pi) but no luck.

Command: java -Xmx2500m -jar szubooru-auto-tagger-1.3.jar --config config-szu-tag.yaml

Additional invocation options

Except standard auto tagger run allow to:

  • tag only new posts
  • update only existing posts tags
  • single post / tag
  • update existing tags that weren't updated since X days
  • notes only

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.